Watchtower README

Watchtower

Tech job monitoring for AI agents. Say what you are looking for once and get only the new postings that match, from the job boards of tech companies and startups, as structured JSON.

Agents are bad at waiting for a job to be posted. A session lasts minutes, careers pages are heavy to re-read, and "has anyone posted an iOS role in Austin yet?" turns into re-running the same search every day. With Watchtower the agent creates a watch once, in plain language:

{ "query": "iOS jobs in Austin making at least 150k a year with a maximum of 6 years of experience" }

Watchtower checks the job boards of tech companies and startups on a schedule and remembers which jobs were open. get_changes then returns only the new postings that match, and a webhook can wake the agent when one arrives. An agent can also watch one company's board by URL and get JOB_ADDED, JOB_REMOVED and JOB_UPDATED events.

• No URL needed. A watch created from a query covers every board Watchtower monitors: a built-in directory of tech company and startup boards plus every board anyone has watched by URL.
• Pay and experience. Jobs carry salary and experience_years when the posting states them, and watches filter on min_salary and max_experience_years.
• Greenhouse, Lever, Ashby, Workable, SmartRecruiters, Recruitee, Workday, iCIMS, Oracle Recruiting, Eightfold and SuccessFactors are read through each platform's own endpoints: no bot walls. So are Apple (jobs.apple.com), Google (careers.google.com), Amazon (amazon.jobs) and Microsoft (careers.microsoft.com), which run their own careers sites. Any other careers page works if it publishes schema.org JobPosting markup.
• Filters on the watch, so get_changes only returns what matters: keywords, all_keywords, exclude_keywords, locations, seniority, remote_only, min_salary and max_experience_years. Every job carries derived remote and seniority fields.
• Many companies in one call: pass urls instead of url.
• MCP (Streamable HTTP) and REST, backed by the same service layer.
• Free and anonymous: a client gets a token and up to 50 watches (a search across all boards is one watch).
• TypeScript, Node.js 22, Fastify 5, PostgreSQL, the official MCP TypeScript SDK. No LLM or third-party API keys required.

Use the hosted service

Watchtower runs at watchtower.lat, free, with no sign-up. MCP endpoint: https://watchtower.lat/mcp.

• Claude Code: claude mcp add --transport http watchtower https://watchtower.lat/mcp, or install the plugin, which adds a skill that tells Claude when to use it:
  CodeCopy

  /plugin marketplace add connorlagana/watchtower
  /plugin install watchtower@watchtower
  

• Claude.ai / Claude Desktop: Settings → Connectors → Add custom connector → https://watchtower.lat/mcp.
• Cursor, VS Code: one-click buttons at watchtower.lat/#install.
• Anything else: { "mcpServers": { "watchtower": { "type": "http", "url": "https://watchtower.lat/mcp" } } }.

Listed in the MCP Registry as lat.watchtower/watchtower (server.json).

Quick start

cp .env.example .env
docker compose up --build          # app on http://localhost:3000, Postgres alongside
docker compose --profile demo run --rm demo   # the end-to-end demo below

Without Docker (Node 22 and a running Postgres):

npm install
export DATABASE_URL=postgres://postgres:postgres@localhost:5432/watchtower
npm run migrate
npm run dev                         # or: npm run build && npm start

Demo

npm run demo (with DATABASE_URL set) runs the core loop end to end against a local fixture careers page:

1. create client → 2. create a job watch with keyword ios (takes the initial snapshot) → 3. a second agent watches the same board and reuses the same resource → 4. a third agent names no board and creates a search watch from "remote iOS jobs paying at least 150k with a maximum of 6 years of experience" → 5. a re-check where only the page around the jobs changed (session token, "N minutes ago") reports no change → 6. the source adds an iOS job and an Android job → 7. Watchtower checks and detects the change → 8. an MCP client calls get_changes:

{
  "changes": [
    { "type": "JOB_ADDED",
      "summary": "New job: Senior iOS Engineer (Remote - US)",
      "data": { "job": { "title": "Senior iOS Engineer", "location": "Remote - US", "company": "Acme Robotics", "url": "…/careers/ios-303" } } }
  ],
  "cursor": 2, "has_more": false
}

The Android job is filtered out by the keyword. 9. A second get_changes call returns []. 10. The second agent, with no keyword, sees both new roles. 11. The third agent gets the iOS role with the salary and experience read from the posting.

The demo runs its fixture on 127.0.0.1, so it turns on ALLOW_PRIVATE_NETWORKS for its own process only.

Connecting an agent (MCP)

{ "mcpServers": { "watchtower": { "type": "http", "url": "http://localhost:3000/mcp" } } }

watch_jobs also accepts webhook_url for push delivery (see below).

The tool descriptions and server instructions tell agents to prefer Watchtower over re-running job searches or re-checking careers pages.

Search watches

A watch with no url is a search watch. It is its filters, and it reads the new postings of every monitored board.

curl -s -X POST localhost:3000/v1/watches -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" \
  -d '{"query":"iOS jobs in Austin making at least 150k a year with a maximum of 6 years of experience"}'

{
  "scope": "all_boards",
  "interpreted": { "filters": { "keywords": ["ios"], "locations": ["austin"], "min_salary": 150000, "max_experience_years": 6, "include_unknown": true }, "notes": [] },
  "coverage": { "boards": 1705 },
  "matching_jobs_count": 3,
  "current_jobs": [ { "title": "Senior iOS Engineer", "company": "Acme", "location": "Austin, TX", "salary": { "min": 165000, "max": 210000, "currency": "USD", "period": "year", "annual_min": 165000, "annual_max": 210000 }, "experience_years": 5, "url": "…" } ]
}

• The query is read by rules, not a model. Watchtower still needs no LLM or API key. The reading comes back as interpreted, with notes for anything it could not use, so the calling agent can check it. Explicit filters always win over the query.
  • Role words become keywords (any of, for "iOS or Android") or all_keywords (all of, for "data scientist"). Generic words such as "engineer" and "developer" are dropped when a more specific word is present.
  • "in Austin", "in Austin, TX", "in New York or remote" become locations. A bare "remote" becomes remote_only.
  • "at least 150k", "$180,000+", "$45/hr" become min_salary, converted to a yearly figure.
  • "a maximum of 6 years of experience", "3-5 years", "I have 4 years of experience" become max_experience_years.
  • "senior", "staff", "entry level" and the other levels become seniority. "no managers" becomes exclude_keywords.
• Coverage is tech companies and startups, not the whole internet. The built-in directory (src/search/boards.ts) lists 1,705 company boards on Greenhouse, Lever, Ashby, Workable, SmartRecruiters, Workday, iCIMS, Oracle Recruiting, Eightfold and SuccessFactors, plus the careers sites of Apple, Google, Amazon and Microsoft, from seed-stage startups to large employers in every industry that hire developers. Each was confirmed against its platform's API with open jobs. Every role those companies post is covered, not only engineering. The live list is published at /companies and through list_companies. On top of that, every board any client watches by URL is covered, and stays covered (see Growing the directory).
• Pay and experience come from the posting. Lever, Ashby, Greenhouse, Recruitee and JSON-LD pay fields are read directly; otherwise the posting text is parsed ("$150,000 - $200,000/yr", "5+ years of experience"). A job passes min_salary when the top of its range reaches it, and max_experience_years when it asks for no more than that.
• Postings that state neither are still reported, without a salary or experience_years field, because many postings state no pay. Pass include_unknown: false to report only postings that state a qualifying value.
• Only new postings are reported (JOB_ADDED), and only those that appear after the watch was created. current_jobs on creation and on get_watch is the baseline of what is open now.

Growing the directory

The directory grows in three ways.

• Agents grow it by using it. When a client watches a platform board by URL and it has open jobs, the board joins the directory and stays there after that watch ends, so every search watch covers it from then on. At most INDEX_MAX_PROMOTED (5,000) boards are kept this way, and one that fails eight checks in a row is dropped. Arbitrary careers pages are never kept; only boards on the platform APIs.
• A careers page works as a way in. watch_jobs with https://acme.com/careers usually finds no job markup there, because the page only links to or embeds the company's board. Watchtower then reads the page once more, finds the link to a supported board, and watches that board instead. The response says so in resolved_from.
• Operators can import a company list. npm run discover -- companies.json boards.txt takes [{ "name", "website" }] and, for each company, reads its careers page for a board link, then tries boards named after the company on Ashby, Greenhouse and Lever. Every candidate is confirmed against the platform's listing API and kept only if it has open jobs. boards.txt is in the format INDEX_BOARDS_FILE reads. The built-in list was produced this way from the public Y Combinator company directory. A board found by name rather than from the company's own page can belong to a different company with the same name; the .json output records how each board was found.

Auth: the MCP connection should carry the identity, so a token never has to pass through a chat (careful assistants refuse to copy secrets from a transcript into tool calls, especially into delete_watch).

• OAuth (MCP authorization spec). /.well-known/oauth-protected-resource/mcp points at the built-in authorization server (/.well-known/oauth-authorization-server): dynamic client registration at /oauth/register, /oauth/authorize with PKCE (S256), /oauth/token. There are no accounts: the consent page creates an anonymous client in one click, or connects an existing one if the user pastes its token there, which keeps its watches. New clients draw on the same per-address budget as POST /v1/clients.
• How a client learns to sign in. Without a token, search_jobs and list_companies still work, and the other watch tools (and watch_jobs once provisioning is off) return UNAUTHORIZED with the challenge in _meta["mcp/www_authenticate"] (ChatGPT's mixed-auth signal; tools also declare securitySchemes). An unknown token gets an HTTP 401 with WWW-Authenticate, as does any request to /mcp?auth=required without one, for clients that only start OAuth on a 401 (Claude Code, Cursor).
• Or a header. Clients without OAuth send Authorization: Bearer <token> with a token from POST /v1/clients.
• Older conversations keep working. Every tool still accepts client_token, and while MCP_ANONYMOUS_PROVISIONING is on (the default) a watch_jobs call with no token at all creates an anonymous client and returns its token, as before. When a call carries both an OAuth connection and a client_token, and the connection's own client has no watches, the connection is moved to the client_token's client, so a user who connects the app keeps their watches. Turn provisioning off once the OAuth version of the ChatGPT app is live; tokenless watch_jobs calls then get the OAuth challenge and the watch tools declare OAuth as required.

REST API

All endpoints except POST /v1/clients need Authorization: Bearer <token>.

Also served: / (homepage/docs), /llms.txt, /.well-known/watchtower.json, /health, and /metrics (Prometheus; set METRICS_TOKEN to require a bearer token).

Errors look like { "error": "WATCH_LIMIT", "message": "…" }.

• Can't be monitored (422): NO_JOB_DATA (no supported platform and no JobPosting markup), ROBOTS_DISALLOWED, BOT_CHALLENGE, ACCESS_DENIED, SSRF_BLOCKED, UNSUPPORTED_CONTENT_TYPE, BODY_TOO_LARGE.
• Capacity: WATCH_LIMIT and HOST_WATCH_LIMIT (409), HOST_CAPACITY (429), CAPACITY (503).
• Transient failures (timeouts, 5xx, 404) keep the watch. They show up in resource.last_error and are retried with exponential backoff.

Delivery options

• Polling. get_changes returns what's new and advances the cursor. For at-least-once processing, call get_changes(peek=true), process the changes, then ack_changes(cursor). If you crash before the ack, the same changes come back.

• Webhooks. Pass webhook_url when creating a watch. The creation response includes a webhook_secret, shown once. Each batch of matching changes is POSTed as JSON with these headers:

  • x-watchtower-timestamp
  • x-watchtower-signature: sha256=<hex HMAC-SHA256 of "<timestamp>.<body>">
  • x-watchtower-delivery

  Deliveries come from an outbox and are retried with exponential backoff (8 attempts). Webhook URLs go through the same SSRF checks as monitored URLs, and redirects are not followed.

Watch lifecycle

A watch stays alive as long as someone uses it: get_changes, get_watch, list_watches or a successful webhook delivery renews it. Watches nobody touches for WATCH_TTL_DAYS (30) expire and stop costing fetches; expires_at is shown on every watch.

How it works

 watches (per client) ──many-to-one──▶ resources (fetch url + adapter)
                                            │  scheduler: due? one per host, host lease held
                                            ▼
                     fetch ─▶ job-board API adapter, or JobPosting JSON-LD
                                            │  diff the job set vs the previous snapshot
                                            ▼
                     snapshot + typed job events, one transaction
                                            │  read through each watch's keyword filter
                                            ▼
                      get_changes (per-watch cursor)   ·   webhook outbox → signed POST

A search watch has no resource of its own. It reads the same change events, from every monitored resource, through its filters.

• The board directory. Boards in the directory are ordinary resources flagged indexed. The scheduler checks them every INDEX_CHECK_INTERVAL_SECONDS (default four hours) whether or not anyone watches them, and the listed part of the directory is synced from src/search/boards.ts and INDEX_BOARDS_FILE at startup. One request per platform host is in flight at a time, so a platform's boards are checked one after another: with the default 5 s scheduler tick that is about 700 boards per platform per hour, or 2,900 per four-hour cycle. A directory larger than that is not an error; boards are checked as fast as politeness allows, and watchtower_directory_overdue_boards shows how far behind it is. A board with a watch on it is still checked at that watch's interval.

• Listing requests carry the posting text. Greenhouse (content=true), Lever, Ashby (includeCompensation=true), Workable (details=true) and Recruitee return each posting's description and pay in the listing response, so salary and experience cost no extra requests. Only the derived fields are stored, never the description. These responses are large, so platform APIs get their own body cap (MAX_API_BODY_BYTES, 64 MB); a board larger than that is read as a plain listing, without pay or experience.

• Board URLs map to platform endpoints. boards.greenhouse.io/acme, jobs.lever.co/acme, jobs.ashbyhq.com/acme, apply.workable.com/acme, jobs.smartrecruiters.com/Acme and acme.recruitee.com (plus their EU and embed variants) are fetched from each platform's public job-board API in one request. Other URLs are fetched as HTML and read through schema.org JobPosting JSON-LD (including @graph and ItemList). Only the job list is compared, so page churn around it (session tokens, "rendered N minutes ago", banners) never registers as a change.

  • Workday (acme.wd5.myworkdayjobs.com/Careers, wd3.myworkdaysite.com/recruiting/acme/External): the site's own search endpoint, a JSON POST that returns postings newest first, 20 per page. A check reads the newest 200 (10 requests). Boards with more postings are marked complete: false on the snapshot and never emit JOB_REMOVED, because a job leaving the window isn't a removal. Job links point at the public site; the relative "Posted 3 Days Ago" is not kept.
  • iCIMS (careers-acme.icims.com): the portal's lists every open job in one request (id plus a slug of the title), and the first page of gives the newest ~50 their real title, location and posting date. A check reads both. Jobs seen only in the sitemap are (title from the slug, no location); details learned earlier are carried forward, and a partial entry becoming a full one is not reported as an update.

Safety and abuse controls

• SSRF.
  • Only http/https on ports 80/443. URLs with credentials are rejected.
  • localhost, *.local, *.internal, single-label and numeric hosts are rejected.
  • Every resolved address must be globally routable unicast. Private, loopback, link-local (including cloud metadata), CGNAT, multicast, reserved, documentation, IPv4-mapped IPv6, 6to4, Teredo and NAT64 are all refused.
  • The check runs inside the socket's DNS lookup, so it holds at connect time for every redirect hop (defeating DNS rebinding) as well as once at watch creation.
  • Webhook URLs get the same treatment.
• Request limits. A 15 s overall deadline across all hops. At most 5 redirects, each re-validated. A 3 MB body cap (64 MB for job-board platform APIs) enforced on the decompressed bytes, so gzip and brotli bombs are caught. Text-like content types only.
• API limits. Rate limits are stored in Postgres, so they hold across replicas and restarts: 120 requests/min, 10 client creations/hour (shared with MCP auto-provisioning), and 10 forced checks/min. Clients are identified by IPv4 address or IPv6 /64.
• Watch and resource caps.
  • 50 watches per client, and 5 per client per website for arbitrary careers pages (platform boards are separate companies behind one API host and are exempt).
  • At most MAX_RESOURCES_PER_HOST distinct careers-page URLs per website across all clients; job-board platform APIs are exempt, and watching an already-monitored board is always allowed.
  • A global MAX_ACTIVE_RESOURCES.
• Tokens. Random 192-bit values; only their SHA-256 is stored. Request bodies are capped at 64 KB.
• No bypassing. Watchtower never works around CAPTCHAs, bot challenges, logins, paywalls or robots.txt. It detects them, tells the caller, logs host is refusing us, and counts them in watchtower_blocked_resources_1h.

Operations

GET /metrics exposes Prometheus metrics:

• check outcomes by error code;
• fetch-duration histogram;
• changes emitted by type;
• webhook results;
• host-busy deferrals;
• gauges for active watches/resources, search watches, directory boards and how many are overdue, failing and blocked resources, and pending webhooks;
• where agents come from: watchtower_mcp_initialize_total by the MCP client name an agent reports (claude-code, cursor, ...), and watchtower_clients_created_7d by the ?ref= tag on the URL a client was created through. Each listing and install path hands out its own tag (/mcp?ref=registry, ?ref=claude-plugin, ?ref=cursor, ...); clients.source and clients.user_agent keep it per client.

Usage stats

/stats shows users, daily active users, new users, tool calls, page views and where users come from, as charts and tables. Log in with any username and STATS_TOKEN (or METRICS_TOKEN) as the password; with neither set the page is off. A user is one anonymous client token. The history lives in usage_daily (one row per client, UTC day, tool and interface) and counts_daily (page views by people, AI assistants and other bots, and MCP connections by app name), kept for 400 days. Nothing per visitor is stored for page views.

Listing in the MCP Registry

server.json is the registry entry. The lat.watchtower/* namespace is proven by HTTP: set MCP_REGISTRY_AUTH to the public-key record and the app serves it at /.well-known/mcp-registry-auth. Then, with the matching private key:

mcp-publisher login http --domain watchtower.lat --private-key "$PRIVATE_KEY_HEX"
mcp-publisher publish          # bump "version" in server.json for each new publish

Robots, the MCP server card (/.well-known/mcp.json, also at /.well-known/mcp-server-card while the path is still a draft) and the install links on the homepage are generated from PUBLIC_BASE_URL.

ops/alerts.yml has example alert rules: sites refusing us, high error rate, checks stalled, webhook backlog, slow fetches, and the directory falling behind its check interval.

Configuration

See .env.example. The most important settings:

• DATABASE_URL and PUBLIC_BASE_URL.
• MIN_CHECK_INTERVAL_SECONDS and HOST_MIN_SPACING_MS.
• INDEX_ENABLED, INDEX_CHECK_INTERVAL_SECONDS, INDEX_BOARDS_FILE and INDEX_MAX_PROMOTED: the board directory that search watches are matched against. With INDEX_ENABLED=false Watchtower fetches only boards someone watches by URL.
• The caps: MAX_WATCHES_PER_CLIENT, MAX_RESOURCES_PER_HOST and MAX_ACTIVE_RESOURCES.
• WATCH_TTL_DAYS and the retention settings.
• TRUST_PROXY: set it behind a load balancer so rate limits see real client IPs.
• ALLOW_PRIVATE_NETWORKS turns SSRF protection off and exists only for tests and the demo.

Development

npm run typecheck
npm test                                  # unit tests; integration tests need a database:
TEST_DATABASE_URL=postgres://postgres:postgres@localhost:5432/watchtower_test npm test

The integration suite drops and recreates the public schema of TEST_DATABASE_URL, so point it at a throwaway database.

The suite has three parts:

• Unit tests.
• Property-based fuzz tests (fast-check) over the HTML, JSON-LD and robots parsers and the job diff invariants.
• Source tests for the Workday, iCIMS, Oracle Recruiting, Eightfold, SuccessFactors, Apple, Google, Amazon and Microsoft parsers and pagination, and the remote/seniority classifier.
• Search tests for the query reader, the pay and experience parsers, whole-word matching, the filters and board discovery.
• Integration tests covering REST, MCP, sharing, filters, search watches and the board directory, batch creation, page churn, NO_JOB_DATA, host leases, concurrent checks, caps, expiry and retention, webhooks, rate limits and metrics.

CI (.github/workflows/ci.yml) runs typecheck, all tests against a Postgres service, the build and the demo. It also builds the Docker image and smoke-tests it: migrations, /health, /metrics, the homepage, and the non-root user.

src/
  server.ts, app.ts         entrypoint; Fastify app (site, REST, MCP)
  config.ts, db.ts          env config; pg pool + migration runner
  security/ssrf.ts          URL validation + connect-time DNS guard
  fetch/                    safeFetch (redirects, limits, decompression), robots.txt
  extract/                  job-board adapters (incl. Workday, iCIMS, Oracle, Eightfold, SuccessFactors, Apple, Google, Amazon, Microsoft), JobPosting JSON-LD, remote/seniority
                            classifier, pay/experience parsing, term matching, job diff (identity, pairing)
  search/                   plain-language query reader; the built-in board directory; board discovery
                            (careers-page links, name guesses)
  services/                 clients, watches (board and search), checker, scheduler (+ maintenance), board
                            directory sync, host leases, rate limits, webhooks, metrics
  mcp/server.ts             MCP tools
  web/site.ts               homepage, llms.txt, well-known metadata
migrations/                 SQL migrations
ops/alerts.yml              example Prometheus alert rules
scripts/demo.ts             end-to-end demo
scripts/discover-boards.ts  grow the directory from a list of companies

Deliberately out of scope

• Covering every job board on the internet, or employers outside tech. A search watch sees the directory and the boards people have watched. Discovery is an operator step from a company list, not a crawler, and aggregators such as LinkedIn or Indeed are not read.
• Filtering to technical roles. "Tech jobs" means jobs at tech companies; a sales role at one is reported if the filters match it.
• Reading each SmartRecruiters, Workday or iCIMS job page for its description, so those jobs carry no pay or experience.
• A maximum-salary filter, currency conversion, and understanding a query with a language model.
• Billing, accounts and dashboards.
• JavaScript rendering. Careers pages that aren't on a supported platform are only readable if their server-rendered HTML carries JobPosting JSON-LD.
• General page, feed or event monitoring. Watchtower used to do these; it now does job boards only (migration 003_jobs_only.sql retires existing page and event watches).
• Pagination beyond the first 100 SmartRecruiters postings or the newest 200 Workday postings per board.
• Reading each iCIMS job page for details; only the newest ~50 jobs per portal get a real title and location.
• Reading Google's job search or job pages, which its robots.txt disallows; Google jobs are known only by their sitemap entry.
• Reading more than the newest 300 Apple, Amazon or Oracle Recruiting roles, the newest 200 SuccessFactors roles, or the newest 50 Eightfold (incl. Microsoft) roles, per check.
• SuccessFactors sites run on the company's own domain, so only the hosts listed in src/extract/successfactors.ts are read as SuccessFactors; any other one is read through its JobPosting markup, if it has any.