Pages

Module id: pages · Mounted at: /api/modules/pages · Owns an empty settings row (settingsSchema: {} — the shell-module convention; the one per-workspace pause is agent.paused, held by the scheduler chokepoint).

Purpose

Pages is the workspace's site-pages surface: the Content → Pages list/detail UI, the discovery/refresh triggers used by the onboarding wizard and Settings, and the nightly site-refresh cron. The crawl/persist/select/enrich logic lives in the spine (workspace/site-context/); this module is its transport + scheduling.

Site context is workspace foundation — the agent, health, backlinks, and the writer all read it — which is why its freshness was decoupled from the content engine: pausing Collections no longer freezes the catalog.

Features

• Site discovery: sitemap crawl → provisional/preview selection → persist → final key-page selection, with a two-phase fast path so a usable key set exists in seconds on 100k-page sites. The wizard starts it fire-and-forget from its first form step (no review step since 2026-08-28) and /setup waits on the run.
• Full site refresh (discovery + key selection + enrichment + GSC URL inspection) via runSiteRefresh, scheduled and on-demand.
• Live run-progress status for the wizard's processing screen / Settings (site_context_runs).
• Untrack / re-track a page (the list's tracking PATCH). Manual page add (POST /page, addManualPage, the source='manual' key pin) was deleted 2026-08-28 with the wizard's Key-pages step: discovery is the one way a page enters the catalog.
• Site-pages list (one SQL pass via the list_site_pages RPC, scales to 100k pages) with GSC + pixel context, a lightweight picker, and a rich detail view (identity + on-page HTML facts + snapshot history + pixel audience).

Routes

Method	Path	Purpose
POST	/:workspaceId/discover	202 + background sitemap discovery + selection
POST	/:workspaceId/refresh	202 + background full runSiteRefresh
GET	/:workspaceId/run	Latest site_context_runs row (live progress)
GET	/:workspaceId/picker	Bounded, key-first list for pickers (tier=key&limit=1 is the wizard's "does a key set exist" probe)
GET	/:workspaceId/resolve	Paths → title + destination, across BOTH catalogs (see below)
GET	/:workspaceId	Paginated list + status counts + snapshot join + pixel rollup
GET	/:workspaceId/:sitePageId	Detail: identity + on-page facts + 30 snapshots + 28d pixel aggregation
PATCH	/:workspaceId/:sitePageId/tracking	Toggle is_tracked

Crons

• Nightly site refresh — '0 3 * * *' (03:00 local, per-workspace jittered via staggerCron; weekly on Sundays until 2026-08-18 — people work on their sites daily, and enrichment now reuses type/summary/embedding for key pages whose body hash is unchanged, so a quiet night costs no AI), gated by the scheduler's fire-time automation gate — setup complete + agent not paused, decided once for every workspace cron (see crons reference); the scheduler body only re-reads the profile settings it hands to runSiteRefresh.
• Operator trigger siteRefresh ("Site catalog refresh") in the internal dashboard's Run-now dropdown — same runSiteRefresh function.

Data

Reads/writes site_pages, site_page_snapshots, site_context_runs, pixel_pageviews; reads workspaces, workspace_collections, and the '1a-blog-publisher' settings row (languages, gscSiteUrl). RPCs: list_site_pages, latest_page_snapshots_for, pixel_pages_summary.

Key files

File	Role
index.js	Manifest (id: 'pages', empty schema, schedule, operatorCrons, routes)
scheduler.js	Nightly cron registration + siteRefreshTrigger operator descriptor
routes/discovery.js	Discovery/refresh/run-status write surface
routes/site-pages.js	List/picker/detail/tracking read surface
workspace/page-identity.js	Spine. resolvePaths() — the resolution below, shared by this route and the analytics agent tool

Path resolution (GET /:ws/resolve)

Paths → the thing that owns them, for surfaces holding a bare path. Two catalogs, because a site has two kinds of page and they live apart: site_pages (crawled) and blog_articles (published content). Resolving only the first was the obvious version and it was wrong for content sites — 47 of 50 landing paths matched on one workspace and 9 of 50 on another, and every miss was an article.

Articles win where both match (the editor is the more useful destination) and are matched on the PATH of their stored url, in JS, because they carry a full URL rather than a path. Bounded — the largest workspace has ~300 — and if that becomes thousands, store a path column rather than growing the loop.

Analytics is the caller. Resolving here rather than joining these tables into the analytics read keeps that read generic: a join would special-case one dimension out of nine and pull two other modules' tables into it. A path that matches nothing simply does not come back, and the caller keeps rendering the path — a missing catalog entry must never hide a real page's numbers.

The logic lives in the spine (workspace/page-identity.js), not in this route, because it has two callers that must agree: the route the Analytics UI calls, and the analytics agent tool. Two copies would drift, and the symptom would be the agent naming a page differently from the screen the user is reading it on — which reads as the agent being wrong.

Detail response addition (2026-08-09): the site-page detail now returns the stored URL-Inspection diagnostics — gsc_google_canonical / gsc_user_canonical / gsc_inspection_link plus crawl forensics (gsc_crawled_as, gsc_page_fetch_state, gsc_robots_txt_state, gsc_referring_urls, gsc_sitemaps) — written by the GSC module's inspection sync.

Index status made honest (2026-08-23): the detail response now SHAPES gscGoogleCanonical / gscUserCanonical / gscInspectionLink into its payload — they were selected and then dropped, so the agent could explain a "not indexed" page (get_item_details has served them since P4) and the page itself could not. SitePageDetail.tsx renders an Index status block: the verdict, how old it is (gscLastInspectedAt as a relative age), Google's own coverage phrase, the canonical-elsewhere explanation, a Search Console deep link, and a Check now button posting to the GSC module's POST /recheck. The status pill also stopped collapsing three states into two — indexed === null now reads "Not checked yet" rather than "Not indexed", which had been claiming a verdict Google never gave.

list_site_pages v4 — noindex is its own status (2026-08-23): the list could only say Indexed / Not indexed, collapsing two opposite facts into one chip: Google declining to serve a page is a PROBLEM, a page asking not to be served (legal pages, on-site search, date archives) is a DECISION. Rows now carry is_noindex, statusCounts gains noindex, and the three states partition the tracked corpus — indexed and notIndexed both exclude noindex rows, and each filter arm matches its own count, so a chip never returns more rows than its number. The rule mirrors deriveNoindex (JS) exactly: a noindex/none token in meta_robots, OR gsc_indexing_state in BLOCKED_BY_META_TAG/BLOCKED_BY_HTTP_HEADER. Both sources are load-bearing (Neue World's /work/ticket-chain has no meta_robots and Google says blocked; Ledoux's date archives have the tag and no GSC verdict). Token-not-substring because max-image-preview:none is an ordinary directive on an indexable page while robots: none is a real noindex synonym. BLOCKED_BY_ROBOTS_TXT is excluded — that blocks crawling, a different problem. ⚠️ The coalesce(..., false) is load-bearing: gsc_indexing_state in (...) is NULL for the 10,234 rows where the column is NULL and false OR NULL is NULL, so without it not is_noindex silently drops most of the corpus from the notIndexed count. Migration: backend/db/list-site-pages-noindex-migration.sql; JS half pinned in test-gsc-tools.mjs.

list_site_pages v3 (2026-08-09): is_indexed now applies the who-is-newer rule in SQL — a false Inspection verdict from inside the 28-day window wins over trailing impressions unless 7-day earnings postdate a >7-day-old verdict. Keeps the list chips, statusCounts and the detail page (JS deriveIndexed) computing one answer. Migration: backend/db/list-site-pages-fresh-deindex-migration.sql.

list_site_pages v4 (2026-09-01) — the list sorts in SQL: two new args, p_sort (impressions | clicks | position, over the filtered CTE's imp28 / cl28 / pos28) and p_dir, applied in the order by that already carries the LIMIT. That placement is the whole point. The list is server-paginated at 25 and the largest live workspace holds 10,159 pages — 407 pages of results — so a client-side sort would reorder the visible slice while presenting itself as a ranking of the site; 12 of 17 workspaces exceed one page, so this is the normal case, not the edge. Both args default such that every existing caller gets the old locale, url ordering byte-for-byte. Three rules inside: NULLS LAST on every direction (a page with no GSC row is missing, not zero, and belongs at the bottom whichever way the column points), locale, url survives as the final tiebreaker (thousands of pages tie at 0 impressions; without it the tie order varies per request and a row can appear on two pages or none), and an unknown sort key degrades to the default order instead of erroring, so a stale bookmark cannot 500 the list. position defaults ASC because rank 1 is the good end. The old 9-arg signature is dropped in the same migration — left in place it would linger as an overload and make every call ambiguous. Visitors is deliberately NOT sortable (SORTABLE in routes/site-pages.js names the three): it is aggregated from pixel_pageviews in the handler after the RPC has paginated, with the humanity proof, so ordering by it would only ever reorder one screen; the UI gives that column no control rather than one that quietly lies. get_content_inventory already sorts on all four for the agent, visitors included, so the tool surface was already at parity and needed no change.

The data surface (S10, 2026-08-12)

The module owns queries.js: the spine's site-context search reads re-exported (searchPages, searchPagesLoose, getPageCorpusFacts, getPageIndexStatusCounts, getInboundLinks, CATALOG_PAGE_TYPES) plus the module's own catalog reads/writes (listPagePicker, listSitePages — the list RPC, getSitePage, getPageSnapshotHistory, setPageTracking — addManualPage/POST /page and untrackPageByUrl/DELETE /page were deleted 2026-08-28 with the wizard's Key-pages step; the tracking PATCH is the one untrack path and discovery the one entry path). Routes are thin on it; search_site_pages lives here now (agent-tools.js, moved from agent-core) and imports the module's own queries — one sanctioned surface for routes, the agent tool, and the S11 cross-module detectors. The module is allowlisted in the detector-coverage gate until its own detector lands.

search_site_pages collects the two passes' matches up to a 200-row ceiling and windows them with offset/limit (the old code sliced at the requested limit and reported nothing about what it dropped): the response carries total, offset, returned, truncated, and scan_truncated when the ceiling itself was reached — the list-tool contract, so a 12-row window is never read as the whole catalog. The frozen search_site_pages RPC is unchanged; the ceiling and the windowing live in the tool.

Link equity (S11, 2026-08-12)

The module's first detector: agent-signals.js link_equity reads the persisted internal link graph (site_page_links, written by enrichment — see Site context) against the pages' own snapshots: pages:link-starved:<path> (key page, real impressions, ≤1 inbound link) and pages:equity-pool:<path> (≥8 inbound links into a page earning nothing). getInboundLinkCounts on queries.js is the read; an unrecorded graph is unconfigured, never "nothing links anywhere". The detector-coverage allowlist entry is gone — the gate enforces this module now.

Tool parity pass (2026-08-23, plans/tool-coverage-audit-2026-08-23.md §4)

• Crawl forensics reach the surface. gsc_crawled_as / gsc_page_fetch_state / gsc_robots_txt_state were written by the inspection sync and selected by getSitePage but returned by nothing. The detail route now serves them (gscCrawledAs, gscPageFetchState, gscRobotsTxtState — the UI can render them later) and get_item_details.indexation carries crawled_as / page_fetch_state / robots_txt_state, with the fetch state described as the CONCRETE "Google could not fetch it" cause (SOFT_404, BLOCKED_ROBOTS_TXT, NOT_FOUND, SERVER_ERROR …). referring_urls / in_sitemaps stay capped (5 / 3) but now carry *_total + *_truncated.
• Inbound link EDGES. getInboundLinks(workspaceId, path, { limit }) lives in the spine (workspace/site-context/link-graph.js — the spine writes the graph, so it owns the edge read) and is re-exported here beside getInboundLinkCounts. get_item_details gains an opt-in links section for pages: inbound[{ source_page_id, path, url, title, page_type, locale }], inbound_total (edges), inbound_truncated (50-row cap), and a note that the graph is the static-HTML crawl of enriched pages — a lower bound, never proof of an orphan.
• The derived index verdict, whole-corpus. getPageIndexStatusCounts (spine, re-exported here) reads the list_site_pages RPC's statusCounts + untrackedCount with p_collection 'all' — the SAME SQL the Pages list filter and the Performance coverage strip use (its is_indexed / is_noindex rules mirror deriveIndexed / deriveNoindex). getPageCorpusFacts carries it as index_status { indexed, not_indexed, noindex, untracked }; the raw gsc_indexed_count / gsc_not_indexed_count stay, labelled raw.
• get_content_inventory page path (collections module, inventory-builders.js): tracked: 'tracked'|'untracked'|'all' (default tracked — the Pages list's own hidden set is reachable on request), collection (pages attributed to one collection via collection_id), locale: 'all', range 7d/28d/90d for the pixel block (keys suffixed by the window, pixel.window — the blog convention), and the list's own columns on every row (path, source, discovered_at, last_enriched_at, inbound_links, word_count, collection_id, is_tracked:false). getPageSnapshotHistory selects the 90d columns too.
• search_site_pages types accepts any string (bounded, trimmed) — page_type is free-form in the catalog, and the old enum refused types a site actually carried. The known list stays in the description as examples; page_corpus.by_page_type lists what a site really has.

read_site_pages — the batch reader (2026-08-17)

Only key-tier pages carry a stored body; the tail (where a site's hundreds of imported blog posts live) has titles and numbers only, and fetch_url is capped at 12/turn — so a corpus audit could not read what it judged (289 of 291 Luniq pages: title-judged). agent-tools/read-site-pages.js reads up to 40 pages per call through the spine's readPagesText (fetch + parse, no AI): title, h1, headings, word_count, published/modified dates, alternate_urls, and a body excerpt that starts at the article's H1 rather than the site chrome. What is read is KEPT: persistLightEnrichment writes the parse fields onto never-enriched rows (last_enriched_at IS NULL, clean 2xx only), so later reads come from the store, get_item_details gains the body, and locale pairs / dates become mechanical. Registered on the manifest; queries.js re-exports the spine fetcher and adds getPageUrlsByIds.