Backlinks

Module id: backlinks (frozen) · Mounted at: /api/modules/backlinks · Empty settings row (the shell-module convention; no switches — the one per-workspace pause is agent.paused, held by the scheduler chokepoint). This module is the reference implementation of the module contract (fullest manifest: queries + agentTools + cron + routes).

Purpose

Workspace backlink/referring-domain authority profile from DataForSEO: authority snapshots over time plus per-page link counts for every URL on the workspace's domain that earned links — including retired URLs (old paths that now redirect, pages that 404), because the links are real whether or not the page still is. Feeds the agent's link-safety reasoning (don't prune/301 pages that earned links; DO redirect gone URLs that still earn them).

Features

Weekly sync (runBacklinksSync, in-flight dedupe), three parts:

1. Summary + bulk new/lost → one backlink_snapshots row/day (rank, backlinks, referring/main domains, 30-day new/lost churn, spam score, broken pages, breakdown jsonb: TLDs/countries/placements/platforms/attributes).
2. domain_pages (backlinksDomainPages, since 2026-08-16) — every page on the domain with ≥1 backlink, strongest first, paginated 1000/call up to 1000 pages, subdomains in, internal links out — → page_backlinks upsert. This is the link graph's side, NOT the site catalog's: it replaced bulk_pages_summary over own URLs (articles + site_pages), which by construction could never see a URL that no longer exists on the site — a re-onboarded workspace lost every old-path row that still earned links (luniq.io: 2 stored vs 15 known). Each item carries the vendor's last crawl verdict for that URL — status_code + redirect location — stored as http_status / redirect_url (additive db/backlinks-linked-page-state-migration.sql); the merged root row keeps neither (the base URL always redirects somewhere — www↔bare, locale home — that is not a retired page). A workspace whose base URL carries a path (two workspaces sharing one domain) keeps only pages under that path (pageScope over the spine's isUnderBase); a root-hosted workspace keeps the whole domain. Zero-link pages age out as before (kept one sync zeroed, deleted on the second consecutive zero — now "absent from the read" instead of "checked and zero"), www+bare root variants merged into canonical. An INCOMPLETE read (a page failed) skips the whole reconcile and keeps last week's rows: absence from a complete read means "zero links", and reconciling against a hole would record a mass link loss that never happened.
3. Backlinks list (one-per-domain grouping + raw union, top 100 by rank) → referring_domains replaced wholesale.

Routes

Method	Path	Purpose
GET	/:workspaceId/overview	Summary (+deltas), trend, top linked pages (≤200), referring domains
GET	/:workspaceId/pages?limit=50	Linked pages, strongest first
POST	/:workspaceId/sync	Manual refresh (joins any in-flight run)

Crons

• Weekly sync — 0 6 * * 1 (Mondays 06:00, jittered) → runBacklinksSync, gated by the scheduler's fire-time automation gate — setup complete + agent not paused, decided once for every workspace cron (see crons reference); the scheduler body re-checks nothing.
• Operator trigger backlinksSync ("Backlinks sync (profile refresh)").

Agent tools & skills

• Tool get_backlink_profile — stored profile (rank/authority 0–1000, referring domains, new/lost churn, spam score, trend, own linked pages). Since 2026-08-22 the profile also returns brokenTargets (getBrokenLinkedPages: every page_backlinks row with http_status ≥ 400, strongest first, never a sample) — the authoritative broken-pages read; the tool description tells the agent to judge broken-link work from that named list only, because summary.broken_pages is the vendor index's historical counter and can exceed it (Luniq: counter 13, link graph 0 — it counts pages whose live links are gone). Optional urls[] (≤50) for pre-refresh/prune/replace link-safety checks; optional referrers_limit (1–100 — any value now honored, ≤10 used to be silently ignored; 2026-08-03) widens topReferrers from the default 10 to the full stored referring-domain set with anchor texts + dofollow — the anchor-audit read (getReferringDomains was already camelCased and route-only). Widened 2026-08-05 (tool-coverage pass): top_pages_limit (1–200 — the full linked-page set the UI shows, was a hard 10; rows carry firstSeen/checkedAt), months (1–24 trend, was fixed 12), include_link_losses (the full per-page link-loss list off the prev_* columns — the recovery worklist beyond the signal's cap of 3), include_url_referrers (per passed URL, its actual referrers {domain, domainRank, urlFrom, anchor, dofollow, firstSeen} — "who links to /pricing"). Status values: link_magnet (≥5 domains), linked (≥1), none. Since 2026-08-16 every topPages row also carries pageState (live | redirect | gone | unknown), httpStatus, redirectUrl — derived in ONE place, classifyLinkedPageState(): the site's own catalog wins when it knows the URL (article/live page = live; a site_pages row with is_redirect/missing_since = redirect/gone), else the vendor's stored http_status (2xx/3xx/4xx+), else unknown. getTopLinkedPages resolves the redirect target too, so a redirect row's resourceType/resourceId deep-link to the page it now lands on while url/title still name the URL that earned the links. Zero API cost. Every embedded list carries its denominator in the truncation block (topPages, topReferrers, pages, urlReferrers, linkLosses, linkChanges → {total, returned, truncated}) — getBacklinkListTotals head-counts page_backlinks + referring_domains (no rows fetched; a failed count is null = unknown, never 0), and the urls list reports anything past the 50 cap as truncated instead of dropping it silently. 2026-08-23 (tool-coverage audit §5): truncation.topReferrers.total is summary.referringDomains (the index's count) with stored = the ≤100 rows the sync keeps — a 900-domain site used to read as "100, complete"; url_referrers_limit (default 25, max 100) replaces the silent per-URL 25 and getReferrersForUrls returns { total, referrers } per URL so truncation.urlReferrers can say when a page has more; include_link_changes returns gains AND losses (direction, signed deltaDomains) off the same prev_* read — getPageLinkLosses became getPageLinkChanges({ includeGains }), losses by default, the signal detector unchanged; referrer filters dofollow / min_domain_rank / anchor_contains apply in the DB (applyReferrerFilters, one place) to both topReferrers and per-URL referrers; page_state keeps only topPages rows in one state (derived after resolution, so the read spans the whole ≤200 store and windows after). Referrers carry pageTitle + firstSeen everywhere; the description names the full breakdown key set. Conventions (2026-08-23): never-synced → { empty: true, reason 'never_synced', message }; every truncation.* section is the shared { total, offset, returned, truncated } envelope (core/utils/tool-envelope.js#listEnvelope); the handler is wrapped in guardRead so a failed read returns { error }.

Data

Writes backlink_snapshots, page_backlinks (incl. http_status, redirect_url), referring_domains; reads blog_articles, site_pages (title + is_redirect/redirect_target_url/missing_since for the linked-page state), workspaces. The sync no longer reads the catalog to decide WHAT to check — the vendor's domain-wide page list is the source; the catalog only resolves rows back to resources on read.

Key files

index.js (manifest), sync.js (the one write path), scheduler.js, routes.js, queries.js (the seam: getBacklinkProfile, summary/trend/referring-domain reads, list totals; re-exports the linked-page reads), linked-pages.js (page_backlinks reads — getTopLinkedPages, getPageBacklinks, getPageLinkChanges, getBrokenLinkedPages, the URL identity + state classifiers; split out 2026-08-23 when queries.js passed 400 lines), agent-tools.js.

Signals (Autopilot v2)

The manifest declares mechanical signal detectors (agent-signals.js, collected via the agent registry's signals hook, run by the nightly sweep): weekly referring-domain LOSSES — since step 6 (2026-09-01) minted ONLY from the sync's NAMED diff (classifyLostReferringDomains over backlink_snapshots.lost_domains: the sync reads the standing referring-domains rows before the wholesale replace and stores up to 25 lost rows with domain/rank/url_from/anchor/dofollow; null = no diff recorded yet, distinct from an empty diff). A lost domain below rank 50 mints nothing (churn), and a bare count delta with no qualifying named domain mints nothing either. The gains card retired with step 5 — good news reads on the module page, never as work. Fingerprinted per snapshot date; and page-level lost links (Signals v4 D3, classifyPageLinkLosses() over getPageLinkChanges — losses only by default): the weekly sync now SHIFTS each page_backlinks row's counts into prev_* columns before writing the new ones (additive migration db/signals-outcomes-migration.sql), so "since the last check" is a plain column diff — a partial drop needs ≥2 previously-referring domains, a total wipe always flags (the sync used to silently DELETE a page that hit zero; zero-link pages are now kept one extra sync and removed only after two consecutive zeros), warning severity on wipes/≥2 lost, cap 3, per-URL fingerprints, no how. Money-page targeting (agency pass §2, the D3 refinement): the detector looks up which losing URLs the site catalog classifies as commercial (getCommercialPageTypes — service/product/pricing/collection/category/contact, trailing-slash tolerant) and those surface FIRST, always at warning severity, with the page type named in title + evidence; a failed catalog read degrades to plain loss ordering, never a failed detector. Pure and asserted in backend/scripts/test-autopilot-safety.mjs.

Links follow redirects (2026-08-17)

getPageBacklinks(urls) matched exact URLs, so a live page whose earned links still point at its OLD path (an old /resources/blog/… URL that now 30x-es to /hub/articles/…) reported referringDomains: 0 — and the link-safety gate the agent applies before any delete/prune was blind exactly where it matters (the 2026-08-17 Luniq re-audit marked eight linked pages delete). The query now selects rows whose url OR redirect_url is an asked URL and folds the redirecting rows into the live URL's totals; the per-URL result carries viaRedirectFrom: [{ url, referringDomains }] so the fold is transparent. agent-tools.js says it in one sentence ("just an old redirect path" never means "no link").