Appearance
Statistics (Orbit Pixel)
Module id: orbit-pixel (frozen) · Authenticated routes: /api/modules/orbit-pixel · Public ingestion: /pixel/* (pre-CORS, host-rewritten from pixel.orbit.luniq.io) · Empty settings row.
Purpose
First-party (cookieless-capable) visitor analytics for content sites: tracks pageviews/engagement/journeys, attributes conversions (leads) back to the articles that earned them, distinguishes real humans from AI crawlers, and surfaces which AI tools index your content. Manual-first conversion goals define what counts as a lead — no goal, no counted lead (hard zero).
Features
- Client pixel (
static/orbit.js, served at/pixel/orbit.js): resolves workspace fromdata-workspace/?w=/window.ORBIT_WORKSPACE_ID; observes automation instead of skipping it (2026-08-10):navigator.webdriver === trueno longer bails out — the pageview carriesau:1andclassifyRequestroutes the visit toautomation_tool(bots lane), so declared automation is measured instead of unmeasurable-by-construction; 4-tier identity (cookie_orbit_v/ localStorage / sessionStorage / server fingerprint fallback) — with verified write-back (2026-08-07): after writing, the pixel reads both tiers back and stampsst: {c,l}on every pageview beacon, so the server can measure how much of the audience has ephemeral identity (in-app webviews ≈44% of fleet traffic wipe storage; a failed write means a fresh "visitor" per visit). The fingerprint fallback is LIVE (2026-08-10): when no tier survives the read-back the client sends no id at all (an invented one fragmented a person into a visitor per pageview —computeFingerprinthad executed 0 times in 65k visitors because the fallback branch was unreachable), and the server resolves identity through the trust ladder inservices/fingerprint.js(resolveVisitorIdentity, asserted intest-pixel-edge-trust.mjs): client id → edge-verified requests fingerprint on Cloudflare's connecting-ip (the real TCP-peer address; SHA256 of ip+UA+language, rotates daily — stitches a storage-blocked person's pageviews/sessions/leads within a day, never across days) → unverified paths get a uniquev_srv_id per request, because their candidate IPs are either a shared infrastructure hop (fingerprinting would merge strangers into a mega-visitor) or a forgeable header (a caller could choose someone else's visitor_id, which drives lead attribution). The engagement RPC takes its authoritative visitor from the pageview row, not the beacon (pixel_ingest_engagement_row_visitor), so a fingerprint rotating at midnight can't detach an engagement from its pageview. Fires pageview, engagement (time-on-page, max-scroll, interacted), and lead beacons viasendBeacon(text/plain, no preflight); engagement persists on visibility-hide/pagehide and via a 15s visible-tab heartbeat (2026-08-07: pagehide/beforeunload frequently never fire on mobile — 31.8% of pageviews carried zero engaged time; the heartbeat sends the non-finalizing snapshot and the MAX-merge RPC makes repeats idempotent, so the humanity bar and avg-time metrics stop under-reading mobile). Engagement events (Signals v4 X1): trusted clicks on links/buttons (kindinternal= same-host pathname,outbound= host only,action= trimmed button label ≤60 chars;tel:/mailto:stay with the conversion path) andform_start(first field focus,deriveFormIdreused) — EVIDENCE, never conversions; batched to/v1/events(flush at 10, on hide/pagehide/SPA-nav), per-pageview cap 25 + per-target de-dupe, no query strings and no field contents ever. - Ingestion (
services/ingest.js, all 204 fire-and-forget):- Pageview — UA classification (hard-block bot → dropped; AI crawler → stored with
crawler_name; human → stored), traffic-source classification, geo country (getVisitorIp→lookupCountry, see Geo below), device/browser parse, URL→article match, atomicpixel_ingest_pageviewRPC; evaluates URL goals — people only (2026-08-20):human+human_agentclasses. Before, only AI crawlers were excluded, so search crawlers/automation/junk walking a thank-you URL inserted counted conversions — and visitor-dedup can't contain that, because unverified traffic gets a fresh visitor id per request. The lead-beacon path keeps its own bot gates; this is the URL path's equivalent. - Engagement —
pixel_ingest_engagementRPC (MAX-merge; qualified thresholds 10s / 25% scroll, in SQL). - Lead — two bot-lead filters (self-reported engagement snapshot; session corroboration — ≥2 pageviews with zero engagement = headless signature), goal resolution, per-goal dedup,
recordConversion(first/last/linear article attribution, session-source attribution, visitorlead_countbump).
- Pageview — UA classification (hard-block bot → dropped; AI crawler → stored with
- Human classification (
services/device.js):HARD_BLOCK_RE(Googlebot, Ahrefs, Semrush, social previewers…),AUTOMATION_RE(self-identifying automation: HeadlessChrome, Chrome-Lighthouse, Puppeteer/Playwright/Selenium, python-requests/curl/wget and other raw HTTP clients — dropped at ingest), and anAI_CRAWLERStable (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, CCBot, Google-Extended, Bytespider, Amazonbot…). UA-spoofing headless traffic that slips past all three is filtered at read time by the visitor-level humanity rule — see its own section below. - Traffic classification (
services/classifier.js, asserted intest-autopilot-safety.mjs): 4-signal (UTM → referrer host → Copilot landing-URL form code → question-URL heuristic) with confidence; AI attribution reviewed 2026-08-05: ~30 known assistant hosts (chatgpt/claude/perplexity/gemini/copilot/grok/deepseek/mistral/meta/kimi/qwen/…),android-app://assistant app packages,form=MG0AV3catches Copilot hiding behind bing.com, and an unknown-assistant catch-all — an AI-shaped referrer host (chat./copilot/assistant/ask/gpt/chatbot) not in the known map is classifiedsource:'ai'with its ROOT DOMAIN asai_tool(confidencelikely), so new assistants surface by name instead of vanishing into "referral" (industry baseline: ~70% of AI visits arrive with no referrer at all — what has a signal must never be wasted). Non-AI buckets: organic/social/paid/email/referral/internal/direct. - Conversion goals (
services/goals.js): manual-first HARD-ZERO — only a matched goal counts; otherwisecandidate(worklist) orignored. Typesform/url/click/custom; dedupsession/visitor/always— defaultvisitor(2026-08-02: the oldsessiondefault let one person recount across sessions; measured live, a single visitor produced 9 counted lead-magnet leads in two days = 37% of the workspace's reported leads. A lead is a person, not a submission; existingsessiongoals were flipped and the duplicate leads re-markedignored). Wildcard URL matching; retroactive reclassify/backfill; form-candidate discovery. Value per conversion is REQUIRED at goal setup (2026-08-20) —createGoalthrows without a positivevalueAmountandupdateGoalcan change but never clear it, because per-goal value is the only way money enters the reports now that the workspace-wideavgLeadValueis retired; auto-provisioned explicit-key goals (ensureGoal) stay the born-valueless exception, and the edit dialog demands a value on their first edit. Retro changes re-sync the derived layer (2026-08-20): pixel_sessions and the daily rollup only learn about conversions at capture time (pixel_session_mark_converted), so claiming or demoting history — promote, URL backfill, goal delete — now stamps the goal'svalue_amounton claimed rows (where the capture carried none) and callsservices/resync.js#resyncConversionRollup(pixel_backfill_sessions+pixel_rollup_daily, idempotent, 400d, workspace-scoped; awaited but never fails the operator action). Incident: 4 retro-promoted Luniq leads visible on the Conversions tab, invisible on Overview — theconversion-rollup-driftinvariant inmodules/agent/data-integrity.jsnow detects both directions nightly and self-heals with the same resync. Detection → promote loop hardened (2026-08-20): iframe providers (HubSpot, Calendly, Cal.com — new:originator:'CAL'+bookingSuccessful*postMessage, both the deprecated V1 andbookingSuccessfulV2; the 60s/visitor dedup absorbs the double-fire —, Typeform, Tally, Jotform) emit STABLE identifiers; Tally's payload arrives as a JSON string and is now parsed (the object-only guard made its submissions silently invisible); both booking providers only notify the parent when embedded via their official snippet (a hand-rolled bare iframe stays silent — the redirect → URL-goal path covers those) — Calendly now sendscalendly:bookinginstead of the per-bookingpayload.event.uri, which had made every booking its own one-row candidate so a promoted goal never matched the next booking. Goals matchmatch.formIdsregardless of type: promotedtel:/mailto:candidates becomeclick-typed goals (correct panel label) that claim history and match future clicks; retro-claims stampconversion_typefrom the goal's type. The Add dialog offers exactly TWO manual options (thank-you page, code snippet — thedata-orbit-goalattribute is the snippet's variant, still auto-provisioning) and leads with what detection already covers. Drupal AJAX forms (2026-08-28): a Webform with "Use Ajax" submits through Drupal'sajax.js(button click →preventDefault→ jQuery POST to?ajax_form=1), so the nativesubmitevent never fires and the capture-phase listener was blind — hrth.be had 72 contact pageviews,form_startrows and "Verzenden" clicks, zero leads, while its non-AJAX login form was captured.orbit.jsnow binds jQuery's globalajaxSuccess(deferred until jQuery, a Drupal footer script, exists) and tracks a lead when the request isajax_form=1for awebform_submission_*form and the command list carries aredirect, awebform-confirmation, or amessages--statuswith noform-item--error/messages--error; the identifier is the form's DOM id (Drupal'sform_idwith hyphens), the same oneform_startand a native submit report, so promotion matches either path. File-upload, wizard-page and draft responses re-render the form without a confirmation and are ignored. No change on the site. - Dashboards: overview, per-article stats + leads, articles summary (cached 60s), server-paginated visitor journeys (outcome filter), AI/LLM crawler activity, install status.
Who is browsing — the agent registry and three lanes (W3)
Ingest stopped dropping traffic. The old path returned early for anything matching HARD_BLOCK_RE, so Googlebot, SEO tools and every automation toolkit were never stored — bot volume was unmeasurable by construction. It also discarded agentic browsers (Perplexity Comet, ChatGPT agent mode), which are people steering software and whose conversions are real leads. Everything is stored now, classified, and filtered by lane at read time.
Rules became data. Three hardcoded regexes in services/device.js are now agent_registry (40 curated rows) — a new assistant is a ROW, not a deploy. Bulk import from Matomo's bots.yml comes later; the table is shaped for it (source='matomo'), and thousands of patterns need care on a hot path.
Seven classes, three product lanes (services/agent-class.js owns the grouping — pure, 19 unit tests):
| Lane | Classes | Meaning |
|---|---|---|
humans | human | the audience; every classic metric counts this lane only |
ai | human_agent · assistant_fetch · crawler_ai | a person mediated by AI · an assistant fetching because someone just asked · AI crawlers reading you |
bots | crawler_search · automation_tool · junk | measured and charted, excluded from human metrics |
self | self | our own health-module crawls — hidden from every surface |
Googlebot is in bots despite feeds_ai. The same crawl feeds Search, AI Overviews and Gemini grounding — there is no separate Gemini crawler, and Google-Extended is a robots.txt policy token, never a user agent. But its volume would drown the genuinely new signal and Search Console already tells the search story; the AI-side outcome is measured on AI Visibility.
A user agent is a CLAIM, and it gets checked (services/agent-verify.js, W3b):
| Method | How | Used by |
|---|---|---|
ip_range | the visit came from the operator's published range file | OpenAI (GPTBot 21 prefixes, ChatGPT-User 258, OAI-SearchBot 35), Perplexity |
rdns | forward-confirmed reverse DNS — address → hostname → back to the same address | Anthropic, Google, Bing, Apple (publish no range file) |
signature | Web Bot Auth, validated at classification time | OpenAI + Google agents |
edge | our CDN's own independent verdict | anything Cloudflare recognises |
Forward confirmation is the load-bearing half: a reverse lookup alone proves nothing, because whoever controls an address controls its PTR record. Only the operator's own DNS satisfies both directions. It is the check Google documents for verifying Googlebot.
Two constraints pull against each other, and the design resolves them by making every check either instant or absent: ingest is a hot path (no DNS round trip, no blocking fetch), and a wrong verdict is worse than none (calling a real crawler junk over our own network hiccup would delete real data). So ranges live in memory and the check is an integer compare; rDNS answers from a cache and a miss returns unknown while queueing a lookup, so the next visit from that address has a verdict. unknown never downgrades anyone — only a definite fail turns a visit into junk, keeping the claim visible rather than erasing it.
Verified live against the real published data: a genuine OpenAI address → pass, an impostor claiming GPTBot → fail, real Googlebot via rDNS → pass, first (uncached) call → unknown. A publisher outage keeps the last good list rather than starting to fail verifications.
⚠️ ua_pattern is compiled by JavaScript, not Postgres. JS does not reject Postgres regex syntax — it reads \m as a literal "m" — so the seeded Comet pattern silently stopped matching and filed every agentic visit as an ordinary human. No error, no symptom, wrong lane. Caught by a unit test, now asserted permanently by pixel.agent-patterns-compile (FAIL).
⚠️ Coverage, and it must be stated wherever lanes are shown: a JS pixel only sees agents that execute JavaScript. Bulk training crawlers mostly do not, so GPTBot and ClaudeBot are structurally invisible here — the two that appear (meta-externalagent, bytespider) are the browser-based ones. That lane needs the customer's own logs or CDN; observed_via is reserved for it. Never read a low crawler count as "AI is not reading this site".
Surfaced by getLaneTrend → GET /:ws/lanes and get_traffic_lanes. getSessionReport gained lane (default humans) before crawler sessions were backfilled, so non-human traffic can never leak into a landing-page number.
The rollup and the one read (pixel_daily + pixel_timeseries, W4)
Sessions answer any single question cheaply. They do not survive the question the product is for — "sessions by source over 90 days, then by country, then filtered to mobile" is a scan of every session row per chart, per dimension, per page load. pixel_daily pre-aggregates to one row per (workspace, local day, lane, dimension, value), so a chart is a bounded indexed read.
Lane is part of the key, not a dimension — every surface is lane-scoped first, so folding it in keeps the humans query narrow instead of making every read filter the same thing every time.
Additive vs not is the trap in every rollup. Counts sum; visitors_distinct does not (a visitor seen Monday and Tuesday is one person and two rows) — named so an accidental SUM() looks wrong at the call site. Rates are never stored, only numerator and denominator: a stored average can't be re-aggregated, and pixel_timeseries recomputes ratios from summed parts.
pixel_timeseries(metric, breakdown, lane, range) is the one read. Series always sum to the KPI (overflow folds into (other) rather than vanishing), missing days are zeros not gaps, and every response carries coverage — currently 68.6% on country, which is the geo hole stated rather than hidden behind a chart quietly answering from two thirds of the data.
The nightly job rides the module manifest (scheduler.js) and runs per workspace in its own timezone — a fleet-wide UTC job would rebuild a Brussels day two hours after it ended and a Los Angeles one before it had. It rewrites 72 hours, because engagement, conversions and rDNS verdicts all keep arriving after a session starts — 72 and not 48 because the window is measured from the 02:20 cron, so at 48h a session started between local midnight and the cron aged out one run early; pixel.rollup-integrity caught two settled days each frozen 2 pageviews short (2026-08-09).
It registers like every other module cron — jobs.push(cron.schedule(staggerCron(…), …, { timezone })) wrapped in makeCronGuard — on the scheduler's measure lane (ctx.cron.measure, 2026-08-16): kill-switch + active workspace only. It does NOT wait for setup to complete or for the agent to be un-paused, because it makes zero paid calls and its 72h window means a skipped night is a hole we cannot refill later — the pixel keeps collecting for any live workspace (ingest is a request path), and so does its rollup. Only the fleet kill-switch or an archived/suspended workspace stops it (a gap from those is backfilled by hand via pixel_rollup_daily's window parameters). The first version called ctx.cron(…) as a function; ctx.cron is the workspace-gated wrapper object, so every workspace threw cron is not a function at scheduler init, and the throw escaping the registry loop unscheduled every module registered after this one (backlinks, health, AI visibility, the whole autopilot) fleet-wide. Nothing was lost — the fix landed the same day, before the rollup's first fire — but it is why scripts/test-module-crons.mjs now executes every schedule() in npm run verify (see Architecture).
Four bugs were found before any of this was trusted — all four in db/pixel-daily-migration.sql: the prune deleting its own output on a slow run; sparse dimensions not summing to their own total; a bare DELETE that PostgREST rejects (worked in the editor, would have failed every cron); and the rewrite window slicing days — now() - 48h left the boundary day partly outside, so the upsert overwrote a complete day with a partial count. That last one was caught by pixel.rollup-integrity within minutes of the alarm existing, on 11 workspace-days, all undercounted, all the same date. Nightly it would have shaved the oldest day of every window forever. The function now selects which local days to rebuild and always rebuilds them whole.
The report surface — three reads, every panel (W5)
modules/orbit-pixel/reports.js is the whole read layer. It calls RPCs and shapes responses; it contains no arithmetic, because a fourth place for the numbers to live is a fourth place for them to disagree.
| Read | Answers | Source |
|---|---|---|
getTimeseries | shape over time — metric × breakdown × lane × window, dense daily series, optional previous-period series | pixel_daily |
getBreakdown | the period, ranked, with KPI totals and a per-row delta vs the previous period | pixel_daily |
getBreakdown ranks on eight measures, both directions (2026-09-01, three migrations in one day). pixel_breakdown gained p_sort beyond the original four summed measures (sessions/conversions/value/pageviews): bounce, engaged, conv_rate and delta are DERIVED, recomputed inside the ORDER BY from the sums in scope (delta needed the previous-period join moved one CTE earlier — it did not exist at ranking time). Then p_dir (literal asc/desc; which direction is NATURAL per measure is the frontend's vocabulary — bounce opens ascending). The ranking has to precede the top-N fold: sources ranked by sessions open with paid, by conversions with ai — a row not in the sessions top five at all; a client-side re-sort of returned rows can never surface what (other) already ate. A 30-session floor on the ratio sorts was built and REMOVED the same day: it demoted low-traffic rows invisibly, and on Byve (~300 sessions/28d) 4 of 7 source rows fell under it — "sorting bounce top down gives 50 and 40 below 39" (Leon). The sessions figure beside every rate is the visible guard the floor tried to be; NULLS LAST stays (absent data, not noise suppression). BREAKDOWN_SORTS in reports.js is the closed allowlist; unknown keys degrade to sessions. The UI half lives in analytics/use-reports.ts (useBreakdownSort per-dimension state, natural→flip→clear cycle) with an instant client-side mirror of the same ordering laws (sortBreakdownRows) so a click reorders on-screen rows immediately while the server's authoritative ranking settles in; swatch hues are pinned to the canonical sessions order so re-sorting never repaints them. | getRealtime | the last N minutes, all lanes — the live roster, the per-minute strip, and the two facts a quiet window needs | raw pixel_pageviews | | getLaneTrend | humans vs AI vs bots + per-agent verification state | pixel_lane_trend |
getRealtime is the one read that deliberately ignores both the rollup (nightly) and the session table (a session is only complete once the visitor stops) — both are behind by exactly the interval it answers.
The roster (2026-08-19). pixel_realtime returned a flat feed of the newest 50 PAGEVIEWS, so one person reading six pages was six rows scattered among everyone else's and no caller could render a person, a journey, or tell one busy visitor from six. It now also returns visitors: one entry per (visitor_id, session_id), carrying where they are now, a trail of up to three pages before it (consecutive repeats collapsed — a reload is not a step), pageviews, firstAt/lastAt and the identity taken from the LATEST hit, not the first. Capped at 40 with visitorsTotal + visitorsTruncated beside it. feed is unchanged and still there for callers whose question really is the order of events.
Also added: sources (the window's people by traffic source), and the pair a quiet window needs — lastVisit (most recent human hit within 30 days, bounded so a crawler-only site does not walk the index back through months) and last24h. Rolling 24 hours rather than "today" on purpose: a calendar day needs the workspace timezone, which means a parameter, which means a second signature for the drift baseline to pin and an old overload left alive.
The change was proved additive against production before it shipped — the same call minus the six new keys hashes identically to the pre-migration response (5cc6d07e…, OysterClamp, 30 min pinned to 2026-08-19 20:00Z). Migration: db/pixel-realtime-roster-migration.sql.
visitorsNow is not "now". It counts distinct people across the WHOLE window, so a 30-minute read gathers everyone in half an hour. Measured on that same production call: 31 against a visitorsLast5 of 5. The UI had it as the headline under the label "People here now" for months. Both fields keep their meaning and their names — the surface leads with visitorsLast5 and reports visitorsNow as window context — and the tool description says so explicitly, because the name is the trap.
Windows. services/reporting-window.js composes the workspace timezone (spine) with the pure calendar math (services/window.js); windowPair resolves a window and its comparison period on one call, so the two can never differ in length or overlap by a day. Presets and explicit from/to resolve to the same shape — a preset enum can only answer "the last N days", and "how did July go?" is a different question a 28-day preset answers wrong by three days. Malformed dates fall back to the preset rather than to an empty window: a bad URL shows the default report, not a zero.
One canonical dimension mapping. pixel_dim_value(dim, session) was extracted from pixel_rollup_daily and the rollup rewritten onto it, so the rollup and both reads bucket identically by construction. Proven by rebuilding the whole table: 45,274 rows before and after, row-by-row identical on every settled day (the only diff was the in-progress day, which gained traffic mid-rebuild). pixel_dims() is the allowlist — both reads RAISE on anything else, because a caller that accepts an unknown dimension buckets a whole workspace into (unknown) and draws a confident chart of nothing.
visitors is not additive, and that was a live wrong number. The series total summed daily distinct counts, so a returning visitor was counted once per day they returned — +9.6% on the busiest workspace (16,582 vs a true 15,125), +3.5% and +2.4% elsewhere. Always high, never absurd, worse the more loyal the audience: the profile of an error that survives review. Per-point values still come from the rollup; the window total is a real COUNT(DISTINCT) over the window's sessions, bucketed through the same helper so series keys match.
Two flags stop a correct number being drawn wrongly:
additive(pixel_metric_additive) —falsemeans the series may not be stacked. Visitors by source totals 15,125 while the series sum to 15,314, and both are right: one visitor arriving from Google and directly is one person in two rows. Stacked, the chart would draw 15,314 people under a KPI reading 15,125. Ratios arefalsefor the same reason. The frontend obeys the flag rather than keeping its own list, which would drift the first time a metric is added.coverage.kind(pixel_dim_kind) —universal(country, device, browser): every session should resolve, so a low share is a defect worth flagging.subset(campaign, ai_tool): 0.0% means 8 of 17,326 visits came from ChatGPT — the finding, not a fault, and showing it as a coverage warning would tell a customer their tracking is broken when it is working perfectly.total(source):directis a real value, so it can never be unknown.
Details and the production verification numbers: db/pixel-reports-migration.sql.
The Analytics page (pages/orbit/performance/analytics/, W5b)
Eight tabs over one window, at /performance/analytics — the Statistics surface since 2026-08-08. /performance/statistics redirects rather than 404s (a year of bookmarks and the sidebar entry people learned), Performance.tsx is Search Console only and lost its two-tab switcher, and components/PixelStatistics.tsx is deleted. Its own sub-page tree, not more tabs in Performance.tsx. The chassis is the Pages header (SitePages.tsx) — one line, text-base sm:text-lg, FilterSelect controls — so it retunes with every other list page rather than drifting; the design mockup was drawn standalone and its hero title is deliberately not used.
| Piece | Notes |
|---|---|
AnalyticsChart | THE chart. shadcn/Recharts, same primitives as Backlinks and Health. Every tab configures it; none forks it |
BreakdownTable | THE table. One component pointed at a different dimension per tab |
CountryMap | THE map panel — the canvas in two frames (inline 420px block, fullscreen dialog). Code-split; WorldMap + world-atlas ride in its chunk |
WorldMap | The canvas only: react-simple-maps + world-atlas, fills its box; ISO-numeric→alpha-2 resolved at build time into lib/country-geo-ids.ts |
MapSummary | The card that floats on the fullscreen map — stats, the ranking, coverage |
Notices | The beta banner, the went-quiet warning, the read-failure banner, the not-installed screen |
additive decides the chart's form, and the BACKEND decides additive. True → stacked areas that sum to the KPI; false → separate lines, never summed. Distinct visitors overlap across dimension values (one person from Google and direct is one visitor in two rows) and a ratio is not a sum. A "safe metrics" list kept in the frontend would drift the first time a metric is added, and the drift would read as a design choice rather than a bug.
Selection is a lens, not a filter of the table. Picking rows narrows the chart and the KPI strip; the table keeps every row, because the table is what you pick from. The picked totals come from their OWN read (filterValues), not from summing the picked rows — sessions would add up fine, distinct visitors would not. Selection is keyed sel.<tab>.<dim> in the URL, so a pick on Acquisition cannot silently narrow Audience.
The map wears the agency's hue. Six steps of the agency's brand colour instead of the default blue, derived by lib/brand-tokens.ts#mapRampStyle and set inline by CountryMap — the only place the brand colour enters the app itself. Rationale, the ladder, and the validation are in Frontend.
The map is one map in two frames. CountryMap owns view, position, hover and the expanded flag; WorldMap is the canvas and MapSummary the card, so expanding is a change of SIZE — the pan and zoom carry both ways rather than snapping back twice per trip. Only one canvas is mounted at a time (each hover re-renders ~180 geographies; two would double it for a map behind an overlay). The canvas measures itself with a ResizeObserver and derives the projection scale from its width (box.w / 2π = one world edge to edge), because a hardcoded scale is right at exactly one size — it was letterboxed at 420px and would be a postage stamp at 1600. Fullscreen hides the stat cards and the ranked table, so MapSummary carries the same three figures, the same ranking (useCountryIndex, the one place the rank→shade ramp is computed, so the list and the shading cannot disagree) and the same coverage line. Clearing picks is clearSelection on the page, NOT a loop of toggles at the call site: each toggle re-reads params from its own render, so a loop writes the same base repeatedly and drops only the last key.
The palette is computed, not chosen. --chart-* is Okabe-Ito, six slots, validated all-pairs (dataviz/scripts/validate_palette.js). Six hand-rolled attempts failed: blue/purple collapse under protanopia, cyan/teal are indistinguishable to normal vision. Two findings remain and are written into index.css — sky and amber below 3:1 on white, and three above the dark lightness band. Both demand the same relief and the design carries it: a legend is always present, bands keep a 2px surface gap, every chart sits above a table naming each colour. Attempts to fix the dark band collapsed amber into vermillion (ΔE 10.4 against a floor of 15), because that lightness gap IS the separation.
Selected state must not use ring-*. Tailwind draws rings as box-shadows and index.css zeroes --tw-ring-shadow under focus, so a button keeps focus after a click and its own decorative ring stays suppressed until blur — the selection appearing a beat late. Border colour + listRowSelectedClass instead.
Locale is pinned (format.ts). toLocale*(undefined, …) follows the reader's OS, which rendered country names in Dutch inside an English app.
Two markers the chart draws rather than leaving to inference: the partial day (a ReferenceArea spanning the last SEGMENT — a categorical axis makes x1===x2 zero-width, which is why the first version drew nothing), and tracking started, from getStatus().firstSeenDate, so a window wider than the install explains its own empty half.
The tool surface (2026-08-08)
Same set serves the in-app agent and MCP. It was rebuilt to mirror the READ layer rather than the old UI: the reads went from nine question-shaped RPCs to three parameterised ones, and the tools had not followed — ten of them, 36% of the app's entire module-tool surface, with two that were strict subsets of newer ones.
One window vocabulary. range OR from+to, on every tool. Since 2026-08-23 the fragment and its parser are the platform-wide ones in core/utils/window.js (Search Console and the workspace rollup spread the same WINDOW_PARAMS); agent-tools/window.js re-exports them and keeps the pixel-only vocabulary (filters, lanes, and PAGE_PARAMS — path OR url, with pagePath(params) deriving the path the browser reports, trailing slash kept as stored). There used to be three spellings for one concept (range, days, from/to on a single tool), so "how did July go?" was askable of exactly one read. The schema fragment and its parser are imported, not retyped, so a tenth tool cannot invent an eleventh spelling. get_journey_sessions reports the shared list envelope (total/offset/returned/truncated; hasMore is gone).
Narrowing. filter_dim + filter_values intersect dimensions — the agent's version of clicking a row. Without it a model can rank countries and rank sources but never ask which countries the paid traffic came from, which is the ordinary next question. lane scopes reads to who was browsing.
Two surfaces that had no tool at all: get_realtime_activity (the module's only live view — MCP could not see it) and get_pixel_status. The second is not new data: install state lived inside get_conversion_picture.tracking, which is the right fact in the wrong place. An agent asked why traffic is zero has no reason to call a CONVERSIONS tool, so it answered from an empty result instead of from the fact that nothing is installed. Reachable is not discoverable.
Two removed, folded into tools that already contained them: get_crawler_activity → get_traffic_lanes.byAgent, get_campaign_performance → get_traffic_timeseries(breakdown='campaign'). Their unique wording — the JS-pixel coverage warning, the untagged-campaign caveat — moved with them; a duplicate tool is a chance to pick the narrower one and report a smaller truth. (2026-08-23: the orphaned getCampaignPerformance query was deleted outright — no caller since the tool went; getCrawlerActivity is back in use behind get_traffic_lanes' per-page block, below.)
Window parity + the two audit bugs (2026-08-23, plans/tool-coverage-audit-2026-08-23.md §3). Every Statistics tool now takes the one WINDOW_PARAMS vocabulary (range | from+to): get_conversion_picture, get_lead_journeys, get_journey_sessions, get_engagement_events joined get_traffic_timeseries/get_traffic_lanes/get_session_report. The mechanism is the windowFor wrapper at the top of queries.js, which accepts either a preset string (the legacy positional call sites) or the { range, from, to } object the tools build — so an exact span reaches every read through the same parameter and none can silently fall back to 28d. B1 was exactly that fallback: get_session_report advertised from/to but passed windowOpts(params).range only, so "how did July go" returned the last 28 days labelled as July; it now passes the whole window (pixel_session_report already took p_since/p_until). B2: getPathEngagement, getDwellByPaths and getCrawlerHitsByPath paginated through selectAllPaginated(maxRows), which STOPS SILENTLY at the ceiling — a busy page under-counted with no error, against the module's refuse-don't-undercount rule. They now go through selectAllOrRefuse (fetches one row past the ceiling, throws exceeds the N-row ceiling); the tools surface it as page.error / agent_pages.error, the detectors as a failed detector. get_ai_referrals keeps days and now says plainly that it is a ROLLING UTC span, not calendar days. The /journeys route honours from/to too (the Journeys page already sent them).
Filters, joins and caps added in the same pass: get_lead_journeys leads carry id, visitorId, sessionId (the join to get_journey_sessions) and take goal_key; get_engagement_events takes path (scopes clicks + forms; continuation stays site-wide), per_kind (default 15, max 100; rowsTotal/truncated per kind) and top_articles (default 5, max 50; articlesTotal/topArticlesTruncated); get_ai_referrals takes landing_pages_limit (15/100) and recent_leads_limit (10/50) with landingPagesTotal/…Truncated; get_realtime_activity takes lane — applied in JS AFTER the RPC's roster/feed caps (the RPC has no lane parameter), which the description and a laneFilter block both state; get_conversion_picture takes attribution (first|last|linear → getOverview, getArticlesSummary, getArticleLeads), returns tracking.days_until_ready, pages_total/ pages_truncated, and per goal count_in_window/value_in_window (from getConversionsByGoal) beside count_28d/value_28d, which pixel_goal_stats pins to a rolling 28d whatever window was asked. get_traffic_timeseries documents that table rows re-rank by metric only for conversions/value/pageviews (everything else ranks by sessions) and gained breakdown: 'goal' (table view only): getConversionsByGoal for the exact window — every configured goal including zero-count ones, the (unassigned) row for deleted goals, previous under compare — the Conversions tab, reachable. get_traffic_lanes takes path/article_id and adds an agent_pages block: getCrawlerHitsForTarget (new in queries.js: UA-named crawlers plus the stealth_scraper bucket for visitors that never cleared the humanity bar, 20k ceiling, refuses) beside the site-wide getCrawlerActivity RPC result.
Not built: a path filter on get_journey_sessions — pixel_journey_sessions filters by article only and post-filtering one page of sessions would make total/paging wrong; it needs a p_path parameter on the RPC (migration + sql-drift snapshot). The description says so and points at get_session_report.landingPages / get_conversion_picture(path) instead.
The tool file split: agent-tools.js keeps the two report-surface tools (get_traffic_timeseries, get_session_report) and the array; agent-tools/journeys.js (leads + sessions), agent-tools/engagement.js (AI referrals + engagement events), agent-tools/lanes.js, agent-tools/live.js, agent-tools/conversion-picture.js, with the shared sentence in agent-tools/honesty.js and the window/filter/lane vocabulary in agent-tools/window.js.
Page identity rides resolve_pages on landing breakdowns rather than a separate tool, which would be a second call the model has to know to make. It goes through workspace/page-identity.js — the SPINE, because both catalogs are spine-owned and a module may never import another module. The same resolver serves the Pages module's HTTP route, so the UI and the agent cannot disagree about which page a path is.
Plan and gap evidence: plans/analytics-tool-surface-2026-08-08.md.
Conversions by goal (getConversionsByGoal)
Not a breakdown, deliberately. Every other report aggregates SESSIONS across a dimension of the session; this counts LEADS joined to the goal definitions that claimed them, and a session row carrying a goal it may have matched zero or several times is not a dimension.
Only counted leads. goalsConfigured is returned SEPARATELY from the count, because zero means two different things and the manual-first model makes that permanent: a workspace with no goal reports zero forever, and telling that operator "no conversions" is unactionable advice. Goals with zero conversions are listed at zero (a quiet configured goal is how you notice a form that stopped submitting); inactive goals keep their history; a lead whose goal was deleted keeps its own row. Cross-checked against getOverview().conversionsByGoal on five workspaces — 13/13, 13/13, 14/14, 0/0, 0/0.
Sessions as an entity (pixel_sessions, W2)
We always had session IDs — client-generated, 30-minute idle window, in localStorage so they span tabs — and Visitor Journeys, which shows any single session's pages and outcome. Verified before building on the id: zero session ids shared across visitors, 0.53% with an internal gap over 30 minutes. The key is sound.
What was missing is a row. Session-shaped facts were re-derived by GROUP BY over raw pageviews at request time, which works for an INSPECTOR (journeys computes attributes for the 25 rows on screen) and is unaffordable for a REPORT. So the product could show you one session's landing page and could not answer "what are my top landing pages, and which of them bounce" — whose answer on the busiest workspace is 8,280 paid sessions landing on two product pages and bouncing at ~89%. Not new visibility. New aggregability.
- Written at ingest, inside the existing RPCs — no extra round trip. The INSERT branch carries every session-scoped attribute; the UPDATE branch touches only what advances per pageview. That asymmetry IS the session-scoped-source fix:
landing_path/source/campaignare written once by the first pageview and never overwritten by a later in-site navigation whose owntraffic_sourcereadsinternal. - Engagement is RECOMPUTED, never incremented. The heartbeat sends a running total every 15s and the pageview MAX-merges it, so adding deltas would double-count every beat after the first.
- ONE definition of bounce, enforced as a generated column so no query can invent a second: no pageview cleared the qualified bar and nothing converted. Deliberately not "one pageview" — someone who reads a single long article to the end is engaged, not bounced.
- Duration is stored literally, engaged time is what we report. GA4 force-closes sessions at 4h; we can't do that retroactively without changing client id generation, and 0.53% of sessions run long — a mean over duration would be dragged anywhere, a mean over engaged time would not.
- Backfill (
pixel_backfill_sessions, idempotent): 76,627 sessions reconciling exactly — 0 missing, 0 orphans, 0 count/time/flag/timestamp mismatches, 122,801 pageviews both ways. pixel.session-integrity(FAIL) keeps it that way. ⚠️ Its first version compared a 7-day raw window against each session's lifetime count and fired on two correct rows; the comparison must be like-for-like — scope by session START, then count all of that session's pageviews regardless of age.
Surfaced by getSessionReport → GET /:ws/sessions and the get_session_report agent tool (totals, landing pages, exit pages, by session-scoped source).
The humanity rule (services/humanity.js + pixel_human_visitors())
"Was there a person here?" sits under every visitor number, every conversion rate, and every claim we put in front of a client. It is pure and isolated so it can be tested case by case.
Where it still runs. W3 moved the answer to ingest time — the report surface, the Analytics page and pixel_daily all read agent_lane off the row, and never call this. What still re-derives it at read time is the pre-W3 set: pixel_overview, pixel_articles_summary, pixel_article_stats, pixel_top_non_blog_pages (the monthly report's JS mirror was cut 2026-08-16). Those are the cut list.
A visitor is human if they converted (conclusive, and unconditional — no clause below may override a form fill), or made a real gesture (pointer/touch/key/click/input, isTrusted-gated client-side), or scrolled AND stayed ≥5s (two weak signals agreeing), or viewed more than one page (navigation is intent) — and are not a single-pageview visitor on an unidentifiable browser who never gestured.
Recalibrated 2026-08-08. The previous rule (time ≥ 5s OR scroll OR gesture, UNION leads) was measured against ground truth built from signals independent of the rule itself — visitor converted, or returned on a different calendar day — over 23,343 visitors / 28 days:
| real humans LOST | known bots COUNTED | |
|---|---|---|
| previous rule | 51 | 432 |
| this rule | 0 | 0 |
It failed in both directions. Dwell time alone was carrying 17% of the "humans", and a headless browser idling on a page produces exactly that. In the other direction it discarded 51 people who converted or came back another day, because every one of their engagement beacons was lost — the mobile pagehide problem the 15s heartbeat now addresses.
The unknown-browser guard is deliberately narrow (single-pageview only) so it can never catch a multi-page reader on an exotic browser, and it sits under the conversion clause so it can never discard a form-filler. Honesty about the 100%: the guard's clause matches how the bot cohort was defined, so that figure is partly definitional — the independent number is the rule without the guard: 96.1% of known bots excluded, 0 humans lost. The guard adds the remaining 43 at zero human cost.
Fleet effect: reported visitors fall ~13% (13,714 → 11,859 over 28d) — 432 bots leave, 51 real people who were being discarded come back. Implemented twice (SQL for RPC paths, JS for in-Node paths); scripts/test-pixel-humanity.mjs pins both and asserts they express the same clauses, because two definitions of "who counts" that disagree is worse than either being wrong. ⚠️ JS callers MUST select browser, or every visitor looks unidentified and the guard over-fires.
Journeys stopped asking it — 2026-08-09
pixel_journey_sessions kept the read-time filter after W3, and it was the wrong question asked of the wrong grain. The rule is a verdict on a visitor ("did this person ever prove they were human"); journeys applied it to sessions, so one unqualified visit deleted that person's other visits too. Someone who read a page carefully and left was dropped — along with the journey that shows exactly that.
On Luniq, 28 days: 1,802 sessions in the humans lane, 532 shown, 1,270 thrown away. It now reads COALESCE(agent_lane, 'humans') = 'humans' like everything else — 1,785 of 1,802 on Luniq, 2,343 of 2,369 on Matubu, 18,050 of 18,163 on OysterClamp, the remainder being sessions whose pageviews fall outside the window. Rows from before classification carry a NULL lane and read as human: they were never classified, and inventing a verdict for them now would be modelling, not measuring.
Nobody noticed for a month because the explorer is this function's only reader and it was throwing a ReferenceError from the W1 window refactor until 2026-08-08 — the page showed nothing, which reads as "no traffic yet", not as "the filter is wrong".
Data-quality alarms (npm run check:data)
Five standing assertions, each a defect we actually shipped. House rule from the geo incident: assert distributions, not row counts — rows kept arriving for all 25 days; it was their shape that was wrong, and nothing watched shape.
| Check | Severity | Guards |
|---|---|---|
pixel.geo-per-lane | fail | geography stays long-tailed AND resolved, per serving lane — a regression on one path would otherwise hide inside the other's healthy average (2026-07-12) |
pixel.window-reconciles | fail | every workspace: KPI total equals the sum of its chart buckets, exactly (the 11%-orphaned defect) |
pixel.identity-churn | warn | near-total single-pageview visitors = a new identity minted per view |
pixel.engagement-capture | warn | a high no-signal share means lost beacons — which the humanity bar then misreads as automation |
pixel.ingest-heartbeat | warn | a workspace that was receiving beacons still is (48h silence = broken install / DNS / newly blocked script) |
Measurements live in the database as pixel_quality_report(); the script only compares them to thresholds. Deliberately not a generic exec-SQL RPC — that is a standing liability (arbitrary SQL under the service role) for a check script's convenience. The split also keeps "what we measure" versioned in a migration and "what we tolerate" reviewable in code. A thrown error inside the block registers as a FAIL rather than being skipped: a broken alarm must be loud. That rule caught its own component (2026-08-10): pixel_session_integrity() joined all-time pageviews to 7-day sessions, so its cost grew with history until the report died on the statement timeout. A scoped session's pageviews cannot precede its started_at, so the scan is now bounded at 8 days (pixel_session_integrity_bounded_scan) — same answer, prunable by the viewed_at index, ~1s fleet-wide.
The RPC bodies themselves are pinned by the sql-drift gate in npm run verify (scripts/check-sql-drift.mjs): pixel_function_defs() (service-role-only, read-only) returns the live definition of every pixel_* function, and the gate hashes each against scripts/baseline/pixel-sql.json. guards.mjs pins the names; this pins the SQL — the gap both 2026-08 defects lived in (the regex patch that silently skipped WITH cur AS (, and pixel_journey_sessions carrying a pre-W3 humanity filter for a month). A body that changed without the snapshot changing is an unrecorded hotfix or a drifted environment. Intentional change = migration + --snapshot, same commit. Old overloads (the pre-p_until signatures) stay pinned until main stops calling them — dropping one shows up as MISSING.
pixel_pageviews.edge_verified records row provenance — whether the request provably transited the CDN. Without it the per-lane geo assertion is impossible, which is exactly the blind spot that let the 2026-07-12 collapse run unseen.
Incident 2026-08-08 — every pageview dropped for 2h20m
handlePageview called cloudflareConnectingIp(req) without importing it. A ReferenceError on the first classified pageview after deploy, then on every one after that. The beacon wrapper catches everything and answers 204, by design — a tracking script must never surface an error into a customer's page — so the client believed each send succeeded and moved on. There is no retry queue: the pageviews are gone.
Why nothing caught it:
node --checkparses; it does not resolve identifiers.resolve-check.mjsimports every module; a name missing inside a function body only fails when that line executes.pixel.ingest-heartbeatwaits 48h before complaining. This was total loss in minute one.- Every other beacon kept working — events, leads, engagement — because only the pageview path classifies. Volume looked healthy from any angle that did not separate the beacons.
Two permanent fixes, both in npm run verify / check:data:
| Gate | Catches |
|---|---|
scripts/check-undefined.mjs | a referenced name that is not declared, imported or a known global — the whole X is not defined class, statically, before it ships |
pixel.beacon-parity | one beacon silent while its siblings flow. Events without pageviews is structurally impossible from a real browser, so it means ingestion, not traffic |
The diagnosis that worked, worth repeating: compare the last write time of every table a subsystem owns. pixel_events seconds ago against pixel_pageviews two hours stale located the break in one query, after the health endpoint, the CDN, the script delivery and the RPC had all reported fine.
Reporting windows — calendar days in the workspace timezone (services/window.js)
Every read in this module resolves its window through ONE helper. queries.js composes two separate concerns: workspace/timezone.js (spine — which calendar the customer lives in) and services/window.js (pure, no I/O, unit-tested in scripts/test-pixel-window.mjs).
The old rangeToDate() returned an open-ended now() - N days timestamp, while pixel_overview generated its chart buckets as N calendar dates ending on today's UTC date. Those are different sets of rows, so the KPI card and the chart disagreed — measured on prod 2026-08-07, busiest workspace, 7d range: 5,339 total pageviews vs 4,963 summed across the seven bars, 376 counted on no bar. Two more consequences of the same root cause: reporting ran in UTC, so a Belgian workspace's day began at 02:00 local; and the in-progress day was drawn as a completed one, which reads as "traffic collapsed today".
resolveWindow() returns a HALF-OPEN interval [startsAt, endsAt) whose bounds are local midnights, plus the dates[] the buckets are cut on, the timeZone actually used, and partialDay (the in-progress day — only ever set on the current window, never on the comparison period). The RPCs take p_since/p_until/p_tz and bucket with (viewed_at AT TIME ZONE p_tz)::date, so SUM(buckets) = total is an identity, not a hope — re-measured after the change: 5,029 vs 5,029, zero orphaned. Direct (non-RPC) queries close their upper bound with .lt(...) for the same reason.
Deliberately still open-ended: the 24h ingest heartbeat in getStatus, the current-vs-prior detector windows (getLeadWindowCounts, getAiReferralWindows — both run to now by design), and humanity evidence in buildHumanSetForVisitors (evidence after the window still proves a visitor is human; the window-dependence this creates is what stored classification fixes in W3).
Migration: db/pixel-calendar-windows-migration.sql — strictly additive. Production runs main while this lands on staging and both share the database, so nothing is dropped; the new signatures differ in arity and PostgREST resolves them by parameter name. The superseded signatures go in W5 with the rest of the cut list.
The outcome loop's attribution read (getLeadsTouchingPaths, 2026-08-20)
One query serves the agent's outcome ledger (plans/outcome-kpis-v1.md): counted leads that touched any of a signal card's linked paths — the lead's conversion or landing path is one of them, or its visitor viewed one within a 30-day lookback before converting (any-touch; callers word it "touched", never "generated"). Returns { leads, value }, value summed from each touched lead's own value_amount (stamped per goal at ingest), each lead counted once however many paths it touched. Truncation refuses, per the getOverview lesson. Lives in queries.js beside getLeadCountsByPath (which counts by CONVERSION page only — a different question); the agent reads it through modules/agent/outcomes.js, never directly.
Serving: two hosts, one origin (orbspan.com + pixel.orbit.luniq.io)
orbspan.com (registered 2026-08-07, Cloudflare-proxied → the same Railway backend) is the ingest host. pixel.orbit.luniq.io remains a plain CNAME straight to Railway and is never removed — the snippets using it are on customer sites we do not control.
Live installs migrate without touching a single client site. A redirect cannot do this: script.src reflects the ATTRIBUTE, not the post-redirect URL, so a redirected old install would fetch from the new host and still post every beacon to the old one — moving the one cheap request and leaving all the valuable ones behind. Instead orbit.js decouples the two: the legacy host serves the file byte-for-byte as installed, and the file it serves posts to INGEST_BASE. Existing customers move as their 1h script cache expires.
send() falls back to the script's own origin the first time navigator.sendBeacon() returns false — which is what a CSP connect-src violation looks like, synchronously — and pins there for the pageview. A customer whose policy allows only the legacy host cannot break.
Both the script and every beacon are mounted twice: /orbit.js + /js/o.js, and /v1/<name> + /o/<letter>. One handler per beacon, so the mounts cannot drift. The neutral paths carry no tracking vocabulary — filter lists match on exactly the shape of orbit.js under a host called pixel.
Edge trust — a header is only as good as its provenance (services/cloudflare.js)
Cloudflare hands us the visitor's true country (cf-ipcountry), their real IP (cf-connecting-ip), and a verified-bot verdict. All three are just HTTP headers, and the legacy host still points straight at Railway — so anyone can bypass the CDN and send whatever they like. Trusting that blindly would hand an attacker the rate-limit key: the exact failure mode the geo incident was created by and then over-corrected for.
isCloudflareRequest() requires BOTH a cf-ray header (only the edge stamps it) and a Cloudflare address somewhere in the forwarding path, matched against the published ranges (bundled so boot never depends on the network; refreshed daily, last-good kept on failure). Header alone is forgeable; address alone would trust any Cloudflare customer's traffic.
Deliberately no hop-counting — the incident came from assuming a fixed chain depth, and adding a CDN changes that depth again. The only question asked is "is Cloudflare anywhere in this path", which stays true however many proxies Railway runs. When cf-ray is present but no CF address is, it logs the real chain (throttled) rather than guessing at production topology.
This upgrades getRateLimitKey() on the new host from shared infrastructure to a real per-visitor identity — two visitors behind one edge finally get separate buckets. The workspace dimension in rate-limit.js stays, because the legacy path still needs it. Asserted in scripts/test-pixel-edge-trust.mjs (14 checks, including the spoof cases).
Geo — two IPs, never one (services/geo.js)
Ingestion asks two questions that look identical and are not. geo.js answers them separately, and they must never be merged again:
| Function | Question | Answer | Spoofable? |
|---|---|---|---|
getVisitorIp(req) | Where is this person? | LEFT-most public X-Forwarded-For entry (every proxy appends, so the chain reads client → edge → … → us; private/CGNAT hops skipped) | Yes — and that is acceptable: a forged header costs one wrong country on one row |
getRateLimitKey(req) | Who do I throttle? | req.ip — the right-most untrusted hop under trust proxy: 1 | No — and it must stay that way |
computeFingerprint (the cookieless identity fallback) also keys on getRateLimitKey, not the visitor IP: a forged header must never let a caller choose someone else's visitor_id, because that id drives lead attribution.
The 2026-07-12 incident. The security hardening pass (c8d77f6) correctly closed a rotating-XFF limiter bypass, but did it by pointing both uses at req.ip. Railway runs more than one hop in front of the container, so req.ip is a Railway edge address. From 15:52 CEST every pageview fleet-wide was geolocated to our own infrastructure: distinct countries per day collapsed 37 → 6 overnight, Belgium and the Netherlands read exactly zero for Belgian and Dutch sites, and the edge addresses resolved to Brazil and the UK because the bundled GeoLite2 snapshot (20231115) still named their pre-transfer 2023 holders. It ran 25 days and 34,130 pageviews / 20,586 visitor rows before detection, feeding the app, the monthly report country table, and the agent. The client IPs were never stored, so that window is unrecoverable — not backfillable, only nullable.
Snapshot freshness. geoip-lite bakes its GeoLite2 snapshot in at package-publish time and nothing refreshes it. scripts/update-geoip.mjs runs as postinstall and pulls a current copy when MAXMIND_LICENSE_KEY is set; without a key it skips and exits 0 by design (a missing key must not break npm install or a deploy). Staleness surfaces as bad data instead, via check:data.
Guarded nightly, without anyone remembering to run anything. pixel-geo-country-collapse lives in modules/agent/data-integrity.js (the canonical per-workspace invariants), so the nightly ops detector runs it on every workspace and both consumers share one definition. It is a TRIPWIRE, not a board card: a geo bug is our ingestion breaking, and no customer can fix that — it emails the operator instead of telling a client their data is wrong. Fires at ≥150 resolved pageviews in 48h with <10 distinct countries; the 48h window means a quarantined past ages out in two nights while a fresh regression surfaces the next morning.
The thresholds are measured, not guessed. The first attempt used ≤2 countries; replayed against the incident's own preserved data it would have caught nothing (six rotating edge IPs still yield 3–6 countries over 48h). Same 48h shape, healthy vs broken, ≥150 resolved:
| Healthy (Jul 5–6) | Broken (Aug 4–5) | |
|---|---|---|
| OysterClamp | 875 pv → 26 | 2076 pv → 6 |
| Luniq | 211 pv → 21 | 260 pv → 5 |
| Matubu | 187 pv → 18 | 374 pv → 5 |
| Give It Forward | — | 193 pv → 3 |
Nothing healthy under 18, nothing broken over 6. Ten sits in the gap; the retuned guard fires on 5/5 broken workspaces and stays quiet on 3/3 healthy ones. Re-measure before moving it — a guard tuned by intuition is a guard that quietly does nothing.
Also in check:data (the fleet-wide operator sweep) — real visitor geography is long-tailed, so a collapsed tail means we stopped resolving visitors and started resolving ourselves:
geo.country-cardinality(fail) — a workspace with ≥200 weekly pageviews resolves ≥5 distinct countries. This is the general detector; it does not care which IP we wrongly resolved, and it would have caught the incident on day one.geo.country-resolves(fail) — a workspace with real traffic resolves a country for at least some pageviews.geo.database-freshness(warn) — the GeoLite2 snapshot is under 90 days old.
Routes
Public (unauth, /pixel/*): GET /orbit.js, POST /v1/pageview, POST /v1/engagement, POST /v1/events (engagement events → pixel_events, validated + capped at 25/beacon, RLS service-role-only, db/pixel-events-migration.sql), POST /v1/lead (rate-limited, 4kb cap). The limiter (services/rate-limit.js) keys on getRateLimitKey (= req.ip, the trusted hop — unspoofable) × workspace, 600/min, under a 3000/min per-workspace ceiling. The workspace dimension is load-bearing: behind Railway's multi-hop edge req.ip is infrastructure shared by the whole fleet, so an IP-only key was six buckets for twelve customers and one site's bot burst could starve ingestion for the rest.
Authenticated (/api/modules/orbit-pixel). Every report route takes the same window params: range (7d|28d|90d) or from+to as YYYY-MM-DD for an exact calendar span, plus compare=previous. Malformed dates fall back to range rather than to an empty window.
| Method | Path | Purpose |
|---|---|---|
| GET | /:workspaceId/overview | Statistics dashboard |
| GET | /:workspaceId/articles/:articleId/stats · /leads | Per-article analytics + leads |
| GET | /:workspaceId/articles-summary | All-article rollup (cached 60s) |
| GET | /:workspaceId/journeys | Visitor journeys (paginated, outcome filter) |
| GET | /:workspaceId/crawlers | AI/LLM crawler activity |
| GET | /:workspaceId/ai-referrals | AI-referred human traffic: visitors current/prior window, per-tool visitors + AI-first-touch leads, recent AI leads (?days=, 7–90). Consumed by the AI Visibility page's proof panel |
| GET | /:workspaceId/timeseries | The chart. metric × breakdown × lane × window, dense daily series. compare=previous adds the preceding period. Carries additive, coverage, partialDay |
| GET | /:workspaceId/breakdown | The table + KPI strip. The period ranked by a dimension, with per-row delta vs the previous period (compare is ON by default here), rowsTotal/truncated, totals.visitors as a true distinct count |
| GET | /:workspaceId/realtime | Last N minutes (5–180, default 30) off raw pageviews: the visitors roster (one row per person, capped 40 + visitorsTotal/visitorsTruncated), per-minute strip by lane, top pages, sources, lastVisit + last24h for the quiet state, and the raw feed. All lanes |
| GET | /:workspaceId/sessions | Session report — landing/exit pages, session-scoped sources |
| GET | /:workspaceId/lanes | Humans vs AI vs bots per day + per-agent verification state |
| GET | /:workspaceId/status | Install status |
| GET/POST/PATCH/DELETE | /:workspaceId/goals[...] | Conversion-goal CRUD + preview + backfill |
| GET/POST | /:workspaceId/form-candidates[...] | Candidate discovery + classify |
Agent tools
| Tool | What it does |
|---|---|
get_conversion_picture | Per-page visitors and leads (articles AND non-article pages), conversion rate, traffic sources incl. AI-tool referrals, top countries, daily trend, and content_contribution — the portfolio rollup (getContentContribution): what share of pageviews, visitors and leads the whole published corpus touches. Deliberately generous on leads (an article anywhere in the journey counts), so a near-zero share is conclusive. Added 2026-08-05 because every existing read answered "which page did well" and none answered "does publishing matter on this site at all" — Forest Forward's 199 articles held 0.9% of pageviews and touched 1 of 145 leads while every per-page number looked survivable — ranges 7d/28d/90d. Leads with a tracking block — because this module is manual-first and a workspace with no defined goal reports 0 leads forever: installed_at/days_since_install/report_ready/total_events_24h, the FULL goal definitions (active, is_conversion, dedup, match, value_amount, count_28d, value_28d, last_fired_at — the status/dedup always-null column-name bug is fixed), and the candidates[] form-candidate worklist + candidates_total (was a bare count). Both source views ride along (traffic_sources/ai_tools by visitor + traffic_sources_pageview/ai_tools_pageview — the UI's toggle). Page rows carry REAL titles (the pixel_articles_summary UUID-naming bug is fixed), real non-article lead counts (the always-0 bug is fixed), ai_share_pct, and visitors_7d/visitors_prev_7d. article_id drills into one article's full stats + recent leads; path drills into ANY page by path — visitors, pageviews, leads, avg time/scroll, sources, AI tools, device + browser splits (refuses with page.error past 10k pageviews — never under-counts). Window: range | from+to (2026-08-23); attribution first|last|linear; tracking.days_until_ready; per goal count_in_window/value_in_window beside the always-rolling-28d count_28d/value_28d; pages_total/pages_truncated. Lives in agent-tools/conversion-picture.js (extracted when the surface outgrew agent-tools.js; re-exported in the module's tool array). The tool description tells the model in plain terms not to report an unmeasured zero as a performance finding. |
get_lead_journeys | Every lead in the window with FULL multi-touch attribution (getLeadJourneys): first/last/all touched articles with titles, session source/medium/campaign, landing path, first-touch AI tool, value_amount, and origin (live = captured as it happened, backfill = synthesized by a retro-applied URL goal) — the read behind "which content influenced our leads / what is this cluster worth". status param: counted (default — real conversions) | candidate | ignored | all — the unclaimed submissions are reachable for tracking review, never reportable as conversions. The query returns { total, leads } (count: 'exact' + .range(offset, …)), and the tool's envelope adds offset/returned/truncated so a capped page is never totalled as the window. Leads carry id/visitorId/sessionId (the join to get_journey_sessions); goal_key narrows to one goal; window range | from+to. |
get_journey_sessions | Visitor sessions as ordered page paths via getJourneys (pixel_journey_sessions RPC): per-page time/scroll, outcome (converted/engaged/bounced), lead details on converted sessions — the funnel read behind "where do journeys die". article_id filters to sessions that touched one article (the per-article funnel, parity with the route); window range | from+to, limit up to 100. No path filter (the RPC filters by article only — needs a p_path migration); the description redirects to get_session_report.landingPages / get_conversion_picture(path). |
get_ai_referrals | getAiReferralWindows: AI-referred human visitors current vs prior window, per-tool visitors + AI-first-touch leads, recent AI leads, and landing pages — the pages assistants actually link (observed-citation ground truth; pixel-but-not-scan pages = citations on untracked questions → prompt-curation candidates). Description pins the floor caveat (most AI visits arrive referrer-less) and forbids lead↔prompt causal claims, and states that days is a ROLLING UTC span (not calendar days). landing_pages_limit (15/100) + recent_leads_limit (10/50) with landingPagesTotal/landingPagesTruncated/recentAiLeadsTruncated. |
get_engagement_events | Three blocks: clicks (getEngagementRollup over pixel_events — per page × kind (internal/outbound/action) × target, clicks + distinct sessions, 10k truncation-refusal), forms (getFormFunnel starts vs converted sessions), continuation (getArticleContinuation — continuation + deep-read share). Window range | from+to; path scopes clicks + forms to one page; per_kind (15/100, rowsTotal + truncated per kind); top_articles (5/50, articlesTotal + topArticlesTruncated). Promise.allSettled per block so one failed read never zeroes the others. EVIDENCE framing throughout — clicks/starts prove intent, only goals count leads. |
Conversion tracking is CONFIGURED when a goal EXISTS, never when leads happened (getConversionGoalConfig, 2026-08-05). tracking.configured and get_workspace_performance's conversion_tracking_configured (+ active_conversion_goals) read pixel_conversion_goals; the five signal detectors do the same. The old form inferred it as conversionsByGoal.length > 0 || totalLeads > 0, which conflates setup with outcome: a workspace with real goals and a quiet month was told to go and create the goals it already had, and a goal created after the sweep's lead window read as none. Absence of leads is a RESULT; absence of goals is a SETUP GAP. The outcome proxy survives only as a fallback when the goals read itself fails (counted leads still prove a goal exists — never the reverse).
get_conversion_picture added in v3/B4 (2026-08-05: + article_id drill-down riding getArticleStats/getArticleLeads, + goal DEFINITIONS in the tracking block via listGoals); the journey/lead/crawler/campaign reads added 2026-08-03 (agency read-completion); get_ai_referrals + get_engagement_events added 2026-08-05 (full-surface completion: the AI channel and the X1 engagement-event stream were dashboard/detector-only — the agent reasoned about conversion without seeing what people click). The 2026-08-05 tool-coverage pass (plans/tool-coverage-audit-2026-08-05.md) widened the whole set to the UI's 90-day windows, fixed the three silent-wrong-answer bugs (goal status/dedup null, UUID article titles, non-article leads: 0), and surfaced the hidden-state rows (candidate/ignored leads, form-candidate worklist, pageview source views, install age). All honesty-gated the same way: read failures return { error }, never zeros, and empty lead reads point the model at the tracking block first.
Signal detectors (mechanical, no LLM)
agent-signals.js → classifyConversionState(), pure and asserted in backend/scripts/test-autopilot-safety.mjs:
| Finding | Severity | Note |
|---|---|---|
| Pixel silent > 7 days | warning | Returns only this. Data is absent, not zero — no derived finding is claimed on top of missing data. |
| Visitors but no conversion goal defined | critical | The root cause. Suppresses every per-page zero-conversion finding, because with nothing being counted every page would qualify and all of it is one cause. |
| Unreviewed form candidates | opportunity / one_click | Real conversions the site is already producing but not counting. |
| — | The bare zero-lead lanes (pixel:no-conversion:* + pixel:no-conversion-portfolio) said a number without a why. Replaced by the money-page engagement card below. | |
Money-page engagement (classifyMoneyPageEngagement() + detector money_page_engagement, step 6) | warning / M | A KEY page (spine listKeyPages) with ≥30 organic-search sessions that either underperforms HALF the site's own engagement median (time or scroll, medians over the measured key set) or records zero action-kind clicks AND zero form starts (getEngagementByPaths with the organic split + getActionCountsByPaths, one read each). Measure page_engagement; the page's own form funnel rides the evidence, so a dying form on a key page is proven on the page's card in the same pass. LISTING key pages (spine LISTING_PAGE_TYPES) and blog-typed pages never qualify — an articles hub exists to route, and an article's conversion story is the dead-end detector's (the first live run minted a card on the blog index; fixed before release). Fingerprint pixel:money-page:<path>. |
| — | classifyLeadConcentration and its detector deleted; which pieces win leads is the Statistics page's fact, and the money-page card carries the per-page conversion read. | |
AI crawl shift (classifyCrawlShift() over two getCrawlerActivity windows) | warning (drop) / opportunity (spike), S | Per-bot 7d vs 28d run rate, GSC-movement mechanics: a crawler at ≥20 hits/28d that fell to ≤30% of its weekly rate has "nearly stopped fetching" (earliest citation-loss warning there is); ≥2× with ≥10 hits is a ramp worth using. The stealth_scraper bucket is excluded by name — a scraper swarm ebbing is not AI-visibility news. |
Form abandonment (classifyFormAbandonment() over getFormFunnel) | warning / M | form_start events (evidence stream) joined with counted leads by session: forms with ≥15 starting sessions and <20% same-session conversion. ONE aggregate card naming the leaking forms. Gated on goalsDefined — with no goal, completions CANNOT count and every form would read 0%; that root cause has its own critical card. The funnel query is count-checked and throws on truncation rather than under-count. |
| — | "Leads are down" is the board's ONE movement card now (agent/detectors/trajectory.js reads the same getLeadWindowCounts windows, with the same-weeks-last-year guard). getTimeseries is re-exported through queries.js for that detector. | |
AI-traffic surge (classifyAiTrafficSurge() over getAiReferralWindows, §2b) | opportunity / S | Distinct AI-referred visitors (classifier-tagged, crawler rows excluded) ≥2× the prior 28d window, floor ≥20. Conversion comparison rides along ONLY with goals defined AND ≥30 AI visitors — a rate on 12 visitors is a coin flip, not a finding. Per-tool split in evidence. |
AI-referred lead (classifyAiLeads() over getAiReferralWindows().recentAiLeads) | info / S | One receipt card per counted lead whose FIRST touch was an AI assistant (fingerprint pixel:ai-lead:<id>, cap 3 newest, pre-cap presence). Exists because the surge floor (≥20 visitors) sits far above a workspace's first AI lead — the loop-closing proof card. Ages out of the 28d window → absence-reconciled to Done. |
Blog dead-end (classifyContinuation() over getArticleContinuation) | warning / M | ONE corpus signal, never per-article: of the human sessions that touched an article, how many continued to any non-article page or converted. Fleet measurement (2026-07-22): 96–99% one-and-done is the NORM everywhere, so per-article flags would be pure noise. Floors: ≥100 article sessions AND continuation < 10%; names the top dead-end articles as examples (each carrying url as well as title since 2026-08-14 — getArticleContinuation selects it so the drawer's affected-pages panel can link every named article). Independent of goal configuration (continuation is navigation, not conversion). The query paginates pixel_pageviews (PostgREST caps a select at 1000 rows) with a 30k hard ceiling that throws rather than under-counts. Deep-read share (Wave 4 X0) rides the same pass: % of article views scrolled ≥70% (fleet baseline 25%) — at ≥40% the diagnosis flips to "the missing piece is the next step, not better writing". Scroll % is only read on ARTICLES: fleet calibration showed non-article pages median 7% scroll because short pages have nothing to scroll, so absolute scroll thresholds are invalid there — the honest above-the-fold detector waits for the click events (X2, after the bake). |
Live across the fleet: Imediaal and Matubu report one honest root cause (no goal defined) plus their uncounted form candidates — Matubu has 10 — while Luniq, which has tracking configured, gets the real per-page findings instead. The dead-end detector probed live: Luniq 201 sessions / 20 continued, Matubu 229 / 13 (5.7%) — with the bot-heavy raw counts (1.3k "sessions") filtered to humans first.
Crons
None.
Data
Tables: pixel_visitors, pixel_pageviews, pixel_leads, pixel_conversion_goals; reads blog_articles. RPCs: pixel_ingest_pageview, pixel_ingest_engagement, pixel_overview, pixel_human_visitors, pixel_articles_summary, pixel_journey_sessions, pixel_crawler_activity, pixel_article_stats, pixel_top_non_blog_pages, pixel_counted_leads, pixel_goal_stats, pixel_form_candidates, pixel_url_goal_preview, pixel_backfill_url_goal.
Key files
index.js, config.js (frozen id), routes.js (dashboard API), agent-tools.js + agent-tools/conversion-picture.js (the extracted flagship read), services/ingest.js (beacon orchestrator + recordConversion), services/goals.js, services/classifier.js, services/device.js, services/article-match.js, services/fingerprint.js, static/orbit.js (client), api/routes/pixel.js (public transport).