SIGN IN SIGN UP

fix+perf: repair FTS maintenance and cut sync/search hot-path waste

Correctness (audit-confirmed, repro-verified where noted):
- v29 migration: rewrite documents_au/_ad triggers with the external-content
  'delete' command form for documents_trigram (plain DELETE silently left
  phantom postings on every rename/delete since v14 — repro-verified) plus a
  one-time trigram rebuild; add a WHEN guard so no-op upserts and USR/platform
  backfills stop paying full FTS churn.
- Reset consecutive_404_count when a page resurrects via upsert — a revived
  page kept its 3-strike counter and a single transient 404 re-deleted it.
- Seed same-root references when pulling modified pages: new symbol pages
  linked from updated parents were never discovered by the crawl.
- Body index: resume an interrupted full rebuild instead of silently leaving
  the FTS empty; stamp body_indexed_at at scan start; drop the stale FTS row
  when a changed document's body renders empty.
- Refuse flat-source mass-tombstoning when discovery lost >50% of tracked
  pages (truncated upstream listings must not wipe catalogs).

Sync wall-clock (28-min full sync measured; ~20+ min of it removable):
- Gate apple-docc HEAD checks per root on the /tutorials/data/index/<slug>
  ETag (304 → skip all page checks under that root); weekly + --full sweeps
  still check every page. The ungated sweep was 343k HEADs for ~0.5% hits.
- updated_at only advances on a real content-hash change, making the body
  index's incremental signal honest (touched-but-unchanged docs re-rendered
  their body FTS every sync).
- Run discovery once per sync and share it between the update and crawl
  phases (WWDC/GitHub/package-catalog discovers ran twice); skip flat
  adapters in the crawl phase when update already handled them (the re-sync
  wiped update's failure diagnostics and re-attempted just-failed keys).
- Update adapters run concurrently; drop the global 'primary' rate-limiter
  bucket that serialized disjoint hosts through one ~440req/s token stream.
- Stream page checks through pool() (no more 343k parked semaphore waiters),
  pull modified pages immediately instead of behind a full-sweep barrier,
  and retry errored checks once in-run (were silent errCount increments).
- If-Modified-Since fallback when no ETag is stored — etag-less pages were
  permanent false-'modified' re-downloads every sync.
- retryFailed only on --full: every sync re-fetched thousands of known-dead
  URLs; consolidate's transient sweep covers the recoverable classes.
- Skip-stamps: SF Symbols codepoint stamp (was 45s + a web scrape per run),
  prerender inventory via one readdir + atomic SVG writes (was 266k stat
  pairs + Swift worker spawns to skip everything), raw-JSON minify walk
  (54s to minify 0 files), Xcode-asset enrich when the content-addressed
  asset path is unchanged (was a full 327k-page re-scan).
- Standalone update default concurrency aligned to sync's 100 cap.

Storage/search:
- v29 also drops idx_documents_key (duplicate of the UNIQUE autoindex),
  idx_pages_title/role (no consumers since v6/v21), idx_pages_status
  (low-selectivity, worse than a scan) and adds crawl_state(root_slug,
  status) — getPendingCrawl/getCrawlStats were full 360k-row scans per
  crawl batch.
- Body index reads sections one batched IN(...) query per 500 docs instead
  of per-document (373k statement executions per rebuild).
- enrichFromAsset streams the asset scan with iterate() instead of
  materializing every JSON blob of the corpus in one array.
- WAL checkpoint(TRUNCATE) after sync/update; downloadMissing routes
  fetches through the shared semaphore instead of just copying its max.
- Web docs route no longer recycles the whole reader pool per cold page
  view (rejected all in-flight searches; WAL already exposes the write).
- Search: CLI default limit 25 (aligned with MCP) and the deep body tier
  only fires when strict tiers found <15 hits; trigram tier ranks with
  bm25 before LIMIT (was arbitrary FTS-cursor truncation); snippets scan
  at most 6 sections / 16KB per result instead of decoding and
  lowercasing entire WWDC transcripts on the main thread.
G
Gigi committed
f8d2685bc1c4c0ab99646489d7e9a3cf6263e18e
Parent: 0128b0c