fix+perf: repair FTS maintenance and cut sync/search hot-path waste
Correctness (audit-confirmed, repro-verified where noted): - v29 migration: rewrite documents_au/_ad triggers with the external-content 'delete' command form for documents_trigram (plain DELETE silently left phantom postings on every rename/delete since v14 — repro-verified) plus a one-time trigram rebuild; add a WHEN guard so no-op upserts and USR/platform backfills stop paying full FTS churn. - Reset consecutive_404_count when a page resurrects via upsert — a revived page kept its 3-strike counter and a single transient 404 re-deleted it. - Seed same-root references when pulling modified pages: new symbol pages linked from updated parents were never discovered by the crawl. - Body index: resume an interrupted full rebuild instead of silently leaving the FTS empty; stamp body_indexed_at at scan start; drop the stale FTS row when a changed document's body renders empty. - Refuse flat-source mass-tombstoning when discovery lost >50% of tracked pages (truncated upstream listings must not wipe catalogs). Sync wall-clock (28-min full sync measured; ~20+ min of it removable): - Gate apple-docc HEAD checks per root on the /tutorials/data/index/<slug> ETag (304 → skip all page checks under that root); weekly + --full sweeps still check every page. The ungated sweep was 343k HEADs for ~0.5% hits. - updated_at only advances on a real content-hash change, making the body index's incremental signal honest (touched-but-unchanged docs re-rendered their body FTS every sync). - Run discovery once per sync and share it between the update and crawl phases (WWDC/GitHub/package-catalog discovers ran twice); skip flat adapters in the crawl phase when update already handled them (the re-sync wiped update's failure diagnostics and re-attempted just-failed keys). - Update adapters run concurrently; drop the global 'primary' rate-limiter bucket that serialized disjoint hosts through one ~440req/s token stream. - Stream page checks through pool() (no more 343k parked semaphore waiters), pull modified pages immediately instead of behind a full-sweep barrier, and retry errored checks once in-run (were silent errCount increments). - If-Modified-Since fallback when no ETag is stored — etag-less pages were permanent false-'modified' re-downloads every sync. - retryFailed only on --full: every sync re-fetched thousands of known-dead URLs; consolidate's transient sweep covers the recoverable classes. - Skip-stamps: SF Symbols codepoint stamp (was 45s + a web scrape per run), prerender inventory via one readdir + atomic SVG writes (was 266k stat pairs + Swift worker spawns to skip everything), raw-JSON minify walk (54s to minify 0 files), Xcode-asset enrich when the content-addressed asset path is unchanged (was a full 327k-page re-scan). - Standalone update default concurrency aligned to sync's 100 cap. Storage/search: - v29 also drops idx_documents_key (duplicate of the UNIQUE autoindex), idx_pages_title/role (no consumers since v6/v21), idx_pages_status (low-selectivity, worse than a scan) and adds crawl_state(root_slug, status) — getPendingCrawl/getCrawlStats were full 360k-row scans per crawl batch. - Body index reads sections one batched IN(...) query per 500 docs instead of per-document (373k statement executions per rebuild). - enrichFromAsset streams the asset scan with iterate() instead of materializing every JSON blob of the corpus in one array. - WAL checkpoint(TRUNCATE) after sync/update; downloadMissing routes fetches through the shared semaphore instead of just copying its max. - Web docs route no longer recycles the whole reader pool per cold page view (rejected all in-flight searches; WAL already exposes the write). - Search: CLI default limit 25 (aligned with MCP) and the deep body tier only fires when strict tiers found <15 hits; trigram tier ranks with bm25 before LIMIT (was arbitrary FTS-cursor truncation); snippets scan at most 6 sections / 16KB per result instead of decoding and lowercasing entire WWDC transcripts on the main thread.
G
Gigi committed
f8d2685bc1c4c0ab99646489d7e9a3cf6263e18e
Parent: 0128b0c