AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.
perf(backend/copilot): cap eager-loaded chat messages at 1000 most-recent (#13047)
## Why Continues the egress reduction stream from #13030 (NotificationEvent), #13033 (3 query patterns), and #13038 (ownership-check routes). The next-largest remaining egress query in prod is the chat-session message load: ```sql SELECT … FROM "platform"."ChatMessage" WHERE "sessionId" IN ($1) ORDER BY "sequence" ASC OFFSET $2 ``` Per `pg_stat_statements`: 84,821 calls / 75,307,064 rows / **887 rows per call**. Sessions with long histories drag the entire conversation back even though the LLM context builder immediately compresses most of it away. The transcript checkpoint mechanism (GCS-backed JSONL) was already designed to keep "the gap since last compaction" small — but the DB load was pulling the full history regardless. ## What The eager-load path now caps the message tail at `MAX_LOADED_CHAT_MESSAGES = 1000` most-recent rows. `get_chat_session` was removed entirely; callers go through `get_chat_messages_paginated` (with `limit=MAX_LOADED_CHAT_MESSAGES`, no cursor) so the LLM-context path inherits the same tool-pair boundary expansion + visibility guarantees the UI scroll-back path already had. The cap-hit signal lives in `PaginatedMessages.has_more` and `_get_session_from_db` logs a warning when it engages. Older history continues to live in the GCS transcript checkpoint and is layered back into the LLM context via `extract_context_messages` exactly as before — plus a new **hole-fill** step that fetches the sequences between transcript watermark and window start when both the cap engages and the transcript is stale (so the LLM never sees a mid-conversation hole). `detect_gap` now filters by each `ChatMessage.sequence` field rather than relying on list-index = absolute-sequence equivalence (which breaks once the list is windowed). The transcript watermark is a *count* of non-reasoning JSONL rows; callers translate it to a real DB sequence via `get_sequence_at_non_reasoning_index` before calling `detect_gap` so reasoning-row interleaving doesn't shift the boundary. ## How The cap (1000) is conservative — pre-fix average rpc was 887 messages per session load, so 1000 covers the realistic short tail comfortably with no behaviour change for typical sessions. For pathological multi-thousand-message sessions, the GCS transcript already provided the bulk of LLM context; the 1000-message DB window provides the gap since the last compaction, plus a DB-side hole-fill when needed. The watermark uploaded with each transcript is now read from a single `COUNT(*)` over non-reasoning rows in DB rather than `len(session.messages)` (which would under-report the moment the cap engages). `add_chat_messages_batch` returns the actual `start_sequence` it used (not just the input — the helper retries from `get_next_sequence` on a unique-constraint collision) so `_save_session_to_db` can back-fill in-memory `ChatMessage.sequence` to whatever the DB actually wrote. ## Tests - `TestDetectGap` covers index-based legacy fallback, sequence-based windowed gaps, role-shape sanity at the boundary, and the cap-engaged window-above-watermark path (user-leading, assistant-leading, and tool-leading rejection). - `TestExtractContextMessages` exercises the watermark→sequence translation, hole-fill range bounds, reasoning-row filtering inside hole-fill results, and translation-failure fallback. - SDK `query_builder_test.py` adds 5 cases for the cap-engaged `_build_query_message` path including hole-fill and DB-failure fallback. - `test_save_session_to_db_uses_actual_start_after_collision` regression-tests the in-memory back-fill follows the actual DB `start_sequence` after a retry. ## Expected impact `ChatMessage IN($1) OFFSET $2` rpc should drop from 887 to ≤1000 (and typically far less, since most sessions don't have hundreds of messages). Total row egress on this query: substantially reduced on the heavy-user tail; near-flat on short sessions. ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my own code - [x] I have made corresponding changes to the documentation (n/a) - [x] My changes generate no new warnings (black/ruff/pyright clean) - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes (CI runs the full suite) - [x] Any dependent changes have been merged and published in downstream modules
Z
Zamil Majdy committed
b2719fd9e4314d02628d0d26f18bb76323047c06
Parent: 27746ba
Committed by GitHub <noreply@github.com>
on 5/8/2026, 10:56:37 AM