SIGN IN SIGN UP

AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.

0 0 77 Python

perf(backend/copilot): cap eager-loaded chat messages at 1000 most-recent (#13047)

## Why

Continues the egress reduction stream from #13030 (NotificationEvent),
#13033 (3 query patterns), and #13038 (ownership-check routes). The
next-largest remaining egress query in prod is the chat-session message
load:

```sql
SELECT … FROM "platform"."ChatMessage"
WHERE "sessionId" IN ($1) ORDER BY "sequence" ASC OFFSET $2
```

Per `pg_stat_statements`: 84,821 calls / 75,307,064 rows / **887 rows
per call**. Sessions with long histories drag the entire conversation
back even though the LLM context builder immediately compresses most of
it away. The transcript checkpoint mechanism (GCS-backed JSONL) was
already designed to keep "the gap since last compaction" small — but the
DB load was pulling the full history regardless.

## What

The eager-load path now caps the message tail at
`MAX_LOADED_CHAT_MESSAGES = 1000` most-recent rows. `get_chat_session`
was removed entirely; callers go through `get_chat_messages_paginated`
(with `limit=MAX_LOADED_CHAT_MESSAGES`, no cursor) so the LLM-context
path inherits the same tool-pair boundary expansion + visibility
guarantees the UI scroll-back path already had. The cap-hit signal lives
in `PaginatedMessages.has_more` and `_get_session_from_db` logs a
warning when it engages.

Older history continues to live in the GCS transcript checkpoint and is
layered back into the LLM context via `extract_context_messages` exactly
as before — plus a new **hole-fill** step that fetches the sequences
between transcript watermark and window start when both the cap engages
and the transcript is stale (so the LLM never sees a mid-conversation
hole).

`detect_gap` now filters by each `ChatMessage.sequence` field rather
than relying on list-index = absolute-sequence equivalence (which breaks
once the list is windowed). The transcript watermark is a *count* of
non-reasoning JSONL rows; callers translate it to a real DB sequence via
`get_sequence_at_non_reasoning_index` before calling `detect_gap` so
reasoning-row interleaving doesn't shift the boundary.

## How

The cap (1000) is conservative — pre-fix average rpc was 887 messages
per session load, so 1000 covers the realistic short tail comfortably
with no behaviour change for typical sessions. For pathological
multi-thousand-message sessions, the GCS transcript already provided the
bulk of LLM context; the 1000-message DB window provides the gap since
the last compaction, plus a DB-side hole-fill when needed.

The watermark uploaded with each transcript is now read from a single
`COUNT(*)` over non-reasoning rows in DB rather than
`len(session.messages)` (which would under-report the moment the cap
engages). `add_chat_messages_batch` returns the actual `start_sequence`
it used (not just the input — the helper retries from
`get_next_sequence` on a unique-constraint collision) so
`_save_session_to_db` can back-fill in-memory `ChatMessage.sequence` to
whatever the DB actually wrote.

## Tests

- `TestDetectGap` covers index-based legacy fallback, sequence-based
windowed gaps, role-shape sanity at the boundary, and the cap-engaged
window-above-watermark path (user-leading, assistant-leading, and
tool-leading rejection).
- `TestExtractContextMessages` exercises the watermark→sequence
translation, hole-fill range bounds, reasoning-row filtering inside
hole-fill results, and translation-failure fallback.
- SDK `query_builder_test.py` adds 5 cases for the cap-engaged
`_build_query_message` path including hole-fill and DB-failure fallback.
- `test_save_session_to_db_uses_actual_start_after_collision`
regression-tests the in-memory back-fill follows the actual DB
`start_sequence` after a retry.

## Expected impact

`ChatMessage IN($1) OFFSET $2` rpc should drop from 887 to ≤1000 (and
typically far less, since most sessions don't have hundreds of
messages). Total row egress on this query: substantially reduced on the
heavy-user tail; near-flat on short sessions.

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my own code
- [x] I have made corresponding changes to the documentation (n/a)
- [x] My changes generate no new warnings (black/ruff/pyright clean)
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes (CI runs
the full suite)
- [x] Any dependent changes have been merged and published in downstream
modules
Z
Zamil Majdy committed
b2719fd9e4314d02628d0d26f18bb76323047c06
Parent: 27746ba
Committed by GitHub <noreply@github.com> on 5/8/2026, 10:56:37 AM