feat(backend/copilot): require library similarity check before create_agent (#13080)

### Why / What / How

**Why.** When a user describes a goal to CoPilot ("build me an agent
that summarises my Gmail every morning"), the LLM has been free to call
`create_agent` immediately — even when the user already has a
near-identical agent in their library. The result is clutter, wasted
credits, and a worse experience than just running what they already
have. A real-world example from this dev account: nine `YouTube Video
Summarizer` agents accumulated over time before this gate existed.

**What.** Before `create_agent` runs, CoPilot must search the user's
library for a functionally similar agent (hybrid semantic + lexical) and
surface any matches. A hard gate refuses `create_agent` until that check
has happened. If the user has been shown the matches and explicitly
chose to build new anyway, the LLM retries with `library_check_ack=true`
to bypass.

**How.**
- **Embedding write hook.** A new `schedule_library_agent_embedding()`
fires `asyncio.create_task(...)` from `create_library_agent` and
`update_library_agent_version_and_settings` (mirrors the existing
`add_generated_agent_image` pattern). Library-agent name + description +
instructions are embedded with the existing
`ensure_content_embedding(ContentType.LIBRARY_AGENT, ...)` into
`UnifiedContentEmbedding`, scoped by `userId`.
- **Backfill.** A new `LibraryAgentHandler` joins `CONTENT_HANDLERS`,
and `LIBRARY_AGENT` is appended to `backfill_all_content_types` so
existing library agents become discoverable on first run.
- **Hybrid search wrapper.** `hybrid_search_library_agents()` in
`backend/api/features/library/search.py` delegates to the existing
`unified_hybrid_search()` via the `db_accessors.search()` shim (so it
works whether Prisma is connected in-process or only via the
database-manager RPC service — same path `find_block` / `search_docs`
already use). Library-specific weights `(semantic=0.50, lexical=0.40,
category=0.0, recency=0.10)` and threshold `0.55`; `category` is zeroed
because LIBRARY_AGENT rows have no categories, and the lexical query is
keyword-extracted before being fed to `plainto_tsquery` so its
AND-of-terms doesn't zero out matches on long natural-language goals.
- **Tool upgrade.** `find_library_agent` gains `for_creation: bool` and
`goal_summary: str`. When `for_creation=true`, it returns matches as the
existing `AgentsFoundResponse` with each description prefixed by `[N%
match]` (using `combined_score`, not post-BM25 `relevance`, since BM25
goes negative for near-duplicate corpora).
- **Gate.** `require_library_check(session, tool_name)` in `helpers.py`
mirrors `require_guide_read`: bypassed in builder-bound sessions,
satisfied once `find_library_agent` has been called this session,
otherwise returns an `ErrorResponse` instructing the LLM to call it.
- **Wire-up.** `create_agent` calls the gate immediately after
`require_guide_read`, accepting an explicit `library_check_ack: bool`
parameter to bypass after explicit user confirmation. The
agent-generation guide (`agent_generation_guide.md`) documents the
workflow as the new step 1.

### Changes 🏗️

- New file `backend/api/features/library/embeddings.py` —
`schedule_library_agent_embedding()` fire-and-forget background task.
- New file `backend/api/features/library/search.py` —
`hybrid_search_library_agents()`, `LIBRARY_SIMILARITY_THRESHOLD = 0.55`,
library-specific `UnifiedSearchWeights`.
- New file `backend/copilot/tools/find_library_agent_test.py` — hybrid
mode (ranked results, no-matches, soft-fails on missing goal or DB
error, default substring path unchanged).
- New `LibraryAgentHandler` in
`backend/api/features/store/content_handlers.py` + registry entry;
`LIBRARY_AGENT` added to `backfill_all_content_types`.
- `backend/api/features/library/db.py` — schedules embedding on create +
version update.
- `backend/copilot/tools/find_library_agent.py` — new `for_creation` /
`goal_summary` parameters.
- `backend/copilot/tools/agent_search.py` —
`search_library_for_creation()` helper; soft-fails (`NoResultsResponse`)
on missing goal or backend errors so the chat UI never renders "Error
finding agents".
- `backend/copilot/tools/helpers.py` — `require_library_check()` gate.
- `backend/copilot/tools/create_agent.py` — new `library_check_ack`
parameter; gate call after the guide-read gate; updated tool
description.
- `backend/copilot/sdk/agent_generation_guide.md` — new step 1
documenting the create-time similarity check, distinguishing it from the
sub-agent-composition use of `find_library_agent`, and fixing the
pre-existing duplicate `8.` numbering.
- `backend/copilot/tools/_test_data.py` — `make_session()` accepts
`library_check=True/False` so tests can opt into exercising the gate.
- Tests added/updated across the touched modules (32 focused tests
passing locally).

No frontend changes (existing `AgentsFoundResponse` SSE rendering is
reused). No Prisma migration (`ContentType.LIBRARY_AGENT` and indices
already exist on `UnifiedContentEmbedding`).

### Checklist 📋

#### For code changes:
- [ ] I have clearly listed my changes in the PR description
- [ ] I have made a test plan
- [ ] I have tested my changes according to the test plan:
- [ ] `poetry run pytest backend/api/features/library/search_test.py
backend/api/features/library/embeddings_test.py
backend/copilot/tools/find_library_agent_test.py
backend/copilot/tools/create_agent_test.py
backend/copilot/tools/helpers_test.py::TestRequireLibraryCheck
backend/api/features/store/content_handlers_test.py` — 32 passed locally
  - [ ] `poetry run ruff check` clean on all touched files
- [ ] Local CoPilot run: empty-library user asks to create an agent →
`find_library_agent(for_creation=true)` returns `NoResultsResponse`,
`create_agent` proceeds (gate satisfied by the call)
- [ ] Local CoPilot run with 9 pre-existing `YouTube Video Summarizer`
duplicates: same goal returns 5 matches at 75–76% combined score; LLM
surfaces them via `AgentsFoundResponse`
- [ ] Builder-bound session (`metadata.builder_graph_id` set): gate is
bypassed
- [ ] Backfill verified via `backfill_all_content_types(50)` →
`get_embedding_stats()['by_type']['LIBRARY_AGENT'].coverage_percent`
rises from 0 to 100

#### For configuration changes:

- [ ] `.env.default` is updated or already compatible with my changes
- [ ] `docker-compose.yml` is updated or already compatible with my
changes
- [ ] I have included a list of my configuration changes in the PR
description (under **Changes**)

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: majdyz <zamil.majdy@agpt.co>