SIGN IN SIGN UP

fix(memory-storage): stop re-adds from duplicating and resetting pending requests (#2074)

- Store pending requests in an `OrderedDict` keyed by unique key instead
of a `deque`. Repositioning a request to the forefront is now O(1)
instead of an O(n) `deque.remove()` scan, so re-adding K already-pending
requests is O(K) instead of O(K*N) — measured 22.4s → 0.004s for 2000
forefront re-adds into a 20k-request queue. The scan only triggered for
requests already waiting in the queue, which is what `add_requests(...,
forefront=True)` hits when a handler re-discovers URLs that are still
pending.
- Fix the same URL being handed out repeatedly. The old scan silently
failed once an earlier regular re-add had replaced the registered
request object with a differently-valued one, leaving a stale entry
behind and appending a second one for the same unique key. Five URLs
re-added over three rounds yielded 20 fetches instead of 5 and left
`pending_request_count` at -15. Duplicates are now impossible by
construction.
- Keep the originally enqueued request on a re-add. The old code
replaced the registered request with the incoming duplicate, which on a
forefront re-add also replaced the queued one — resetting `retry_count`
(a request could retry forever), `label` (dispatch to a different
handler) and `user_data`. On a regular re-add it left the two out of
sync, so `get_request` and `fetch_next_request` disagreed. This restores
the documented contract ("Duplicates will be identified but not re-added
to the queue") and matches the file-system, SQL and Redis clients, as
well as `@crawlee/memory-storage` in Crawlee for JS.
- Make `get_request` consistent after a reclaim: it now returns the
reclaimed object, the same one `fetch_next_request` hands back.
- Write the queue metadata once per batch instead of once per new
request, saving N-1 `datetime.now()` calls on a batch add.

*✍️ Drafted by Claude Code*

---------

Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>
A
Anas Khan committed
fae204bc2c4557072fe9e13c3ec0cfaf49aa4717
Parent: 9acf0c4
Committed by GitHub <noreply@github.com> on 7/29/2026, 6:49:56 AM