[inductor][cudagraphs] Make cached-tensor liveness see pending backwards (#194124)
Fixes `Detected N tensor(s) in the cudagraph pool not tracked as outputs` (and its worse sibling, silent gradient corruption) in multi-graph training workloads. **Root cause.** `check_refcount` — the liveness predicate for cached output tensors — deliberately discounted one C++-side reference (`_use_count() > 1 -> return refcount == 3`). Autograd's `SavedVariable`, a pending backward's only grip on the forward's saved activations, is exactly one C++-side reference. So a forward's path could read all-dead mid-generation and be abandoned while its backward was still pending. The backward then recorded as a root in a foreign tree; the path-scoped `_is_cuda_graph_recorded_tensor` misclassified its pool-resident saved activations as persistent static inputs (parameters), outputs aliasing them were deliberately untracked, and `check_memory_pool` raised — or, when the allocator had already re-handed the activation memory to another recording, replays silently overwrote the activations and training produced NaN. The invisibility dates to #98944 (`refcount == 2` could never see C++ holders under the old refcount scheme); the PyObject-preservation rework (#167564) would have accidentally fixed it (a C++ holder now induces a Python incref) but the compatibility branch restored the old behavior. **Fix**, in `check_refcount`: 1. A user-dropped cached output (`refcount == 2`, the exact cache-only baseline) with stale `grad_fn` is `detach_()`ed. This dissolves the cache's vacuous self-pin (cached output -> grad_fn -> SavedVariables -> pins sibling cached outputs forever), the same stale-per-run-autograd-state scrub `reconstruct_outputs` already does for `_backward_hooks`. A genuine pending backward survives via the user's loss tensor's own edge to the node. `refcount == 2` provably implies no C++ holder exists (preservation increfs the PyObject on `use_count` 1->2), guarded additionally by `_use_count() == 1` and a GIL check (free-threaded refcounts are biased; the destructive detach fails toward "live" there). 2. After that, `_use_count() > 1` is a real external C++ holder — the pending backward — and the output is LIVE, so the path is never abandoned mid-generation and the backward records as a child of its forward. 3. `CUDAGraphNode.all_outputs_are_dead` evaluates to a fixpoint comparing consecutive live sets: one output's detach can release an output an earlier index already reported live, and a fresh pin appears each iteration, so any single fixed-order pass stays permanently one collapse behind (unbounded recording growth in multi-graph inference without this). Behavior changes (all silent-wrongness -> correct-or-loud): pending backwards hold their paths (gradient accumulation now keeps the N microbatches of activations its gradients actually require; previously all N backwards read the last microbatch's activations); `mark_step_begin` mid-pending-backward still overrides liveness (prior outputs declared dead; the cached output impl is rebound to the next replay), so differentiating a stale handle raises autograd's "not have been used in the graph" error instead of silently corrupting (pinned by `test_mark_step_overrides_pending_backward`); weakref-resurrected dropped outputs see cleared `grad_fn`. This diff was authored with an AI assistant. Differential Revision: [D116650568](https://our.internmc.facebook.com/intern/diff/D116650568/) Pull Request resolved: https://github.com/pytorch/pytorch/pull/194124 Approved by: https://github.com/Microve
E
eellison committed
7d84b26b7e0e5c47f3773cbda9b272e111afdde7
Parent: 13e1315
Committed by PyTorch MergeBot <pytorchmergebot@users.noreply.github.com>
on 8/24/2026, 8:38:17 PM