test: sweep the protocol attribute across every flow entry point (#14613)
* test: sweep the protocol attribute across every flow entry point
The flow span's protocol attribute answers the operator's first question when an error
rate moves: which door did this come through. A webhook failing and the canvas failing
are different incidents with different owners, and this attribute is the only thing that
separates them. Nothing pinned it end to end.
Two halves, because they fail differently.
The static gate reads the AST for the values the source can emit and compares them
against a declared list, in both directions: a new surface with no cell fails, and a
declared cell nothing emits any more also fails. That is what stops the table rotting
between releases, which is the usual fate of a sweep done once by hand.
It matches on call shape rather than on the identifier, because two shapes carry a
protocol -- execution_protocol("v1") at the entry point, and protocol="v2.background"
passed into _stream_event_frames and bound one level down. Reading only the first is how
the ticket's own table came to claim v2 background was unlabelled when it is not. The
identifier is also overloaded: stream_protocol and background_execution's protocol=
argument are the SSE wire format, unrelated, and a word-match sweeps them up.
The per-cell tests drive the real routes and read the exported span, which is what
catches a binding that exists in source but never reaches the span. Six here: v1,
v1.advanced, v2, v2.background, webhook, openai_responses. Four are already driven
elsewhere and are cited rather than duplicated.
CELL_EVIDENCE is the table itself, and it records the five cells that are NOT yet driven
end to end with the reason each is blocked, rather than omitting them. A sweep that
silently skips a surface reads as more coverage than it has.
Verified rather than assumed: mutating execution_protocol("v1") to a different value
fails three tests in three ways -- the undeclared-value gate, the stale-declaration gate,
and the live v1 cell reporting the mutant. All green on restore. 181 telemetry tests pass.
* fix(test): attach to the existing tracer provider instead of installing over it
The sweep installed its own provider and asserted it won the install. That works when the
module runs alone and fails the moment xdist puts another provider-installing module in the
same worker: set_tracer_provider is first-write-wins, so the second module's provider is
ignored and its exporter is fed by nothing.
The sibling PR hit exactly this on one Python version and not the others, which is the
signature of a scheduling-order dependency rather than a version problem.
Attaching a processor to whichever provider is already there has no ordering dependency.
The cells still cannot pass vacuously, because each asserts its own span arrived before
asserting anything about it.
Reproduced by running this file alongside test_build_flow_span.py, which installs a provider
of its own: red before, 17 passed after.
* fix(tests): exclude test files from the protocol scanner and check cited evidence
The scanner skipped only path components named exactly "test", so 32 test
files under the two scanned roots were being read. A test that binds a
made-up protocol would have registered as an emitted value and quietly
weakened the gate.
The evidence map also cited a path that was never checked, and one of them
was still a placeholder with an ellipsis in it. Add a test that resolves
every non-blocked evidence path and asserts the file exists.
* fix(tests): shut the attached span processor down on teardown
A tracer provider has no removal API. Clearing the exporter empties the list
but leaves the processor registered, so it kept appending every later span in
the worker to an exporter nothing reads. Probed: after clear() a later span
still lands, after shutdown() it does not.
Move the teardown into a finally block so it also runs when a test fails.
* test: close the remaining exit criteria in the runtime x protocol table
Three gaps against the sweep's own acceptance criteria.
The N/A cells were never tested. Every status assertion in the sweep was a
happy-path 200, so nothing checked that langflow-runtime routes are absent
from lfx serve. They now must answer 404 or 405 AND produce no flow span,
because a route that quietly accepts the call and runs nothing is the failure
worth catching, and a status check alone would not see it.
The server metric half of a passing cell was asserted nowhere in the repo. A
flow span says the run happened; the duration histogram says the surface in
front of it is instrumented at all. Now asserted end to end through a real
instrumented app.
lfx run had no live cell. It is the runtime with no server in front of it,
which makes the flow span the only record the run happened. Driven through
the real CLI, and asserted to record no HTTP server metric.
Every assertion here was mutation checked: each was made to fail against a
deliberately wrong expectation before being trusted green.
---------
Co-authored-by: Eric Hare <ericrhare@gmail.com> G
Gabriel Luiz Freitas Almeida committed
d73fc4049f07f3684a6fd7964868d6d3b347a4e7
Parent: 7299e14
Committed by GitHub <noreply@github.com>
on 8/19/2026, 5:20:46 PM