[CI] Check vLLM PR compatibility with vllm-ascend (#14560)
### What this PR does / why we need it?
This PR adds a source-only vLLM interface compatibility check to
vllm-ascend and exposes it through the existing
`tests/e2e/vllm_interface` pytest collection. The vLLM Ascend NPU job
already checks out vllm-ascend and runs this
directory, so vLLM PRs can receive vllm-ascend interface feedback
without first adding a new Buildkite job.
The analyzer is consumer-first. It discovers contracts that the
checked-out vllm-ascend source actually consumes,
resolves those contracts independently at this PR's base and head
commits, and attributes a failure to this PR
only when the base endpoint was compatible and the head endpoint is not.
This package has one fixed purpose: the vLLM `vllm-interface` PR check.
It does not expose a main2main scenario.
Monkey-patch installation analysis, patch-flow analysis,
inheritance-only findings, and broad generator reviews are
intentionally outside this PR. Ordinary imports, verified overrides, and
exact calls remain detectable even when the
consumer source happens to be located under a vllm-ascend `patch/`
directory.
#### End-to-end execution path
```mermaid
flowchart TD
A["Existing vLLM Ascend NPU pytest job"] --> B["test_vllm_pr_interface_compatibility"]
B --> C["Resolve merge-base and PR HEAD"]
C --> D["Verify vLLM and vllm-ascend SHAs"]
D --> E["Parse complete source indexes"]
E --> F["Resolve inheritance and C3 MRO"]
F --> G["Generate verified override relations"]
E --> H["Discover direct imports"]
E --> I["Discover exact vllm-ascend calls to vLLM"]
G --> J["Compare PR base/head override contracts"]
H --> K["Compare PR base/head symbol presence"]
I --> L["Compare PR base/head call and return contracts"]
J --> M["Merge, deduplicate, and classify findings"]
K --> M
L --> M
M --> N["Render summary in memory"]
N --> O["Print summary to the pytest Job Log"]
O --> P["PASS or pytest failure"]
```
All implementation and validation code is colocated under the existing
test directory:
```text
tests/e2e/vllm_interface/
├── vllm_interface_contracts/ # source analyzer and CLI
├── test_vllm_pr_interface_compatibility.py
├── singlecard/ # existing NPU sampler test
└── README.md
```
#### 1. CI entry and exact PR range
`test_vllm_pr_interface_compatibility.py` looks for the vLLM checkout at
`/workspace/vllm`. It skips only when
that checkout is absent, which keeps ordinary vllm-ascend local runs
usable. If the checkout exists without Git
metadata, or the analyzer cannot complete a valid analysis, the test
fails rather than reporting a false PASS.
`vllm_ci.py::resolve_vllm_range` derives this PR's exact range from Git
evidence:
```text
pr_head = git rev-parse HEAD
fetch vllm-project/vllm main into FETCH_HEAD
pr_base = git merge-base pr_head FETCH_HEAD
require pr_base to be an ancestor of pr_head
```
A shallow checkout is unshallowed and retried when necessary. If a Git
command fails while resolving the range, the adapter reports the exit
code, complete command, stdout, and stderr directly in the pytest Job
Log. The adapter also records the exact vllm-ascend `HEAD`.
`analyze_range` verifies all selected SHAs before reading source.
The executed command is intentionally limited to this CI use case:
```bash
python -m tests.e2e.vllm_interface.vllm_interface_contracts analyze-range \
--vllm-root /workspace/vllm \
--ascend-root <checked-out-vllm-ascend> \
--old <merge-base-sha> \
--new <vllm-pr-head-sha> \
--expect-ascend-sha <vllm-ascend-head-sha> \
--fail-on introduced \
--analysis-workers 3 \
--index-workers 4
```
#### 2. Fixed analysis scope
| Capability | State | Role in this PR |
| --- | --- | --- |
| Inheritance / C3 MRO | `prerequisite` | Proves override ownership;
does not emit inheritance-only findings. |
| Overrides | `analyzed` | Checks parameters, descriptors, and
constrained replacement-return contracts. |
| Direct imports | `analyzed` | Checks exact vLLM module and symbol
presence. |
| Exact vllm-ascend calls to vLLM | `analyzed` | Checks observed call
arguments and constrained return use. |
| Monkey patches | `skipped` | Patch installation and dispatch proof are
not included. |
| Broad generator reviews | `skipped` | Ambiguous candidates do not fail
a vLLM PR. |
The summary prints this capability boundary so reviewers can see what
was and was not analyzed.
#### 3. Source indexing and in-job parallelism
`generator.py::RepositoryIndex` parses Python with `ast`; it never
imports or executes vLLM or vllm-ascend and does
not require CUDA or NPU hardware. It records import bindings, class
bases and methods, callable signatures, descriptor
kinds, statically recoverable return contracts, receiver evidence, and
source locations.
Every CI run parses the complete selected source trees. The vLLM Python
files are grouped into bounded batches and
parsed by a `ProcessPoolExecutor`; the CI entry requests four index
workers. The parent process merges fragments in
sorted source order and reruns global class-variant, star-import,
dataclass, callable-alias, and consistency
finalization. The vllm-ascend index is built directly in the parent
process.
This PR does not create, read, or restore persistent pickle/SQLite index
data. That state would not be reused by the
current ephemeral job container without an additional trusted persistent
volume. The process-parallel source parsing
and the three analysis branches described below remain enabled.
#### 4. Dependency discovery and comparison
##### Verified overrides
Inheritance is resolved first because override proof depends on the
combined vLLM/vllm-ascend C3 MRO. An override
relation is emitted only when the vllm-ascend class, vLLM owner,
replacement method, and descriptor/signature evidence
can be resolved without guessing. Transitive vllm-ascend subclasses
retain their impact locations and override
paths while repeated relationships are deduplicated.
##### Direct imports
Executable vllm-ascend imports are resolved as exact dependencies on
vLLM modules or symbols. Base/head resolution uses
Git rename evidence where available and does not infer relocation from a
similar name.
##### Exact vllm-ascend calls to vLLM
`call_contracts.py::DirectCallDetector` emits a dependency only when a
callsite can be tied to one vLLM callable.
Supported evidence includes imported aliases, proven `self`/`super`
paths, constructed or annotated receivers, exact
positional/keyword calls, and Triton launch syntax such as
`kernel[grid](...)`.
The observed call shape is bound independently to the base and head
callable signatures. Dynamic `*args`, `**kwargs`,
ambiguous receivers, and incomplete MROs remain unresolved instead of
being guessed.
##### Constrained return protocols
Return checks cover statically proven consumption such as fixed
unpacking, literal subscripts, iteration, `await`, and
context-manager use. Transparent `return super().same_method(...)`
follows the vLLM return protocol. Unused,
forwarded, escaping, or dynamically transformed return values do not
create a return break without a concrete use
constraint.
##### Deleted vLLM targets
A new-tree index cannot normally resolve a method deleted by the PR. The
override and direct-call collectors retain
narrow historical candidates. During comparison, a candidate is promoted
only when the old snapshot proves the
vLLM callable and the complete new MRO proves it is absent. This
preserves detection of verified deleted
`self.method(...)`, `super().method(...)`, and override targets without
broad same-name matching.
#### 5. Parallel comparison and classification
After shared indexing and relation generation, the analyzer runs three
independent branches through a
`ThreadPoolExecutor`:
1. override relation comparison;
2. direct-import comparison; and
3. direct-call discovery/comparison.
Each branch uses isolated base/head `GitSnapshot` caches. Results are
merged in a fixed order and sorted by a stable key,
so `--analysis-workers 1` and the default parallel path produce the same
findings.
Classification is based on this PR's base/head transition:
| Old state | New state | Classification |
| --- | --- | --- |
| compatible | incompatible | `introduced_break` |
| incompatible | compatible | `fixed` |
| incompatible | incompatible | `preexisting` |
| compatible | compatible, contract changed | `compatibility_warning` |
| required evidence unknown | unknown | `analysis_unresolved` |
An item becomes actionable only when the dependency is verified, the
relevant contract changed, the runtime path is
statically reachable, and the exact version lane matches. Optional-only
override additions remain review input unless
an exact vLLM call and dispatch path proves the new argument reaches the
vllm-ascend implementation.
#### 6. Job Log and exit behavior
The report is rendered in memory and printed directly to the existing
pytest Job Log. It includes exact SHAs,
introduced/review counts, root causes, source locations, and concise
compatibility-impact descriptions.
The CI path does not create JSON, CSV, Markdown, or metadata files,
upload Buildkite artifacts, or create a separate
Buildkite annotation.
Exit behavior:
- exit 0: this PR introduced no actionable break;
- exit 1: an actionable `introduced_break` was found and pytest fails;
and
- any other non-zero exit: invalid input/range or analyzer failure.
The existing Ascend NPU job is currently soft-fail, so this integration
provides PR-level awareness rather than a
required merge gate.
### Does this PR introduce any user-facing change?
No. Runtime inference behavior and public APIs are unchanged. This PR
adds CI-only source analysis and tests.
### How was this patch tested?
```bash
python -m ruff check tests/e2e/vllm_interface
python -m ruff format --check tests/e2e/vllm_interface
for python_version in 3.10 3.11 3.12; do
python -m mypy --follow-imports skip --check-untyped-defs \
--python-version "${python_version}" \
tests/e2e/vllm_interface/vllm_interface_contracts \
tests/e2e/vllm_interface/test_vllm_pr_interface_compatibility.py
done
python -m pytest -q \
--confcutdir=tests/e2e/vllm_interface \
tests/e2e/vllm_interface/test_vllm_pr_interface_compatibility.py
```
Local results for the latest revision:
- Ruff lint and format checks: passed.
- Mypy with Python 3.10, 3.11, and 3.12 targets: passed with no issues.
- E2E adapter outside the vLLM Ascend NPU image: **1 skipped** as
designed.
- `compileall` and `git diff --check`: passed.
- Git range failure-path validation: passed; pytest shows the failed
command, exit code, stdout, and stderr without a duplicated subprocess
traceback.
- Full pytest execution through the CI entry against vLLM PR #50685: **1
passed** in 808.25 seconds.
- Historical vLLM PR #39568 replay: exit 1 with the expected single P1
direct-call break for deleted
`SchedulerInterface._get_routed_experts`; output matches the result
before cache removal.
The trusted local-cache prototype is preserved separately on
`zhao-stack/vllm-ascend:codex/vllm-interface-pickle-cache-backup-20260825`.
It is not part of this PR or its CI path.
- vLLM main:
https://github.com/vllm-project/vllm/commit/ba07e4a48fc951300d97eb506217dd530583dea3
---------
Signed-off-by: shenzhao <shenzhao9@huawei.com>
Signed-off-by: zhao-stack <80399320+zhao-stack@users.noreply.github.com>
Co-authored-by: shenzhao <shenzhao9@huawei.com> Z
zhao-stack committed
d4fc06f08ebcc7a934e569dd9f546155d52d375e
Parent: d20eb0b
Committed by GitHub <noreply@github.com>
on 8/29/2026, 3:27:27 AM