fix(a11y): scanner reported rules from no policy, inflating our violation counts (#14665)
* fix(a11y): scan against IBM policy so scanner counts match the compliance policy
The Python route scanner called checker.check(document) with no guideline
list, so it reported rules that belong to no accessibility policy
(rulesets: []) and inflated every violation count derived from it. The
Playwright suite was never affected -- .achecker.yml always carried
policies: [IBM_Accessibility].
- pass --policies (default IBM_Accessibility) through evaluate_ace and
record each issue's rulesets in the report so non-policy rules are
self-evident
- pin DEFAULT_ACE_URL to accessibility-checker-engine@4.0.26, matching
the Playwright suite, with a CI guard against future drift (advisory
on PRs)
- add a scroll state action so states files can scan post-scroll DOM
(reproduces the aria_child_tabbable failure on /settings/messages that
default-load scans cannot see); commit the states file
- wait for the messages grid or its empty-state alert before scanning
/settings/messages, removing a 6-vs-0 violation race in CI
- surface baselineSuppressedCount/rawViolationCount in route-summary.json
so the published headline split is generated, not hand-computed
- add a nightly-only python-route-scan job scanning real (unmocked)
routes; reporting only, never runs on PRs
* fix(a11y): let a spec declare a request it deliberately leaves in flight
`scans provider list loading state` mocks GET /api/v1/models with a 6000ms
delay so the loading state is still on screen when the a11y scan runs. The
teardown drain only waits API_REQUEST_DRAIN_TIMEOUT_MS (2000ms), so that
request can never settle in time and the test fails -- unless the test body
happens to burn the full 6s first. Fast machines fail, slow ones pass, which
is why CI stayed green while local sharded runs failed.
There was no way to express "this request is meant to still be pending", so
add one: page.expectPendingRequest({ method, path }). Unresolved requests are
matched against the declarations and only undeclared ones fail the test.
Also split the failure message. A pending-drain failure previously reported
"Server-error contract failed", which reads as a 4xx/5xx and sends you looking
for an error response that does not exist; it now says the requests did not
settle and points at the new helper.
Verified on shard 7/8 (--workers=3), where the failure reproduces:
before 24 passed + 1 failed, after 25 passed. With the declared path changed
to a non-matching one the test fails again, so the guard is narrowed rather
than disabled. V
Viktor Avelino committed
165519121db8774a76be27b4bb57975c721356dd
Parent: 24fea58
Committed by GitHub <noreply@github.com>
on 8/20/2026, 2:49:33 AM