ci: run both systemd scopes and tear down unconditionally (#336)
## Summary
The `systemd-service` job gated its system-scope step on `steps.probe.outputs.user_scope != 'true'`. On `ubuntu-latest` a per-user systemd manager is always present, so that condition never held and **the step had never executed once** since #309 added it. Everything inside it was unverified code sitting in the workflow, and that is the only reason three #309 acceptance criteria stayed unchecked. All three are reachable on a hosted runner through passwordless sudo.
Both scopes now run on every execution.
## What changed
**The `if:` gate is gone from the system-scope step**, and it is no longer described as a "fallback": it covers strictly more than the user-scope step rather than substituting for it. The probe survives, narrowed to its one honest use, gating the user-scope step on a runner that genuinely has no user manager. The system-scope step no longer consults it.
**Ordering is now deliberate rather than incidental.** User scope runs first because the system-scope step writes `/etc/all-smi/config.toml`, and that path is a discovery candidate for *every* all-smi process on the host (`LINUX_SYSTEM_CONFIG_PATH`, `src/common/paths.rs`). A user-scope daemon started after it would silently inherit port 19191, and the user-scope assertions would quietly be testing something other than what they claim. The scopes are otherwise disjoint: different unit paths, different managers, ports 9090 and 19191. This is written into the job header so the order is not "cleaned up" later.
**The config-file criterion is now proved, not implied.** Answering on 19191 would also pass for a daemon that ignored the file and happened to bind 19191, or that bound both. The step now also asserts the compiled default 9090 is *not* listening, which is what "the listener moves" actually means.
**New `if: always()` teardown step.** Every pre-existing cleanup line sat on a success path, so a step dying mid-lifecycle left an enabled unit, a daemon on 19191, and the config file behind. With both scopes running there is more state to strand. It removes the system unit, the user unit, `/etc/all-smi/config.toml`, `/etc/default/all-smi`, and an `all-smi` system user if one exists, then `daemon-reload`s and `reset-failed`s.
Two deliberate choices there. It is best-effort and always `exit 0`, because a teardown that fails turns a green run red for no reason and masks the real failure when the run was already red; it emits residue as `::warning::` and prints a positive "clean" line otherwise. And the `userdel` line is belt and braces: `all-smi service install` never runs `useradd` (`--service-user` only writes `User=` into the unit and expects the account to exist), so the criterion is now satisfied by construction rather than by that argument continuing to hold.
The `ERR` trap diagnostics pattern from #319 and #323 is untouched, and the new port-9090 check routes through the same `diagnose` function.
## Test plan
- [x] `yaml.safe_load` parses the workflow; step gating verified programmatically: only `User-scope lifecycle` carries a probe condition, `System-scope lifecycle` is unconditional, `Clean up all systemd state` is `if: always()`.
- [x] `bash -n` across every `run:` block in the systemd and launchd jobs: 0 syntax errors.
- [x] **Both scopes ran.** CI run [31103744522](https://github.com/lablup/all-smi/actions/runs/31103744522): `User-scope lifecycle` success, `System-scope lifecycle` success, `Refuse to clobber a foreign unit` success, `Clean up all systemd state` success. This is the first execution of the system-scope path in the project's history.
### The first run failed, and the failure was real
Run [31102859355](https://github.com/lablup/all-smi/actions/runs/31102859355) failed with `status=203/EXEC`, the journal saying only "Main process exited, code=exited, status=203/EXEC". The unit sets `ProtectHome=true` and `ExecStart` pointed at `/home/runner/work/all-smi/all-smi/target/debug/all-smi`, so systemd could not execute the binary at all. This is a genuine property of the hardened unit, not a CI artifact, and it had gone unnoticed for exactly the reason this issue exists: the step had never run.
Fixed in the workflow, not the unit. Relaxing `ProtectHome` would weaken every real deployment to accommodate a binary location no real deployment uses, and #309's criterion says "from a plain tarball install", which puts the binary in `/usr/local/bin`. The step now stages it there with `install -m 0755`, which makes it a *more* faithful test of the criterion. The `ExecStart` assertion also gained an explicit "not under /home" check so this failure mode is named in the log rather than arriving as an opaque 203.
Two things worth noting from that failed run: the `/-/ready` gate from #329 timed out correctly after 120 s and its diagnostics (empty `ss -lntp`, connection refused on 19191, the 203/EXEC status) pointed straight at the cause, and the `if: always()` teardown ran and succeeded even though the lifecycle step had died mid-flight, which is precisely the case it was added for.
### Per-criterion evidence
| #309 criterion | Evidence from run 31103744522 |
|---|---|
| `kill -9` restart within `RestartSec` | `sudo kill -9 4332`, then MainPID `4435` within `RestartSec=5`, then a readiness wait and a metric assertion. |
| Tarball install, `uninstall` leaves no unit | Staged to `/usr/local/bin/all-smi`; `sudo ... service install --now`; `systemctl is-enabled`; `status --json` → `"running": true`; `service uninstall` → "Removed the all-smi system service."; `test ! -f /etc/systemd/system/all-smi.service` passed. |
| `/etc/all-smi/config.toml` moves the listener | `[api] port = 19191` written; `/-/ready` and `all_smi_memory_total_bytes` served on 19191; default 9090 asserted **not** listening. |
All three are now ticked on **#309** with this evidence.
### #332: nothing ticked, deliberately
The brief for this work said #332's group A owns these same three items. It does not. #332's Scope section explicitly **excludes** them ("Three of the 19 items are already tracked by #330 ... Those three are excluded from the checklist below so the two issues do not overlap"), and its group A holds three different items: a real deb install, a non-systemd host, and a dpkg-managed binary refusing without `--force`. None of those is exercised by this job, so ticking anything there would have been false. #332 received an [evidence comment](https://github.com/lablup/all-smi/issues/332#issuecomment-5205053958) recording that its documented exclusion is now discharged, plus the `ProtectHome` finding since it bears on group A's deb item. It stays **OPEN** with **no `status:done`** and 0 of 16 verified.
### Wall-clock cost
`System-scope lifecycle` 11 s, `Clean up all systemd state` 1 s, so roughly **12 s of added step time**. Job totals: **173 s** before (run 31101864563, system-scope skipped) vs **151 s** after. The job is dominated by the cargo build and cache restore, and run-to-run variance there exceeds the cost of the added scope.
Closes #330 J
Jeongkyu Shin committed
4646da6cd7f492bc70a2cc163180ee234e0c683d
Parent: cdeccc8
Committed by GitHub <noreply@github.com>
on 8/6/2026, 1:11:44 PM