KEP-766: DisaggregatedSet implementation (#773)

* KEP-766: Add DisaggregatedSet controller implementation

This adds the reference implementation for KEP-766 DisaggregatedSet.

Signed-off-by: Mathis Felardos <mathis@mistral.ai>

* feat(disaggregatedset): propagate user labels/annotations to LWS and Service

Signed-off-by: Mathis Felardos <mathis@mistral.ai>

* feat(disaggregatedset): Replace ServiceTemplate with automatic headless portless Services

This change simplifies the DisaggregatedSet API by removing the user-facing
ServiceTemplate field and automatically creating headless portless Services
for pod discovery via EndpointSlices.

API Changes:
- Remove ServiceTemplate from DisaggregatedSet spec
- Services are now created automatically without user configuration

Service Behavior:
- Headless Services (clusterIP: None) are created for each side (prefill/decode)
- Services are portless to enable EndpointSlice-based discovery
- Services are only created when both sides have ReadyReplicas >= 1
- Old services are cleaned up only after workloads are fully drained

Testing:
- Add e2e test verifying automatic Service and EndpointSlice creation
- Add Kind cluster cleanup in AfterSuite to prevent image caching issues
- Fix .dockerignore to use explicit exclusions

Also fixes typo in Makefile helm target path.

Signed-off-by: Mathis Felardos <mathis@mistral.ai>

* feat(disaggregatedset): Rename "side" terminology to "phase" throughout codebase

This is a breaking change that standardizes terminology:
- Label: disaggregatedset.x-k8s.io/side → disaggregatedset.x-k8s.io/phase
- Type: DisaggSideConfig → DisaggregatedPhaseSpec
- Constants: SidePrefill/SideDecode → PhasePrefill/PhaseDecode
- Struct: SideReplicaState → PhaseReplicaState
- Map field: Sides → Phases

YAML field names (spec.prefill, spec.decode) remain unchanged.

Updated files:
- API types and CRD manifests
- All controller code and tests
- E2E tests with updated label selectors
- CLI tool (plan-steps)
- README documentation
- Helm chart CRD

Signed-off-by: Mathis Felardos <mathis@mistral.ai>

* feat(disaggregatedset): Refactor API from hardcoded prefill/decode to flexible phases array

This commit introduces a flexible N-phase architecture for DisaggregatedSet,
replacing the hardcoded prefill/decode fields with a dynamic phases array.

API Changes:
- Replace Prefill/Decode pointer fields with Phases []DisaggregatedPhaseSpec
- Add CEL validation requiring >= 2 phases with unique names
- Update PhaseReplicaState to use dynamic slices instead of fixed arrays

Controller Changes:
- Refactor executor to use dynamic slices for N-phase support
- Update planner to work with arbitrary phase counts
- Update service manager for N-phase workloads

CLI Changes (plan-steps):
- Accept JSON maps for phase config: --source '{"prefill": 6, "decode": 2}'
- Dynamically generate table headers from phase names
- Fully N-phase ready (planner still has 2-phase limitation)

Testing:
- Add 3-phase rolling update e2e test
- Add phase rename e2e test
- Update all existing tests for new API

This is a breaking API change for v1alpha1.

Signed-off-by: Mathis Felardos <mathis@mistral.ai>

* feat(disaggregatedset): Add phasePolicy field and progressive rollout for phase changes

Introduces a phasePolicy field (Strict/Flexible) to control how DisaggregatedSet
handles phase additions, removals, and renames during updates.

API Changes:
- Add PhasePolicy type with Strict (default) and Flexible values
- Strict: rejects adding, removing, or renaming phases
- Flexible: allows phase changes with progressive rollout

Controller Changes:
- Detect phase changes by comparing spec vs old workload phases
- Pass removed phases to Planner with target=0 for progressive drain
- Compute allPhaseNames as union of spec phases + old workload phases
- Clean up ALL phases belonging to drained revisions, not just spec phases
- Refactor executor.go into logical sections with focused helper functions
- Move verbose investigation logs to V(1) debug level

Testing:
- Add unit tests for phasePolicy enforcement (Strict blocks, Flexible allows)
- Update e2e tests for progressive drain behavior
- Add 3-phase sample manifests

File reduced from 658 lines to 507 lines while maintaining all functionality.

Signed-off-by: Mathis Felardos <mathis@mistral.ai>

* feat(disaggregatedset): Refactor e2e tests with kubectl fluent builder and fixtures packages

Introduces reusable test utilities to simplify e2e test code and improve maintainability.

New Packages:
- test/utils/kubectl: Fluent builder API for kubectl commands
  - kubectl.go: Chainable methods (Get, Delete, Apply, Label, JSONPath, etc.)
  - queries.go: Higher-level helpers (LWSByPhase, CountPods, GetTotalReplicas)
  - waiters.go: Eventually-based wait helpers (ForPodCount, ForRevisionDrained)
- test/utils/fixtures: YAML builders for DisaggregatedSet manifests
  - fixtures.Phase and fixtures.Config for flexible YAML generation
  - fixtures.PrefillDecode helper for common 2-phase configs

Test Refactoring:
- Replace inline helpers with kubectl package imports
- Use fluent builder for all kubectl operations
- Consolidate YAML builders into fixtures package
- Add TrackProgressiveRollout helper for rollout tests

Code Reduction:
- executor.go: 507 → 456 lines
- e2e_test.go: 1669 → 1004 lines (~40% reduction)

Signed-off-by: Mathis Felardos <mathis@mistral.ai>

* feat(disaggregatedset): Port planner fix: source-aware surge baseline and maxUnavailable floor

Port fixes from commit 7c66541 to the N-phase planner:

1. Source-aware surge baseline:
   - Change surge constraint from old + new <= target + surge to
     old + new <= max(source, target) + surge
   - Ensures scale-down scenarios don't block scale-up since the system
     already runs source replicas

2. maxUnavailable floor enforcement:
   - Add minOld constraint: old + new >= target - maxUnavailable
   - Only enforce when source >= target (scale-down scenario)
   - Applied to both proportional drain and fallback drain paths

Test updates:
- Update expected sequences for scale-down and mixed-scale scenarios
- Add asymmetric_5_3_surge2 test case

All tests pass with 90.2% coverage.

Signed-off-by: Mathis Felardos <mathis@mistral.ai>

* feat(disaggregatedset): Verify labels and annotations propagate to pods

The existing test only verified that labels and annotations were set on the
LWS workerTemplate. This extends the test to also verify that they are
actually propagated to the running pods by LWS.

Adds PodsByPhase helper to kubectl queries for querying pods by phase.

Signed-off-by: Mathis Felardos <mathis@mistral.ai>

* feat(disaggregatedset): Add Metadata field for Kueue/exclusive-topology support

Adds a metadata field to DisaggregatedPhaseSpec that allows users to set
labels and annotations on the LWS CR's ObjectMeta. This enables:
- Kueue queue assignment (kueue.x-k8s.io/queue-name label)
- LWS exclusive-topology scheduling

User-provided labels/annotations are merged onto the LWS CR, with system
labels taking precedence over user labels.

Includes e2e test verifying metadata propagation to LWS CR.

Signed-off-by: Mathis Felardos <mathis@mistral.ai>

* feat(disaggregatedset): Add disaggregatedset tests to make test and test-e2e targets for prow CI

Signed-off-by: Mathis Felardos <mathis@mistral.ai>

* feat(disaggregatedset)!: Remove PhasePolicy field

BREAKING CHANGE: The phasePolicy field has been removed from
DisaggregatedSetSpec. The controller now always uses Flexible behavior,
allowing phase additions, removals, and renames during rollouts.

- Remove PhasePolicy type, constants, and spec field
- Remove getPhasePolicy() and rejectPhaseChanges() controller logic
- Remove all PhasePolicy unit tests
- Update fixtures and sample YAMLs
- Remove PhasePolicy from e2e tests

Phase changes are now always allowed (Flexible behavior is the default).

Signed-off-by: Mathis Felardos <mathis@mistral.ai>

* fix(disaggregatedset): Fix Kind not found in e2e tests

- Add KIND ?= kind default in disaggregatedset Makefile
- Pass KIND=$(KIND) from parent Makefile to disaggregatedset test-e2e

This fixes the "Kind is not installed" error when running
disaggregatedset e2e tests from the parent LWS Makefile.

Signed-off-by: Mathis Felardos <mathis@mistral.ai>

* feat(disaggregatedset): Add hack/e2e-test.sh for Prow CI compatibility

Add a dedicated e2e test script that mirrors the LWS pattern for running
e2e tests in Prow CI. The script handles the full e2e lifecycle:

- Kind cluster creation/deletion with cleanup trap
- Docker image building and loading to Kind
- LWS controller installation from release manifests
- DisaggregatedSet operator deployment via kustomize
- Running ginkgo tests with junit output
- Log collection on cleanup for debugging

Update Makefile to call the hack script with all required environment
variables (KIND, KUBECTL, KUSTOMIZE, GINKGO, etc.) and add ginkgo tool.

Update e2e tests to skip redundant operations when run via hack script
(detected via LWS_INSTALL_SKIP=true), allowing tests to work both
standalone and in Prow CI.

* feat(disaggregatedset): Embed LeaderWorkerSetSpec in DisaggregatedPhaseSpec

* feat(disaggregatedset): Refactor RollingUpdateConfig and split ComputeNextStep

- Change RollingUpdateConfig from slice fields to per-phase structs
  ([]RollingUpdateConfig instead of RollingUpdateConfig with []int fields)
- Simplify extractRollingUpdateConfig by removing unused specPhaseSet param
- Split ComputeNextStep into focused helpers: isComplete, isNewAtTarget,
  canScaleUp, computeMinOld, tryScaleUp, tryProportionalDrain, tryForceDrain
- Remove config() test helper in favor of direct struct literals

* chore(disaggregatedset): Move plan-steps utility to hack/

* docs(disaggregatedset): Update READMEs for phases[] API and N-dimensional rollouts

* fix(disaggregatedset): Remove redundant phase count validation

The MinItems=2 constraint on the phases array is sufficient.
Removed redundant checks:
- Controller: runtime len(phases) < 2 check
- CEL rule: size(self.phases) >= 2

The API server rejects invalid resources at admission time via MinItems=2.

* fix(disaggregatedset): Disable tablewriter auto-format to preserve phase names

The tablewriter library by default converts headers to title case and
adds spaces around dashes (e.g., "decode-long-context" becomes
"DECODE - LONG - CONTEXT"). Disable this by setting WithHeaderAutoFormat
to tw.Off BEFORE WithHeader, since the Header() method defaults to
AutoFormat=On if not explicitly set beforehand.

* fix(disaggregatedset): fix a weird behavior in the DisaggregatedSet planner when dealing with imbalanced updates.

* Rename phase to role in DisaggregatedSet API

- Rename DisaggregatedPhaseSpec → DisaggregatedRoleSpec
- Rename PhaseStatus → RoleStatus
- Rename spec.phases → spec.roles
- Rename status.phaseStatuses → status.roleStatuses
- Rename all labels from disaggregatedset.x-k8s.io/phase
  to disaggregatedset.x-k8s.io/role
- Rename all internal variables, comments, and function parameters
- Update sample YAML files and rename test files
- Regenerate CRDs and deepcopy code

* fix(e2e): Wait for webhook readiness before running tests

Add a webhook readiness check in the "Operator Deployment" test that
waits for the validating webhook to be ready before proceeding with
other tests. This fixes flaky test failures where the webhook service
wasn't ready yet when the "Webhook Validation" test ran.

The check uses a dry-run server-side apply to verify the webhook
responds, retrying for up to 2 minutes.

---------

Signed-off-by: Mathis Felardos <mathis@mistral.ai>