AI Agents
Fixed
- The model pods download their weights from Hugging Face without the Xet path (giantswarm/agent-platform#520; proof 1 of giantswarm/giantswarm#37639). On a Giant Swarm cluster a served model’s weights never arrived: KServe’s storage-initializer fetched the six small repository files over HTTPS from
huggingface.co and then hung forever — no error, no restart, 0 CPU, eight sleeping threads (gazelle 2026-09-17 02:14Z and 02:30Z, chart 4.28.15, huggingface_hub 0.32.4 + hf_xet 1.5.1 + hf_transfer 0.1.9 in the initializer image, HF_HUB_ENABLE_HF_TRANSFER=1, HF_XET_HIGH_PERFORMANCE=1, HF_XET_NUM_CONCURRENT_RANGE_GETS=8). hubble observe --verdict DROPPED on the pod showed every SYN to the Xet CDN (15.236.85.177:443, world) policy-denied, and the endpoint’s cilium fqdn cache held addresses for cas-server.xethub.hf.co, us.aws.cdn.hf.co and huggingface.co — not that one. The connectivity chart’s serving-namespace CiliumNetworkPolicies are a toFQDNs allow-list (huggingface.co, *.huggingface.co, *.hf.co, *.*.hf.co on 443): Cilium admits exactly the addresses its DNS proxy saw resolved for an allowed name, for the answer’s TTL; the Xet CDN (CloudFront) answers with ~30 s TTLs and rotating address sets, and hf_xet — a Rust client with its own resolver cache and one connection per shard range-GET — keeps connecting to addresses the proxy has forgotten or never saw, so the shards’ SYNs are dropped and the client waits on its connect timeouts without reporting. The allow-list stays as it is — a model download is no reason to open world on 443 to the serving pods, and a CIDR block for a CDN is a list that rots — and the client comes off the Xet path: the pods ClusterPolicy gains one strategic-merge rule per pod shape, model-pod-env-<shape>, that merges modelServing.policies.env (default HF_HUB_DISABLE_XET=1) by name onto the storage-initializer’s and the shape’s runtime container’s env (kserve-container / main; every model pod with a storage-initializer, named model or not; the containers’ own entries stay, so HF_HUB_ENABLE_HF_TRANSFER=1 is kept). With Xet off, huggingface_hub downloads the shards with hf_transfer from the LFS fronts (cdn-lfs*.hf.co, cdn-lfs.huggingface.co — within the patterns) through the system resolver: every connection is a DNS query the proxy sees, and its address is admitted. The values document the mechanism, that widening the egress is not the fix, and the cluster-level alternative the chart does not own — a tofqdns-min-ttl on the Cilium agent long enough for the client’s cache, after which an installation may set env: [] to have Xet back; a mirror’s HF_ENDPOINT belongs in the same list. make verify-model-serving-policies (kyverno apply of the rendered policy over a fixture pod of each shape, the initializer now carrying KServe’s three HF variables and the classic runtime the ClusterServingRuntime’s two) holds the entry next to the containers’ own env on both containers of both shapes, added where a container has none, on a pod without a model name too, a no-op over the policy’s own output (webhook reinvocation), the download Job’s pod and a pod without an initializer untouched, no env rule with the list empty, and the mutated pod still passing the fleet’s restricted PSS with the chart’s exception. The live half — the 8 GB of qwen3-4b-instruct on the claim, the predictor Ready and /v1/chat/completions answering, a second load downloading nothing — is the proof’s. model-manager’s own download Job takes the same path on its side.
Fixed
- The model pods pass the fleet’s restricted Pod Security Standard, and the chart adds no container to them (giantswarm/agent-platform#518; proof 1 of giantswarm/giantswarm#37639). The connectivity chart’s
model-serving pods policy injected hf-cache-init into every model pod — root, CHOWN/DAC_OVERRIDE/FOWNER added, chmod 0777 of the cache subdirectory — and the fleet’s Kyverno set denied the pod on two rules the chart’s PolicyException does not name (disallow-capabilities-strict/adding-capabilities-strict, require-run-as-non-root-user/run-as-non-root-user): on gazelle (4.28.14) the LLMInferenceService’s ReplicaSet was refused on every attempt (validate.kyverno.svc-fail, ReplicaFailure=True), no pod, no node, nothing served — the first time the container reached admission at all (#506 made the rule match these pods, #514 made it idempotent). The init container is removed, neither excepted nor made rootless: the redirect-model-storage-* rules give the pod the claim’s fsGroup next to the claim mount (modelServing.cache.fsGroup, default 1000 — the uid and fsGroup model-manager’s cache pods use; a pod that declared another fsGroup gets the claim’s, one claim, one group), and the kubelet does the rest — it makes the claim’s contents group-owned, group-writable and setgid on every mount (the CSI driver’s fsGroupPolicy permitting: EBS gp3 and Azure Disk apply it) and creates a missing subPath directory with the claim root’s mode and group, so <claim>/<model> is root:1000 2775 and the storage-initializer (uid 1000), the runtime and model-manager’s download Job (uid 1000, fsGroup 1000) write it, mkdir -m 0777 -p it and rm -rf it without root or a capability (the kubelet’s exact sequence replayed in a container; without the fsGroup uid 1000 is denied, which is what the init container had papered over). modelServing.policies.cacheInit (image, resources) is gone. The chart’s PolicyException is unchanged: it names the four rules the vLLM image and the storage-initializer violate, and nothing the chart adds needs one. make verify-model-serving-policies applies the fleet’s five restricted-PSS ClusterPolicies (tests/fixtures/restricted-pss-clusterpolicies.yaml: kyverno-policies 3.7.2 as installed on a management cluster, every rule Enforce) with the chart’s exception over the mutated pod of both shapes and holds every rule to pass or skip (the four excepted rules skip, adding-capabilities-strict and run-as-non-root-user pass), the bare pod to fail exactly the four excepted rules, and a pod with the 4.28.14 init container to fail exactly the two that denied it on gazelle; it holds the mutated pod’s fsGroup and that the mutation adds no container. The live half — the workload pod admitted on the first ReplicaSet attempt and the predictor Ready — is the proof’s.
Fixed
muster call – and every other command that connects to an aggregator – fails fast with auth_required instead of opening a browser, and a rejected token is no longer removed from the store. Without a usable token a command now exits with status 2 at once; the error starts with auth_required, names muster auth login --context <ctx> (or --endpoint <url>) and the new --login flag, which opens the browser from the command itself for a person at a terminal (--auth auto and MUSTER_AUTH_MODE=auto still do the same; the default --auth mode is now none, and a stored token is used and refreshed in every mode). A script, an agent or a CI job can never complete a browser flow, and inside a tool call the old behaviour hung until killed. The 401 handling of the commands, of auth status and of the agent’s re-authentication no longer deletes the token file first: several muster processes share ~/.config/muster/tokens/, the ID token in the file serves other clients, and a sign-in replaces the file only when it completes. The store writes each token file atomically (temporary file and rename), so a process reading an endpoint’s file while another refreshes it sees the old or the new token and never a truncated file it would take for a missing one; every write and removal is logged at debug with the pid and the command words of the process. auth logout removes the current context’s token only, --all every token. (#1273)
Fixed
- The
add-cache-init pod mutation is idempotent under Kyverno webhook reinvocation (giantswarm/agent-platform#514; proof 1 of giantswarm/giantswarm#37639). The connectivity chart’s model-serving pods policy added hf-cache-init with a JSON patch (add /spec/initContainers/0) and no guard against being applied twice. Kyverno’s pod mutation webhook is registered with reinvocationPolicy: IfNeeded, and the API server reinvokes it whenever another mutating webhook changed the pod after Kyverno ran — the case for every LLMInferenceService workload pod on a Giant Swarm cluster — so the rule inserted the container a second time and the pod was refused by validation: on gazelle (4.28.11) the ReplicaSet of the served preset failed nine times in a row with spec.initContainers[2].name: Duplicate value: "hf-cache-init" while the GPU node ran empty (the two redirect-model-storage-* rules, strategic merges, were idempotent; before #506 the rule never reached these pods). add-cache-init is now a strategic merge keyed by the container’s name: the container is added the first time and found on a reinvocation, and Kyverno’s strategic merge leads a list with the entries it patched, so hf-cache-init stays ahead of the storage-initializer the way the insert at index 0 put it — the rule stays last in the policy, since a later merge on the storage-initializer would move that one ahead again. Audit of the chart’s other mutation rules: agent-sandbox-pod-security’s add-controller-security-context merges by (name) and +(securityContext) anchors, and no rule of either chart appends to a list with a JSON patch; nothing else to change. make verify-model-serving-policies now applies the rendered policy to its own output for both pod shapes and asserts a no-op (one hf-cache-init, one hf-cache volume) — the check failed on the previous rule with exactly the duplicate seen on gazelle. The live half — the workload pod admitted on the first ReplicaSet attempt with hf-cache-init, the hf-cache mount and the 4Gi initializer limit — is the proof’s.
Fixed
make live-serving-slice reads the controller’s jwks-store in the shape the controller writes it (giantswarm/agent-platform#515; proof 1 of giantswarm/giantswarm#37639): tests/verify-serving-slice-live.py asserted a positive nkeys on the jwks-store-<hash> entry, a field the store has never had — the entry is {requestKey, url, fetchedAt, jwks} as the controller’s jwks package persists it, jwks the fetched JWKS document as a JSON string — so against a healthy controller (gazelle 2026-09-16 23:41Z: models-jwt Accepted=Valid, two keys for the in-cluster Dex including the token’s kid) the one live signal #505 introduced failed. The check now parses jwks and counts its keys, printing their kids (a 401 token uses the unknown key "<kid>" names the one to compare with), fails naming the URL on an empty set as before, and fails naming the ConfigMap when jwks is not a JWKS document; the policy’s Accepted.reason stays the primary signal. make verify-serving-slice-store (tests/verify-serving-slice-store.py) holds the reader over tests/fixtures/jwks-store-configmaps.json, a copied store in the writer’s shape (names and request keys derived as the controller derives them: the issuer with two keys, an issuer with an empty set, a labelled ConfigMap without the entry) with kubectl stubbed — pass, empty set, unknown URL, malformed document; verify-all runs it.
Fixed
- klaus-gateway reaches the kube-apiserver for its team-review endpoint on the bolt link store (giantswarm/agent-platform#511; towards giantswarm/klaus-gateway#273): klaus-gateway >= 1.11.0 verifies every caller of
POST /reviews and POST /notices with an authentication.k8s.io/v1 TokenReview, but the connectivity chart granted the gateway pod the kube-apiserver entity only for the Secret link store (klausGateway.obo.store: secret with OBO on), so an installation that keeps its links in the bolt store and turns klausGateway.reviews.enabled on had no rule to the API server in its Cilium default-deny egress — every TokenReview timed out and the endpoint answered 503 to every caller (the OBO policy’s world/cluster :443 allowance never reaches the API server, which Cilium’s socket-LB resolves to the control plane before the policy applies). agent-platform.klausGateway.apiServerEgress is truthy also while klausGateway.reviews.enabled is true, so -klausgateway-store-egress carries toEntities: [kube-apiserver] (cilium) or the networkPolicy.kubernetes.apiServerCIDR ipBlock (kubernetes) whatever the link store — one rule, rendered once next to the Secret store’s. The connectivity chart declares klausGateway.reviews with the klaus-gateway chart’s defaults (enabled: false, audience: klaus-gateway, allowedCallers: []); the default shape renders byte-identically. make verify-klausgateway-netpol holds the rule on the bolt store in both flavours, its absence with reviews off, and that reviews next to the Secret store render the rule once; ci/test-klausgateway-reviews-values.yaml lints the shape. Lab-verified in agentlab with the connectivity chart of the PR: the in-cluster gateway with reviews.enabled and an allow-list answered POST /notices from the allowed ServiceAccount with 201 and from a ServiceAccount outside the list with 403.
Fixed
- Plain
muster auth login renews an expired stored ID token. The renewal decision hung on the handler’s status path, which reaches Authenticated only when the stored access token is valid for another minute by the CLI’s own reading and otherwise probes the server; the mcp-go transport connects with that same token regardless, so a session the aggregator accepted answered Already authenticated while the decision never saw the ID token (muster auth status then said No authentication required). The decision now reads the ID token’s exp from the token file alone, before the “already authenticated” answer and without a probe; a session without an ID token is renewed too. (#1268)
Fixed
- The agentgateway controller can fetch the models Gateway’s JWKS from the public issuer, and a failed fetch is visible (giantswarm/agent-platform#505; proof 1 of giantswarm/giantswarm#37639). The controller — not the data plane — fetches the JWKS of every
AgentgatewayPolicy and pushes the keys to the data plane over xDS; a fetch its network policy denies pushes an empty key set, and every token is refused 401 token uses the unknown key while the data plane has nothing to log. On gazelle the serving slice’s models-jwt policy (JWKS from dex.<installation> on 443, the chart’s default) was served by the platform release’s controller, whose policy admitted the API server, muster and the in-cluster Dex on 5556 and nothing public: error fetching jwks … dial tcp <edge LB>:443: i/o timeout, every id_token refused until the backend was pointed at the in-cluster Dex by hand. Two changes: (1) global.identity.issuerUrl’s host is always an external JWKS target of the controller policy (agent-platform.jwks.issuerTarget, joined into agent-platform.jwks.externalTargets): opened by name on 443 (the URL’s own port when it carries one) behind the DNS proxy clause in the cilium flavour, as port 443 of the wide rule in the kubernetes flavour, in every release that runs the controller and whatever the routes name — the controller serves the JWT policies of every release in the cluster, and the slice’s models policy beside the platform’s release is one the platform’s policy cannot see; an in-cluster issuer adds nothing (gateway.jwksEgress covers it). (2) The controller policies render wherever the controller runs (agent-platform.agentgateway.controller: an agentgateway-* ingress mode, or components.agentgateway on): the serving slice on a workload cluster (components.agentgateway.enabled: true, ingress.mode at its muster-direct default, no edge Gateway) rendered no controller policy at all, though its own controller serves the models Gateway there. The chart’s comments (models-gateway.yaml, the jwks values, gateway.jwksEgress, docs/authentication.md) now name the controller as the fetcher and its egress as the requirement. Visibility: make live-serving-slice NAMESPACE=<the slice's> (tests/verify-serving-slice-live.py) reads the models-jwt policy’s Accepted condition on every ancestor and the controller’s jwks-store entry for the backend’s URL, and fails naming the URL and the controller’s error fetching jwks lines on reason: PartiallyValid or an empty key set — the check the gazelle proof runs before a token is sent. make verify-serving-slice asserts the slice-on-workload-cluster controller policy in both flavours (rendered without an ingress mode; the issuer on 443 with the DNS proxy; an own jwks.host beside it; none beside the platform’s release); make verify-wiring asserts a public issuer is opened from the controller policy with and without a route policy of the release’s own, on the URL’s port, that an in-cluster issuer adds no rule, and holds the all-in-cluster shape byte-identical to origin/main. Every installation with the controller on and a public issuer gets one controller policy change: a toFQDNs rule for the issuer on 443 and the DNS proxy variant of the DNS rule (cilium), or 443 on the wide rule (kubernetes); no pod rolls.
Fixed
- The model-serving pod policies match
LLMInferenceService workload pods (giantswarm/agent-platform#506): the connectivity chart’s model-serving Kyverno mutations (redirect-model-storage, storage-initializer-memory, add-cache-init, the Deployments’ progress-deadline), its network policies and the predictors’ PolicyException selected pods by serving.kserve.io/inferenceservice, the classic InferenceService predictor’s label; the workload pod the llm-d controller creates from an LLMInferenceService carries kserve.io/component=workload, app.kubernetes.io/part-of=llminferenceservice and app.kubernetes.io/name=<model> instead and serves from a container named main, so none of it applied — on gazelle (4.28.8, proof 1 of giantswarm/giantswarm#37639) the predictor composed from qwen3-4b-instruct downloaded 8 GB into an emptyDir with its storage-initializer OOMKilled twelve times at KServe’s 1Gi default, while the policies stood Ready beside it. The two pod shapes are one list now (agent-platform.modelServing.podShapes: selector, model-name label, runtime container), and every selector of the serving namespace renders from it: the mutations match both shapes, take the model’s name from whichever label the pod carries (the cache subdirectory), anchor the /mnt/models cache mount on the shape’s runtime container (kserve-container or main; an anchored entry never adds a container) and skip a pod that names no model; the network policies render one policy per shape (-model-serving-llmisvc-workload[-ingress|-egress] next to -predictor…) and the kagent agents’ egress admits both; the PolicyException matches both shapes’ Pods, ReplicaSets and Deployments by label next to the *-kserve* name. make verify-model-serving-policies (tests/verify-model-serving-policies.py) runs kyverno apply of the rendered policies over a fixture pod of each shape (tests/fixtures/model-serving-*-pod.yaml, the llm-d one as observed on gazelle) and asserts the claim mount and subPath on the initializer and the runtime, the 4Gi limit, hf-cache-init first, a pod without an initializer and a download-Job pod untouched, the Deployments’ deadline, and that the network policies, the agents’ egress and the PolicyException select exactly the same pods; the CI job installs the kyverno CLI for it. The live half — the workload pod’s volumes on a GPU cluster and a second load of the same preset downloading nothing — is the proof’s.
Added
muster auth login renews the stored ID token. A valid session is still reused, but the session’s automatic refresh renews the access token only: the OIDC ID token in ~/.config/muster/tokens/<hash>.json kept the exp of the first sign-in, so the CLI carried an ID token expired for weeks – signed with a key the identity provider no longer publishes – while muster auth status said Authenticated. Now login signs in again through the browser when the stored ID token has expired (or expires within 30 s), and always with the new --force; the stored session is replaced only when the new sign-in completes. muster auth status and muster auth whoami show the ID token’s expiry (ID token: expired 12 days ago (renew with: muster auth login)), and the new muster auth token [--id] prints the access token or the ID token alone on stdout for other clients – a script calling an API that validates the person’s ID token – and refuses an expired ID token instead of handing it out. (#1268)
