AI Agents
Fixed
- An OAuth server’s
spec.timeout reaches the client a tool call builds on a pool miss. #1284 gave the login-time client of a server with auth.type: oauth the server’s timeout; the client the tool-call path builds when a session is already authenticated but holds no pooled connection – after a restart of the aggregator, after an eviction – was still built without it (and without spec.meta), so the first call after such a restart was cut at the default 30 s although the server declared more. Both are handed to that client too. (#1283)
Changed
components.kserve-runtime-configs follows the 0.3.x line (giantswarm/kserve#82). kserve-runtime-configs 0.3.0 adds kserve.llmisvcConfigs.images.<preset>.<container>, a per-config image override beside imageRegistry; its default render is byte-identical to 0.2.4’s, and the meta chart’s kserve-runtime-configs: block passes nothing new. The BOM example pins 0.3.0.
Fixed
modelServing.prepull.nodeSelector set by an installation renders alone (giantswarm/agent-platform#562). Karpenter’s karpenter.k8s.aws/instance-gpu-manufacturer: nvidia was the values default in both charts, and Helm merges maps: a selector an installation set rendered next to the default key — a DaemonSet no node matched — and a null for the key was coalesced away when the meta chart forwarded its values, so the README’s “or another selector here” was not reachable, and an installation whose GPU nodes are not launched by Karpenter stripped the key with a post-renderer. Both values files now carry nodeSelector: {} and the template holds the default (agent-platform.modelServing.prepull.nodeSelector): empty renders Karpenter’s label, a set map renders alone, the pool’s label (gpuPool.nodeSelector) is merged under either; a non-string label value fails the render naming the key. tests/verify-mirrored-values.py counts an empty mapping as a leaf, so a copy that kept the map is drift; make verify-model-serving-policies asserts the selector cases, the guard, and a selector set on the meta chart reaching the DaemonSet alone. README “Pre-pulling the runtime image”; UPGRADE.md.- The pre-pull DaemonSet is a hook object: an image that cannot be pulled yet no longer fails the connectivity upgrade (giantswarm/agent-platform#563). As a release resource the DaemonSet was in the release’s wait, and a release’s wait counts a DaemonSet ready by its pods: helm-controller from 1.6 waits with the kstatus poller, which needs every pod of a DaemonSet Ready (Helm 3’s legacy waiter, still on helm-controller 1.4, takes
maxUnavailable: 100% as ready once the pods are scheduled — the fleet runs both). A pod whose image cannot be pulled is never Ready, so one listed image not yet pullable — a runtime image or a prepull.modelPresets image still being built, a registry outage — held the pods in ImagePullBackOff, timed the wait out and marked the whole connectivity release failed and retried, every other object applied. The DaemonSet now carries helm.sh/hook: post-install,post-upgrade,post-rollback (weight 0, before-hook-creation): created after the release’s objects are applied and waited for, waited for by nobody — a hook’s WatchUntilReady watches Jobs and Pods only, in Helm 3, Helm 4 and helm-controller alike — so a missing image leaves the pods retrying on the kubelet’s backoff and the release Ready; the next event replaces the DaemonSet and the pods come back on images already present. A hook object is not Helm’s to delete on uninstall, so a pre-delete hook Job (<release>-model-serving-prepull-cleanup, the hook include agent-platform.hooks.job; the identity <release>-hooks is created for that event too, with delete on exactly that DaemonSet) removes it. The deny-all policy stays a release resource. make verify-model-serving-policies asserts the hook shape, the cleanup Job, the identity’s rule and that prepull.enabled: false renders neither; verify-serving-slice holds the identity to the pre-delete event alone with the cache off; verify-gpu-pool leaves the DaemonSet out of the golden comparison while origin/main renders it as a release resource. README “Pre-pulling the runtime image”; UPGRADE.md.
Changed
- The connectivity chart README names the GPU node’s locked-memory requirement (giantswarm/agent-platform#564): a container inherits containerd’s
RLIMIT_MEMLOCK and has no CAP_IPC_LOCK, so a serving runtime that mlock()s its weights dies at engine start under the unit’s 8 MB default; a pool node created through the platform runs containerd with LimitMEMLOCK=infinity (gpu-node-pool’s memlock.conf drop-in), and a GPU node not created through the platform needs the same drop-in. No rendered object changes.
Fixed
- The storage-initializer’s memory limit is
8Gi, headroom for a download that overlaps the runtime image’s unpack (giantswarm/agent-platform#576). modelServing.policies.storageInitializerMemoryLimit defaulted to 4Gi, sized after KServe’s 1Gi default was OOM-killed by an 8 GB download; the limit also holds the page-cache pages of the files the initializer writes, which the kernel charges to the container’s cgroup until write-back has drained them to the disk. On a 4 vCPU / 16 GiB L4 node the GPU became allocatable about 77 s before the runtime pre-pull had finished gunzipping the 8.8 GB image, so a 9.4 GB download into the emptyDir on the node’s local disk ran inside a CPU-bound unpack (load average 30, write-back saturated), its dirty pages stayed charged to the initializer and the init container was OOM-killed (exit 137) a minute in; the same download had passed whenever it started after the pre-pull, and a 16 vCPU node with the same overlap was fine. The zstd re-layered runtime image (giantswarm/agent-platform#568, 4.40.0) unpacks in about 30 s and ends the pre-pull before the GPU appears, which removes the collision on the L4 size; 8Gi is the headroom for a download that still overlaps an unpack (a bigger model, a slower disk). It fits the smallest curated L4 size: init and main containers do not run at once, so the pod’s effective request stays the predictor’s own 10Gi, within the ~12 GiB a 16 GiB node leaves to pods after the kubelet’s reservations. The meta chart mirrors the default; the README says what the limit must hold.
Fixed
- An MCPServer in sync is left alone. In Kubernetes mode the reconciler logged
Reconciling MCPServer: <name> for every server about every 14 s – some 250 passes per server per hour, each a definition read and a status write against the API server, while nothing changed. Two periodic sources of the same length ran out of phase: every successful pass requeued itself after 30 s (the status sync from before the periodic resync existed), and the Manager’s resync enqueued every server every 30 s on top. A pass that finds the server in sync now asks for nothing further – the next one comes from a definition change, a runtime state change (StateChangeBridge) or the resync – and the resync is a safety net for a lost event rather than a heartbeat: its default is ten minutes instead of 30 s. MUSTER_RECONCILER_RESYNC_INTERVAL still overrides it and muster test still runs instances at 2 s. DefaultStatusSyncInterval is gone. (#1285)
Fixed
- An OAuth server’s
spec.timeout reaches the client a person’s grant connects. #1282 wired the budget into the token-exchange and token-forwarding clients only; the client establishConnection builds for a server with auth.type: oauth – the core_auth_login callback and the grant reuse on first use – kept the zero timeout, so a tool call on such a server was cut after the default 30 s (no answer within the server's timeout of 30s) although its definition declared more. The entry’s spec.timeout is read with its spec.meta and handed to the DynamicAuthClient (new WithTimeout) and to the static-bearer client alike. (#1283)
Fixed
- Triton’s kernel cache lives on the cache claim beside vLLM’s compiled graphs: the redirect rule sets
TRITON_CACHE_DIR=/mnt/vllm-cache/triton on the runtime container (giantswarm/agent-platform#572). Triton — the compiler behind every @triton.jit kernel a model runs, the GDN and Mamba kernels of the hybrid architectures among them — keeps its compiled kernels and, with vLLM’s default TRITON_CACHE_AUTOTUNING=1, its autotuning results in ~/.triton/cache inside the container, gone with the pod. A preset that compiles had them on the claim already: at compile init vLLM redirects TRITON_CACHE_DIR in-process to a directory under VLLM_CACHE_ROOT’s torch_compile_cache/. A preset that serves --enforce-eager — six of the shipped ones — never compiles, so that redirect never ran and every pod recompiled and re-autotuned its kernels during the profiling run. The redirect rule now sets TRITON_CACHE_DIR=/mnt/vllm-cache/triton in the same patch as VLLM_CACHE_ROOT: a directory of the claim’s .vllm-cache beside torch_compile_cache/, which Triton creates as the runtime’s uid under the setgid root:1000 2775 directory exactly as vLLM creates torch_compile_cache/; a compiled preset is unchanged (vLLM’s redirect takes over at compile init, the env is the floor for what Triton compiles before it), and a pod without the cache keeps Triton’s default as it keeps vLLM’s. make verify-model-serving-policies holds both envs on the runtime container of both shapes, both under the cache mount, and neither on a pod without the cache. README “vLLM’s compile cache and Triton’s kernel cache on the claim”.
Changed
- The serving slice’s llm-d images come from the
llm-d-fast/ prefix, and the pre-pull names that image (giantswarm/agent-platform#568). The well-known LLMInferenceServiceConfigs and the pre-pull DaemonSet named the runtime image as mirrored byte-identically, gsoci.azurecr.io/giantswarm/llm-d-cuda:v0.8.0 — 8.2 GiB compressed in gzip layers of 5.5, 1.4 and 1.0 GB. containerd pulls one HTTPS stream per layer, so the stream carrying the 5.5 GB layer bounded the pull: about 60 MB/s effective, 132–149 s on a fresh pool node where one stream measures 95 MB/s and four 306 MB/s, and the operands pulled beside it ran three to five times slower meanwhile — with the pre-pull in place, the longest cold-start step left. giantswarm/llm-d publishes the same images re-layered under llm-d-fast/ of the same registry path (zstd, llm-d-cuda split into layers of at most 1.2 GB). The meta chart’s kserve-runtime-configs: block now passes kserve.llmisvcConfigs.imageRegistry: gsoci.azurecr.io/giantswarm/llm-d-fast/, so every reference the configs render carries the prefix, and modelServing.prepull.images defaults to gsoci.azurecr.io/giantswarm/llm-d-fast/llm-d-cuda:v0.8.0 in both charts, so the pre-pull warms exactly the image the predictor runs. Going back is two values: the block’s imageRegistry at gsoci.azurecr.io/giantswarm/ (the byte-identical mirror) or ghcr.io/llm-d/ (upstream), and the matching reference in prepull.images. make verify-serving-slice asserts the prefix on the rendered release and that the pre-pull’s reference carries it; verify-gpu-pool leaves the pre-pull DaemonSet out of the golden comparison while origin/main names the mirror’s image. README “The well-known configs” and “Pre-pulling the runtime image”; UPGRADE.md.
Changed
- The cache claim’s defaults are
100Gi at 500 MiB/s and 3000 IOPS; the default class is named with a digest of its parameters, and an existing claim keeps its class and its size (giantswarm/agent-platform#570). The claim defaulted to 500Gi on a gp3 class at 1000 MiB/s and 4000 IOPS — sized far above what the serving slice uses: the shipped presets hold 8–9 GiB (L4) and 25 GiB (L40S) of weights each plus a few GiB of compiled graphs, so two served models fit in well under 40 GiB, and the volume is billed at its size from the first bind; vLLM loads a model’s weights at about 300 MB/s on a 1000 MiB/s volume exactly as on a 500 MiB/s one (the loader is the ceiling, measured on two GPU instance sizes) and the storage-initializer writes at the Hub’s ≈ 200 MB/s, so only the baseline tier bounds either. At list prices the old volume cost about three and a half times the new one a month for the same load and download times. modelServing.cache.pvc.size is now 100Gi, modelServing.cache.storageClass.parameters throughput: "500", iops: "3000" (the tier’s included IOPS), mirrored in the meta chart. Two mechanics made the change safe for an installation that already has the claim: a StorageClass’s provisioner and parameters are immutable, so the default class name now carries eight hex characters of their sha256 (agent-platform-connectivity-hf-cache-<digest>; agent-platform.modelServing.cacheStorageClass.digest) and a parameter change renders a new class, the old Helm-owned one going with the upgrade instead of a forbidden update failing the release (a name of the operator’s carries no digest); and the hook applies an existing claim on the class it has (storageClassName is immutable) at the larger of its size and the rendered one (quantities compared in bytes; a claim never shrinks), its log line naming what was kept — a Pending claim on a class the cluster no longer has fails the hook naming the claim and the way out (delete it; the next upgrade applies it on the rendered class). make verify-serving-slice asserts the defaults and the digest name, that a parameter change renders a new class the claim references, and runs the hook’s script against a stub kubectl over the cases (a Bound 500Gi claim on the former class kept as it is, a smaller one grown, an equal size in another spelling left alone, the Pending-on-a-missing-class refusal, a claim without a class or on the empty class). README “The Hugging Face cache claim” and “The claim’s StorageClass”; UPGRADE.md.
Fixed
- Preset
qwen3-8-flash-next-nvfp4: memory limit 118Gi (was 64Gi) (giantswarm/agent-platform#553). The B12X serving stack keeps the resident weights as mlock()ed managed memory of the engine process — file-backed pages the memory cgroup charges to the container, unlike the cudaMalloc device memory other runtimes hold outside the cgroup — so the first live serve was OOM-killed by the container limit at 61 GiB of file RSS while the node had 100 GiB free. The limit now covers the resident weights (about 74 GiB with the n-gram table read from disk, up to the 99 GiB checkpoint) plus the runtime; the request stays at 48Gi so the predictor schedules on a node that also hosts platform pods. The same stack needs the node’s containerd to run with LimitMEMLOCK=infinity (it mlock()s every weight allocation and a container has no CAP_IPC_LOCK): giantswarm/agent-platform#564.
Fixed
- The serving namespace is kept on uninstall whatever the cache switch says, so a slice composed without the cache no longer deletes a cache claim an earlier slice left there (giantswarm/agent-platform#565). The connectivity chart put
helm.sh/resource-policy: keep on the serving namespace only while modelServing.cache.enabled was on; a slice composed with the cache off rendered the namespace as an ordinary Helm-owned object, and its uninstall deleted the namespace — and with it the hf-cache claim an earlier slice (cache on) had applied there, the claim’s own keep policy notwithstanding (a namespace Helm deletes takes every object in it along). On an installation the claim was still Pending, so nothing was lost; a Bound claim’s volume (the class reclaims with Delete) would have gone, with the weights and compiled graphs on it. Now the namespace carries the policy whenever the chart creates it; the new modelServing.namespace.keep (default true) is the one way to the old behaviour — false renders the namespace without the policy, and the namespace and everything in it, a cache claim included, go with the release. The meta chart mirrors the block it forwards, and make verify-meta holds the two copies equal (tests/verify-mirrored-values.py). README “What an uninstall leaves behind” states the lifecycle in one paragraph, independent of the cache switch: the namespace outlives the release, the claim outlives the pool, and removing either is a person’s explicit act. make verify-serving-slice asserts the policy with the cache on and off and its absence with namespace.keep: false.
