This is the first Azure release featuring Kubernetes 1.23.
Furthermore, this release is the first to use out-of-tree controller manager and CSI providers.
Upgraded from version 2.24.1.
FlexVolume is deprecated. Out-of-tree CSI driver is the recommended way to write volume drivers in Kubernetes.
See this doc for more information.
Maintainers of FlexVolume drivers should implement a CSI driver and move users of FlexVolume to CSI.
Users of FlexVolume should move their workloads to CSI driver.
Kubernetes releases are now generating provenance attestation files describing the staging and release phases of the release process and artifacts are verified as they are handed over from one phase to the next.
This final piece completes the work needed to comply with Level 1 of the SLSA security framework (Supply-chain Levels for Software Artifacts).
Version 2 of the HorizontalPodAutoscaler API graduates to stable in the 1.23 release. The HorizontalPodAutoscaler
autoscaling/v2beta2 API is deprecated in favor of the new
autoscaling/v2 API, which the Kubernetes project recommends for all use cases.
The generic ephemeral volume feature moved to GA in 1.23.
This feature allows any existing storage driver that supports dynamic provisioning to be used as an ephemeral volume with the volume’s lifecycle bound to the Pod.
All StorageClass parameters for volume provisioning and all features supported with PersistentVolumeClaims are supported.
The feature to configure volume permission and ownership change policy for Pods moved to GA in 1.23.
This allows users to skip recursive permission changes on mount and speeds up the pod start up time.
The feature to allow CSI Drivers to declare support for fsGroup based permissions graduates to GA in 1.23.
Structured logging reached its Beta milestone. Most log messages from kubelet and kube-scheduler have been converted. Users are encouraged to try out JSON output or parsing of the structured text format and provide feedback on possible solutions for the open issues, such as handling of multi-line strings in log values.
The kube-scheduler is adding a new, simplified config field for Plugins to allow multiple extension points to be enabled in one spot.
multiPoint plugin field is intended to simplify most scheduler setups for administrators.
Plugins that are enabled via
multiPoint will automatically be registered for each individual extension point that they implement.
For example, a plugin that implements Score and Filter extensions can be simultaneously enabled for both.
This means entire plugins can be enabled and disabled without having to manually edit individual extension point settings.
These extension points can now be abstracted away due to their irrelevance for most users.
CSI Migration enables the replacement of existing in-tree storage plugins such as
kubernetes.io/aws-ebs with a corresponding CSI driver.
If CSI Migration is working properly, Kubernetes end users shouldn’t notice a difference.
After migration, Kubernetes users may continue to rely on all the functionality of in-tree storage plugins using the existing interface.
Data corruption issue was found in etcd v3.5.0 release that was shipped with 1.22 Kubernetes release. Please read up-to-date production recommendations for etcd.
(beta feature) If the CSI driver supports the NodeServiceCapability
VOLUME_MOUNT_GROUP and the
DelegateFSGroupToCSIDriver feature gate is enabled, kubelet will delegate applying FSGroup to the driver by passing it to NodeStageVolume and NodePublishVolume, regardless of what other FSGroup policies are set. (#106330, @verult) [SIG Storage]
Add a new
distribute-cpus-across-numa option to the static
CPUManager policy. When enabled, this will trigger the
CPUManager to evenly distribute CPUs across NUMA nodes in cases where more than one NUMA node is required to satisfy the allocation. (#105631, @klueska)
Add fish shell completion to kubectl. (#92989, @WLun001)
Add mechanism to load simple sniffer class into fluentd-elasticsearch image (#92853, @cosmo0920)
Add support for Portworx plugin to csi-translation-lib. Alpha release
Portworx CSI driver is required to enable migration.
This PR adds support of the
CSIMigrationPortworx feature gate, which can be enabled by:
- Adding the feature flag to the kube-controller-manager
- Adding the feature flag to the kubelet config:
CSIMigrationPortworx: true (#103447, @trierra) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scalability, Scheduling, Storage, Testing and Windows]
Add support to generate client-side binaries for windows/arm64 platform (#104894, @pacoxu)
Added PowerShell completion generation by running
kubectl completion powershell. (#103758, @zikhan)
Processing condition for the
Shutdown for the
workqueue API to wait until the work queue finishes processing all in-flight items. (#101928, @alexanderConstantinescu)
Added a new feature gate
CustomResourceValidationExpressions to enable expression validation for Custom Resource. (#105107, @cici37)
Added a new flag
kubectl proxy that will automatically append the kube context server path to each request. (#97350, @FabianKramm)
Added ability for
kubectl wait to wait on arbitary JSON path (#105776, @lauchokyip)
Added support for
PodAndContainerStatsFromCRI feature gate, which allows a user to specify their pod stats must also come from the CRI, not
cAdvisor. (#103095, @haircommander)
Added support for setting controller-manager log level online. (#104571, @h4ghhh)
Added the ability to specify whether to use an RFC7396 JSON Merge Patch, an RFC6902 JSON Patch, or a Strategic Merge Patch to perform an override of the resources created by
kubectl run and
kubectl expose. (#105140, @brianpursley)
Adding option for
kubectl cp to resume on network errors until completion, requires tar in addition to tail inside the container image (#104792, @matthyx)
Adding support for multiple
--from-env-file flags. (#104232, @lauchokyip)
Adding support for multiple
--from-env-file flags. (#101646, @lauchokyip)
--as-uid flag to
kubectl to allow uid impersonation in the same way as user and group impersonation. (#105794, @margocrawf)
Adds new [alpha] command ‘kubectl events’ (#99557, @bboreham)
Allow node expansion of local volumes. (#102886, @gnufied)
Allow to build kubernetes with a custom
kube-cross image. (#104185, @dims)
Allows users to prevent garbage collection on pinned images (#103299, @wgahnagl) [SIG Node]
CRI v1 is now the project default. If a container runtime does not support the v1 API, Kubernetes will fall back to the v1alpha2 implementation. (#106501, @ehashman)
CSIMigrationAWS to on by default. This feature requires the AWS EBS CSI driver to be installed. (#106098, @wongma7)
DeleteOptions down to the fake client
Reactor (#102945, @chenchun)
Cloud providers can set service account names for cloud controllers. (#103178, @nckturner)
Display Labels when kubectl describe ingress. (#103894, @kabab)
VolumeBinding plugin to handle Lost PVC as
UnschedulableAndUnresolvable (#105245, @yibozhuang)
Ensures that volume is deleted from the storage backend when the user tries to delete the PV object manually and the PV
ReclaimPolicy is set to
Delete. (#105773, @deepakkinni)
NewUnstructuredExtractor from apply configurations
meta/v1 package that enables extracting objects into unstructured apply configurations. (#103564, @kevindelgado)
StorageObjectInUseProtection has been deprecated and cannot be disabled. It will be completely removed in 1.25 (#105495, @ikeeip)
apiserver_response_sizes metrics to stable. (#106122, @rezakrimi) [SIG API Machinery, Instrumentation and Testing]
schedule_attempts_total metrics to stable. Also
e2e_scheduling_duration_seconds is renamed to
scheduling_attempt_duration_seconds and the latter is graduated to stable. (#105941, @rezakrimi) [SIG Instrumentation, Scheduling and Testing]
Health check of kube-controller-manager now includes each controller. (#104667, @jiahuif)
Integration testing now takes periodic Prometheus scrapes from the etcd server.
There is a new script ,
hack/run-prometheus-on-etcd-scrapes.sh, that runs a containerized Prometheus server against an archive of such scrapes. (#106190, @MikeSpreitzer) [SIG API Machinery and Testing]
Introduce a feature gate
DisableKubeletCloudCredentialProviders which allows disabling the in-tree kubelet credential providers.
The feature gate
DisableKubeletCloudCredentialProviders is currently in Alpha, which means is currently disabled by default. Once this feature gate moves to beta, in-tree credential providers will be disabled by default, and users will need to migrate to use external credential providers. (#102507, @ostrain)
Introduces a new metric:
admission_webhook_request_total with the following labels: name (string) - the webhook name, type (string) - the admission type, operation (string) - the requested verb, code (int) - the HTTP status code, rejected (bool) - whether the request was rejected, namespace (string) - the namespace of the requested resource. (#103162, @rmoriar1)
Kubeadm: add support for dry running
kubeadm join. The new flag
kubeadm join --dry-run is similar to the existing flag for
kubeadm init/upgrade and allows you to see what changes would be applied. (#103027, @Haleygo)
Kubeadm: do not check if the
/etc/kubernetes/manifests folder is empty on joining worker nodes during preflight (#104942, @SataQiu)
Kubectl will now provide shell completion choices for the
--output/-o flag (#105851, @marckhouzam)
Kubelet should reconcile
kubernetes.io/arch labels on the node object. The side-effect of this is kubelet would deny admission to pod which has nodeSelector with label
kubernetes.io/arch which doesn’t match the underlying OS or arch on the host OS.
- The label reconciliation happens as part of periodic status update which can be configured via flag
--node-status-update-frequency (#104613, @ravisantoshgudimetla) [SIG Node, Testing and Windows]
Kubernetes is now built with Golang 1.16.7. (#104199, @cpanato)
Kubernetes is now built with Golang 1.17.1. (#104904, @cpanato)
Kubernetes is now built with Golang 1.17.2 (#105563, @mengjiao-liu)
Kubernetes is now built with Golang 1.17.3 (#106209, @cpanato) [SIG API Machinery, Cloud Provider, Instrumentation, Release and Testing]
ConfigurableFSGroupPolicy to GA and rename metric
volume_apply_access_control (#105885, @gnufied)
GetAllocatableResources Endpoint in PodResource API to the beta that will make it enabled by default. (#105003, @swatisehgal)
WindowsHostProcessContainers feature to beta (#106058, @marosset)
Node affinity, Node selectors, and tolerations are now mutable for Jobs that are suspended and have never been started (#105479, @ahg-g)
Pod template annotations and labels are now mutable for Jobs that are suspended and have never been started (#105980, @ahg-g)
PodSecurity: in 1.23+ restricted policy levels, Pods and containers which set
runAsUser=0 are forbidden at admission-time; previously, they would be rejected at runtime (#105857, @liggitt)
Shell completion now knows to continue suggesting resource names when the command supports it. For example
kubectl get pod pod1 <TAB> will suggest more Pod names. (#105711, @marckhouzam)
Support to enable Hyper-V in GCE Windows Nodes created with
kube-up (#105999, @mauriciopoppe)
The CPUManager policy options are now enabled, and we introduce a graduation path for the new CPU Manager policy options. (#105012, @fromanirh)
The Pods and Pod controllers that are exempted from the PodSecurity admission process are now marked with the
pod-security.kubernetes.io/exempt: user/namespace/runtimeClass annotation, based on what caused the exemption.
The enforcement level that allowed or denied a Pod during PodSecurity admission is now marked by the
The annotation that informs about audit policy violations changed from
pod-security.kubernetes.io/audit-violation. (#105908, @stlaz)
/openapi/v3 endpoint will be populated with OpenAPI v3 if the feature flag is enabled (#105945, @Jefftree)
CSIMigrationGCE feature flag is turned
ON by default (#104722, @leiyiz)
DownwardAPIHugePages feature is now enabled by default. (#106271, @mysunshine92)
PodSecurity admission plugin has graduated to
beta and is enabled by default. The admission configuration version has been promoted to
pod-security.admission.config.k8s.io/v1beta1. See https://kubernetes.io/docs/concepts/security/pod-security-admission/ for usage guidelines. (#106089, @liggitt)
ServiceAccountIssuerDiscovery feature gate is removed. It reached GA in Kubernetes 1.21. (#103685, @mengjiao-liu)
constants/variables from k8s.io for STABLE metrics is now supported. (#103654, @coffeepac)
kubectl describe namespace now shows Conditions (#106219, @dlipovetsky)
The etcd container image now supports Windows. (#92433, @claudiubelu)
The kube-apiserver’s Prometheus metrics have been extended with some that describe the costs of handling LIST requests. They are as follows.
- apiserver_cache_list_total: Counter of LIST requests served from watch cache, broken down by resource_prefix and index_name
- apiserver_cache_list_fetched_objects_total: Counter of objects read from watch cache in the course of serving a LIST request, broken down by resource_prefix and index_name
- apiserver_cache_list_evaluated_objects_total: Counter of objects tested in the course of serving a LIST request from watch cache, broken down by resource_prefix
- apiserver_cache_list_returned_objects_total: Counter of objects returned for a LIST request from watch cache, broken down by resource_prefix
- apiserver_storage_list_total: Counter of LIST requests served from etcd, broken down by resource
- apiserver_storage_list_fetched_objects_total: Counter of objects read from etcd in the course of serving a LIST request, broken down by resource
- apiserver_storage_list_evaluated_objects_total: Counter of objects tested in the course of serving a LIST request from etcd, broken down by resource
- apiserver_storage_list_returned_objects_total: Counter of objects returned for a LIST request from etcd, broken down by resource (#104983, @MikeSpreitzer)
The pause image list now contains Windows Server 2022. (#104438, @nick5616)
csi-proxy v1.0.1-gke.0. (#104426, @mauriciopoppe)
This PR adds the following metrics for API Priority and Fairness.
- apiserver_flowcontrol_priority_level_seat_count_samples: histograms of seats occupied by executing requests (both regular and final-delay phases included), broken down by priority_level; the observations are taken once per millisecond.
- apiserver_flowcontrol_priority_level_seat_count_watermarks: histograms of high and low watermarks of number of seats occupied by executing requests (both regular and final-delay phases included), broken down by priority_level.
- apiserver_flowcontrol_watch_count_samples: histograms of number of watches relevant to a given mutating request, broken down by that request’s priority_level and flow_schema. (#105873, @MikeSpreitzer) [SIG API Machinery, Instrumentation and Testing]
Topology Aware Hints have graduated to beta. (#106433, @robscott) [SIG Network]
Turn on CSIMigrationAzureDisk by default on 1.23 (#104670, @andyzhangx)
Update the system-validators library to v1.6.0 (#106323, @neolit123) [SIG Cluster Lifecycle and Node]
Updated Cluster Autosaler to version
1.22.0. Release notes: https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.22.0. (#104293, @x13n)
debian-iptables to v1.6.7 to pick up CVE fixes. (#104970, @PushkarJ)
Updates the following images to pick up CVE fixes:
Upgrade etcd to 3.5.1 (#105706, @uthark) [SIG Cloud Provider, Cluster Lifecycle and Testing]
When feature gate
JobTrackingWithFinalizers is enabled:
- Limit the number of Pods tracked in a single Job sync to avoid starvation of small Jobs.
- The metric
job_pod_finished_total counts the number of finished Pods tracked by the Job controller. (#105197, @alculquicondor)
RequestedToCapacityRatio ScoringStrategy, empty shape will cause error. (#106169, @kerthcet) [SIG Scheduling]
client-go event library allows customizing spam filtering function.
It is now possible to override
SpamKeyFunc, which is used by event filtering to detect spam in the events. (#103918, @olagacek)
client-go, using log level 9, traces the following events of a HTTP request:
- DNS lookup
- TCP dialing
- TLS handshake
- Time to get a connection from the pool
- Time to process a request (#105156, @aojea)
Allow KUBE_TEST_REPO_LIST to be a remote url (#109512, @eddiezane) [SIG Cloud Provider and Testing]
Kubernetes is now built with Golang 1.17.11 (#110423, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing]
Kube-apiserver: when merging lists, Server Side Apply now prefers the order of the submitted request instead of the existing persisted object (#107567, @jiahuif) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Storage and Testing]