This is the first Azure release featuring Kubernetes 1.23.
Furthermore, this release is the first to use out-of-tree controller manager and CSI providers.
Upgraded from version 2.24.1.
FlexVolume is deprecated. Out-of-tree CSI driver is the recommended way to write volume drivers in Kubernetes.
See this doc for more information.
Maintainers of FlexVolume drivers should implement a CSI driver and move users of FlexVolume to CSI.
Users of FlexVolume should move their workloads to CSI driver.
Kubernetes releases are now generating provenance attestation files describing the staging and release phases of the release process and artifacts are verified as they are handed over from one phase to the next.
This final piece completes the work needed to comply with Level 1 of the SLSA security framework (Supply-chain Levels for Software Artifacts).
Version 2 of the HorizontalPodAutoscaler API graduates to stable in the 1.23 release. The HorizontalPodAutoscaler autoscaling/v2beta2 API is deprecated in favor of the new autoscaling/v2 API, which the Kubernetes project recommends for all use cases.
The generic ephemeral volume feature moved to GA in 1.23.
This feature allows any existing storage driver that supports dynamic provisioning to be used as an ephemeral volume with the volume’s lifecycle bound to the Pod.
All StorageClass parameters for volume provisioning and all features supported with PersistentVolumeClaims are supported.
The feature to configure volume permission and ownership change policy for Pods moved to GA in 1.23.
This allows users to skip recursive permission changes on mount and speeds up the pod start up time.
The feature to allow CSI Drivers to declare support for fsGroup based permissions graduates to GA in 1.23.
Structured logging reached its Beta milestone. Most log messages from kubelet and kube-scheduler have been converted. Users are encouraged to try out JSON output or parsing of the structured text format and provide feedback on possible solutions for the open issues, such as handling of multi-line strings in log values.
The kube-scheduler is adding a new, simplified config field for Plugins to allow multiple extension points to be enabled in one spot.
The new multiPoint plugin field is intended to simplify most scheduler setups for administrators.
Plugins that are enabled via multiPoint will automatically be registered for each individual extension point that they implement.
For example, a plugin that implements Score and Filter extensions can be simultaneously enabled for both.
This means entire plugins can be enabled and disabled without having to manually edit individual extension point settings.
These extension points can now be abstracted away due to their irrelevance for most users.
CSI Migration enables the replacement of existing in-tree storage plugins such as kubernetes.io/gce-pd or kubernetes.io/aws-ebs with a corresponding CSI driver.
If CSI Migration is working properly, Kubernetes end users shouldn’t notice a difference.
After migration, Kubernetes users may continue to rely on all the functionality of in-tree storage plugins using the existing interface.
Data corruption issue was found in etcd v3.5.0 release that was shipped with 1.22 Kubernetes release. Please read up-to-date production recommendations for etcd.
(beta feature) If the CSI driver supports the NodeServiceCapability VOLUME_MOUNT_GROUP and the DelegateFSGroupToCSIDriver feature gate is enabled, kubelet will delegate applying FSGroup to the driver by passing it to NodeStageVolume and NodePublishVolume, regardless of what other FSGroup policies are set. (#106330, @verult) [SIG Storage]
Add a new distribute-cpus-across-numa option to the static CPUManager policy. When enabled, this will trigger the CPUManager to evenly distribute CPUs across NUMA nodes in cases where more than one NUMA node is required to satisfy the allocation. (#105631, @klueska)
Add fish shell completion to kubectl. (#92989, @WLun001)
Add mechanism to load simple sniffer class into fluentd-elasticsearch image (#92853, @cosmo0920)
Add support for Portworx plugin to csi-translation-lib. Alpha release
Portworx CSI driver is required to enable migration.
This PR adds support of the CSIMigrationPortworx feature gate, which can be enabled by:
- Adding the feature flag to the kube-controller-manager
--feature-gates=CSIMigrationPortworx=true - Adding the feature flag to the kubelet config:
featureGates:
CSIMigrationPortworx: true (#103447, @trierra) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scalability, Scheduling, Storage, Testing and Windows]
Add support to generate client-side binaries for windows/arm64 platform (#104894, @pacoxu)
Added PowerShell completion generation by running kubectl completion powershell. (#103758, @zikhan)
Added a Processing condition for the workqueue API.
Changed Shutdown for the workqueue API to wait until the work queue finishes processing all in-flight items. (#101928, @alexanderConstantinescu)
Added a new feature gate CustomResourceValidationExpressions to enable expression validation for Custom Resource. (#105107, @cici37)
Added a new flag --append-server-path to kubectl proxy that will automatically append the kube context server path to each request. (#97350, @FabianKramm)
Added ability for kubectl wait to wait on arbitary JSON path (#105776, @lauchokyip)
Added support for PodAndContainerStatsFromCRI feature gate, which allows a user to specify their pod stats must also come from the CRI, not cAdvisor. (#103095, @haircommander)
Added support for setting controller-manager log level online. (#104571, @h4ghhh)
Added the ability to specify whether to use an RFC7396 JSON Merge Patch, an RFC6902 JSON Patch, or a Strategic Merge Patch to perform an override of the resources created by kubectl run and kubectl expose. (#105140, @brianpursley)
Adding option for kubectl cp to resume on network errors until completion, requires tar in addition to tail inside the container image (#104792, @matthyx)
Adding support for multiple --from-env-file flags. (#104232, @lauchokyip)
Adding support for multiple --from-env-file flags. (#101646, @lauchokyip)
Adds --as-uid flag to kubectl to allow uid impersonation in the same way as user and group impersonation. (#105794, @margocrawf)
Adds new [alpha] command ‘kubectl events’ (#99557, @bboreham)
Allow node expansion of local volumes. (#102886, @gnufied)
Allow to build kubernetes with a custom kube-cross image. (#104185, @dims)
Allows users to prevent garbage collection on pinned images (#103299, @wgahnagl) [SIG Node]
CRI v1 is now the project default. If a container runtime does not support the v1 API, Kubernetes will fall back to the v1alpha2 implementation. (#106501, @ehashman)
Changed feature CSIMigrationAWS to on by default. This feature requires the AWS EBS CSI driver to be installed. (#106098, @wongma7)
Client-go: pass DeleteOptions down to the fake client Reactor (#102945, @chenchun)
Cloud providers can set service account names for cloud controllers. (#103178, @nckturner)
Display Labels when kubectl describe ingress. (#103894, @kabab)
Enhance scheduler VolumeBinding plugin to handle Lost PVC as UnschedulableAndUnresolvable (#105245, @yibozhuang)
Ensures that volume is deleted from the storage backend when the user tries to delete the PV object manually and the PV ReclaimPolicy is set to Delete. (#105773, @deepakkinni)
Expose a NewUnstructuredExtractor from apply configurations meta/v1 package that enables extracting objects into unstructured apply configurations. (#103564, @kevindelgado)
Feature gate StorageObjectInUseProtection has been deprecated and cannot be disabled. It will be completely removed in 1.25 (#105495, @ikeeip)
Graduating controller_admission_duration_seconds, step_admission_duration_seconds, webhook_admission_duration_seconds, apiserver_current_inflight_requests and apiserver_response_sizes metrics to stable. (#106122, @rezakrimi) [SIG API Machinery, Instrumentation and Testing]
Graduating pending_pods, preemption_attempts_total, preemption_victims and schedule_attempts_total metrics to stable. Also e2e_scheduling_duration_seconds is renamed to scheduling_attempt_duration_seconds and the latter is graduated to stable. (#105941, @rezakrimi) [SIG Instrumentation, Scheduling and Testing]
Health check of kube-controller-manager now includes each controller. (#104667, @jiahuif)
Integration testing now takes periodic Prometheus scrapes from the etcd server.
There is a new script ,hack/run-prometheus-on-etcd-scrapes.sh, that runs a containerized Prometheus server against an archive of such scrapes. (#106190, @MikeSpreitzer) [SIG API Machinery and Testing]
Introduce a feature gate DisableKubeletCloudCredentialProviders which allows disabling the in-tree kubelet credential providers.
The feature gate DisableKubeletCloudCredentialProviders is currently in Alpha, which means is currently disabled by default. Once this feature gate moves to beta, in-tree credential providers will be disabled by default, and users will need to migrate to use external credential providers. (#102507, @ostrain)
Introduces a new metric: admission_webhook_request_total with the following labels: name (string) - the webhook name, type (string) - the admission type, operation (string) - the requested verb, code (int) - the HTTP status code, rejected (bool) - whether the request was rejected, namespace (string) - the namespace of the requested resource. (#103162, @rmoriar1)
Kubeadm: add support for dry running kubeadm join. The new flag kubeadm join --dry-run is similar to the existing flag for kubeadm init/upgrade and allows you to see what changes would be applied. (#103027, @Haleygo)
Kubeadm: do not check if the /etc/kubernetes/manifests folder is empty on joining worker nodes during preflight (#104942, @SataQiu)
Kubectl will now provide shell completion choices for the --output/-o flag (#105851, @marckhouzam)
Kubelet should reconcile kubernetes.io/os and kubernetes.io/arch labels on the node object. The side-effect of this is kubelet would deny admission to pod which has nodeSelector with label kubernetes.io/os or kubernetes.io/arch which doesn’t match the underlying OS or arch on the host OS.
- The label reconciliation happens as part of periodic status update which can be configured via flag
--node-status-update-frequency (#104613, @ravisantoshgudimetla) [SIG Node, Testing and Windows]
Kubernetes is now built with Golang 1.16.7. (#104199, @cpanato)
Kubernetes is now built with Golang 1.17.1. (#104904, @cpanato)
Kubernetes is now built with Golang 1.17.2 (#105563, @mengjiao-liu)
Kubernetes is now built with Golang 1.17.3 (#106209, @cpanato) [SIG API Machinery, Cloud Provider, Instrumentation, Release and Testing]
Move ConfigurableFSGroupPolicy to GA and rename metric volume_fsgroup_recursive_apply to volume_apply_access_control (#105885, @gnufied)
Move the GetAllocatableResources Endpoint in PodResource API to the beta that will make it enabled by default. (#105003, @swatisehgal)
Moving WindowsHostProcessContainers feature to beta (#106058, @marosset)
Node affinity, Node selectors, and tolerations are now mutable for Jobs that are suspended and have never been started (#105479, @ahg-g)
Pod template annotations and labels are now mutable for Jobs that are suspended and have never been started (#105980, @ahg-g)
PodSecurity: in 1.23+ restricted policy levels, Pods and containers which set runAsUser=0 are forbidden at admission-time; previously, they would be rejected at runtime (#105857, @liggitt)
Shell completion now knows to continue suggesting resource names when the command supports it. For example kubectl get pod pod1 <TAB> will suggest more Pod names. (#105711, @marckhouzam)
Support to enable Hyper-V in GCE Windows Nodes created with kube-up (#105999, @mauriciopoppe)
The CPUManager policy options are now enabled, and we introduce a graduation path for the new CPU Manager policy options. (#105012, @fromanirh)
The Pods and Pod controllers that are exempted from the PodSecurity admission process are now marked with the pod-security.kubernetes.io/exempt: user/namespace/runtimeClass annotation, based on what caused the exemption.
The enforcement level that allowed or denied a Pod during PodSecurity admission is now marked by the pod-security.kubernetes.io/enforce-policy annotation.
The annotation that informs about audit policy violations changed from pod-security.kubernetes.io/audit to pod-security.kubernetes.io/audit-violation. (#105908, @stlaz)
The /openapi/v3 endpoint will be populated with OpenAPI v3 if the feature flag is enabled (#105945, @Jefftree)
The CSIMigrationGCE feature flag is turned ON by default (#104722, @leiyiz)
The DownwardAPIHugePages feature is now enabled by default. (#106271, @mysunshine92)
The PodSecurity admission plugin has graduated to beta and is enabled by default. The admission configuration version has been promoted to pod-security.admission.config.k8s.io/v1beta1. See https://kubernetes.io/docs/concepts/security/pod-security-admission/ for usage guidelines. (#106089, @liggitt)
The ServiceAccountIssuerDiscovery feature gate is removed. It reached GA in Kubernetes 1.21. (#103685, @mengjiao-liu)
The constants/variables from k8s.io for STABLE metrics is now supported. (#103654, @coffeepac)
The kubectl describe namespace now shows Conditions (#106219, @dlipovetsky)
The etcd container image now supports Windows. (#92433, @claudiubelu)
The kube-apiserver’s Prometheus metrics have been extended with some that describe the costs of handling LIST requests. They are as follows.
- apiserver_cache_list_total: Counter of LIST requests served from watch cache, broken down by resource_prefix and index_name
- apiserver_cache_list_fetched_objects_total: Counter of objects read from watch cache in the course of serving a LIST request, broken down by resource_prefix and index_name
- apiserver_cache_list_evaluated_objects_total: Counter of objects tested in the course of serving a LIST request from watch cache, broken down by resource_prefix
- apiserver_cache_list_returned_objects_total: Counter of objects returned for a LIST request from watch cache, broken down by resource_prefix
- apiserver_storage_list_total: Counter of LIST requests served from etcd, broken down by resource
- apiserver_storage_list_fetched_objects_total: Counter of objects read from etcd in the course of serving a LIST request, broken down by resource
- apiserver_storage_list_evaluated_objects_total: Counter of objects tested in the course of serving a LIST request from etcd, broken down by resource
- apiserver_storage_list_returned_objects_total: Counter of objects returned for a LIST request from etcd, broken down by resource (#104983, @MikeSpreitzer)
The pause image list now contains Windows Server 2022. (#104438, @nick5616)
The script kube-up.sh installs csi-proxy v1.0.1-gke.0. (#104426, @mauriciopoppe)
This PR adds the following metrics for API Priority and Fairness.
- apiserver_flowcontrol_priority_level_seat_count_samples: histograms of seats occupied by executing requests (both regular and final-delay phases included), broken down by priority_level; the observations are taken once per millisecond.
- apiserver_flowcontrol_priority_level_seat_count_watermarks: histograms of high and low watermarks of number of seats occupied by executing requests (both regular and final-delay phases included), broken down by priority_level.
- apiserver_flowcontrol_watch_count_samples: histograms of number of watches relevant to a given mutating request, broken down by that request’s priority_level and flow_schema. (#105873, @MikeSpreitzer) [SIG API Machinery, Instrumentation and Testing]
Topology Aware Hints have graduated to beta. (#106433, @robscott) [SIG Network]
Turn on CSIMigrationAzureDisk by default on 1.23 (#104670, @andyzhangx)
Update the system-validators library to v1.6.0 (#106323, @neolit123) [SIG Cluster Lifecycle and Node]
Updated Cluster Autosaler to version 1.22.0. Release notes: https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.22.0. (#104293, @x13n)
Updates debian-iptables to v1.6.7 to pick up CVE fixes. (#104970, @PushkarJ)
Updates the following images to pick up CVE fixes:
Upgrade etcd to 3.5.1 (#105706, @uthark) [SIG Cloud Provider, Cluster Lifecycle and Testing]
When feature gate JobTrackingWithFinalizers is enabled:
- Limit the number of Pods tracked in a single Job sync to avoid starvation of small Jobs.
- The metric
job_pod_finished_total counts the number of finished Pods tracked by the Job controller. (#105197, @alculquicondor)
When using RequestedToCapacityRatio ScoringStrategy, empty shape will cause error. (#106169, @kerthcet) [SIG Scheduling]
client-go event library allows customizing spam filtering function.
It is now possible to override SpamKeyFunc, which is used by event filtering to detect spam in the events. (#103918, @olagacek)
client-go, using log level 9, traces the following events of a HTTP request:
- DNS lookup
- TCP dialing
- TLS handshake
- Time to get a connection from the pool
- Time to process a request (#105156, @aojea)
Allow KUBE_TEST_REPO_LIST to be a remote url (#109512, @eddiezane) [SIG Cloud Provider and Testing]
Kubernetes is now built with Golang 1.17.11 (#110423, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing]
Kube-apiserver: when merging lists, Server Side Apply now prefers the order of the submitted request instead of the existing persisted object (#107567, @jiahuif) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Storage and Testing]
