Workload cluster releases for AWS

  • This release provides a fix for cert-operator to ensure certConfig is in the same org namespace as the Cluster resource.

    Change details

    cert-operator 1.2.0

    Changed

    • Introducing v1alpha3 CR’s.

    Added

    • Add check to ensure that the Cluster resource is in the same namespace as the certConfig before creating the secret there.
  • This release provides support for Kubernetes 1.21.

    Highlights

    • Kubernetes 1.21 support;
    • New clusters must be created in the organization’s namespaces. Clusters which don’t respect this rule will be rejected;
    • Tags on custom resources are propagated to S3 buckets and AWS CNI ENIs;
    • aws-operator and cluster-operator have support for Cluster API v1alpha3 resources;
    • Security fixes:
      • 7 Linux CVEs;
      • 2 Systemd CVEs;
      • 2 openssl CVEs;
      • 1 Kubernetes CVE;
      • 1 Go CVE.

    Warning: Starting with this release new clusters have to be created in the organization’s namespace. Any new cluster defined in a different namespace is going to be rejected. Automation that creates Cluster CRs directly needs to be updated to define these clusters in the organization’s namespace. Existing clusters are not impacted by this change and will be migrated later.

    Change details

    aws-operator 10.9.1

    Added

    • Add cloud tags propagation to S3 buckets.
    • Add provider tags to the AWS CNI ENIs.
    • Add configuration for systemd-networkd to ignore network interfaces used for AWS CNI.
    • Add changes to run properly on Flatcar 2905 and newer.

    Changed

    • Update aws-attach-etcd-dep image version to 0.2.0 to include bugfixes.
    • Upgrade k8scloudconfig which is required for k8s 1.21.
    • Introducing v1alpha3 CR’s.
    • Update Flatcar AMI’s to the latest stable releases.

    containerlinux 2905.2.3

    Security fixes

    Bug Fixes

    Changes

    Updates

    kubernetes 1.21.5

    What’s New (Major Themes)

    Deprecation of PodSecurityPolicy

    PSP as an admission controller resource is being deprecated. Deployed PodSecurityPolicy’s will keep working until version 1.25, their target removal from the codebase. A new feature, with a working title of “PSP replacement policy”, is being developed in KEP-2579. To learn more, read PodSecurityPolicy Deprecation: Past, Present, and Future.

    Kubernetes API Reference Documentation

    The API reference is now generated with gen-resourcesdocs and it is moving to Kubernetes API

    Kustomize Updates in Kubectl

    Kustomize version in kubectl had a jump from v2.0.3 to v4.0.5. Kustomize is now treated as a library and future updates will be less sporadic.

    Default Container Annotation

    Pod with multiple containers can use kubectl.kubernetes.io/default-container annotation to have a container preselected for kubectl commands. More can be read in KEP-2227.

    Immutable Secrets and ConfigMaps

    Immutable Secrets and ConfigMaps graduates to GA. This feature allows users to specify that the contents of a particular Secret or ConfigMap is immutable for its object lifetime. For such instances, Kubelet will not watch/poll for changes and therefore reducing apiserver load.

    Structured Logging in Kubelet

    Kubelet has adopted structured logging, thanks to community effort in accomplishing this within the release timeline. Structured logging in the project remains an ongoing effort – for folks interested in participating, keep an eye / chime in to the mailing list discussion.

    Storage Capacity Tracking

    Traditionally, the Kubernetes scheduler was based on the assumptions that additional persistent storage is available everywhere in the cluster and has infinite capacity. Topology constraints addressed the first point, but up to now pod scheduling was still done without considering that the remaining storage capacity may not be enough to start a new pod. Storage capacity tracking addresses that by adding an API for a CSI driver to report storage capacity and uses that information in the Kubernetes scheduler when choosing a node for a pod. This feature serves as a stepping stone for supporting dynamic provisioning for local volumes and other volume types that are more capacity constrained.

    Generic Ephemeral Volumes

    Generic ephermeral volumes feature allows any existing storage driver that supports dynamic provisioning to be used as an ephemeral volume with the volume’s lifecycle bound to the Pod. It can be used to provide scratch storage that is different from the root disk, for example persistent memory, or a separate local disk on that node. All StorageClass parameters for volume provisioning are supported. All features supported with PersistentVolumeClaims are supported, such as storage capacity tracking, snapshots and restore, and volume resizing.

    CSI Service Account Token

    CSI Service Account Token feature moves to Beta in 1.21. This feature improves the security posture and allows CSI drivers to receive pods' bound service account tokens. This feature also provides a knob to re-publish volumes so that short-lived volumes can be refreshed.

    CSI Health Monitoring

    The CSI health monitoring feature is being released as a second Alpha in Kubernetes 1.21. This feature enables CSI Drivers to share abnormal volume conditions from the underlying storage systems with Kubernetes so that they can be reported as events on PVCs or Pods. This feature serves as a stepping stone towards programmatic detection and resolution of individual volume health issues by Kubernetes.

    Important Security Information

    This release contains changes that address the following vulnerabilities:

    A security issue was discovered in Kubernetes where a user may be able to create a container with subpath volume mounts to access files & directories outside of the volume, including on the host filesystem.

    API Change

    • “Auto” is now a valid value for the service.kubernetes.io/topology-aware-hints annotation. (#100728, @robscott) [SIG Apps, Instrumentation and Network]
    • We have added a new Priority & Fairness rule that exempts all probes (/readyz, /healthz, /livez) to prevent restarting of “healthy” kube-apiserver instance(s) by kubelet. (#101111, @tkashem) [SIG API Machinery]

    Known Issues

    TopologyAwareHints feature falls back to default behavior

    The feature gate currently falls back to the default behavior in most cases. Enabling the feature gate will add hints to EndpointSlices, but functional differences are only observed in non-dual stack kube-proxy implementation. This is fixed by #101054.

    Urgent Upgrade Notes

    (No, really, you MUST read this before you upgrade)
    • Kube-proxy’s IPVS proxy mode no longer sets the net.ipv4.conf.all.route_localnet sysctl parameter. Nodes upgrading will have net.ipv4.conf.all.route_localnet set to 1 but new nodes will inherit the system default (usually 0). If you relied on any behavior requiring net.ipv4.conf.all.route_localnet, you must set ensure it is enabled as kube-proxy will no longer set it automatically. This change helps to further mitigate CVE-2020-8558. (#92938, @lbernail) [SIG Network and Release]
    • Kubeadm: during “init” an empty cgroupDriver value in the KubeletConfiguration is now always set to “systemd” unless the user is explicit about it. This requires existing machine setups to configure the container runtime to use the “systemd” driver. Documentation on this topic can be found here: https://kubernetes.io/docs/setup/production-environment/container-runtimes/. When upgrading existing clusters / nodes using “kubeadm upgrade” the old cgroupDriver value is preserved, but in 1.22 this change will also apply to “upgrade”. For more information on migrating to the “systemd” driver or remaining on the “cgroupfs” driver see: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/. (#99471, @neolit123) [SIG Cluster Lifecycle]
    • Newly provisioned PVs by EBS plugin will no longer use the deprecated “failure-domain.beta.kubernetes.io/zone” and “failure-domain.beta.kubernetes.io/region” labels. It will use “topology.kubernetes.io/zone” and “topology.kubernetes.io/region” labels instead. (#99130, @ayberk) [SIG Cloud Provider, Storage and Testing]
    • Newly provisioned PVs by OpenStack Cinder plugin will no longer use the deprecated “failure-domain.beta.kubernetes.io/zone” and “failure-domain.beta.kubernetes.io/region” labels. It will use “topology.kubernetes.io/zone” and “topology.kubernetes.io/region” labels instead. (#99719, @jsafrane) [SIG Cloud Provider and Storage]
    • Newly provisioned PVs by gce-pd will no longer have the beta FailureDomain label. gce-pd volume plugin will start to have GA topology label instead. (#98700, @Jiawei0227) [SIG Cloud Provider, Storage and Testing]
    • OpenStack Cinder CSI migration is on by default, Clinder CSI driver must be installed on clusters on OpenStack for Cinder volumes to work. (#98538, @dims) [SIG Storage]
    • Remove alpha CSIMigrationXXComplete flag and add alpha InTreePluginXXUnregister flag. Deprecate CSIMigrationvSphereComplete flag and it will be removed in v1.22. (#98243, @Jiawei0227)
    • Remove storage metrics storage_operation_errors_total, since we already have storage_operation_status_count.And add new field status for storage_operation_duration_seconds, so that we can know about all status storage operation latency. (#98332, @JornShen) [SIG Instrumentation and Storage]
    • The metric storage_operation_errors_total is not removed, but is marked deprecated, and the metric storage_operation_status_count is marked deprecated. In both cases the storage_operation_duration_seconds metric can be used to recover equivalent counts (using status=fail-unknown in the case of storage_operations_errors_total). (#99045, @mattcary)
    • ServiceNodeExclusion, NodeDisruptionExclusion and LegacyNodeRoleBehavior features have been promoted to GA. ServiceNodeExclusion and NodeDisruptionExclusion are now unconditionally enabled, while LegacyNodeRoleBehavior is unconditionally disabled. To prevent control plane nodes from being added to load balancers automatically, upgrade users need to add “node.kubernetes.io/exclude-from-external-load-balancers” label to control plane nodes. (#97543, @pacoxu)

    Deprecation

    • Aborting the drain command in a list of nodes will be deprecated. The new behavior will make the drain command go through all nodes even if one or more nodes failed during the drain. For now, users can try such experience by enabling –ignore-errors flag. (#98203, @yuzhiquan)

    • Delete deprecated service.beta.kubernetes.io/azure-load-balancer-mixed-protocols mixed procotol annotation in favor of the MixedProtocolLBService feature (#97096, @nilo19) [SIG Cloud Provider]

    • Deprecate the topologyKeys field in Service. This capability will be replaced with upcoming work around Topology Aware Subsetting and Service Internal Traffic Policy. (#96736, @andrewsykim) [SIG Apps]

    • Kube-proxy: remove deprecated –cleanup-ipvs flag of kube-proxy, and make –cleanup flag always to flush IPVS (#97336, @maaoBit) [SIG Network]

    • Kubeadm: deprecated command “alpha selfhosting pivot” is now removed. (#97627, @knight42)

    • Kubeadm: graduate the command kubeadm alpha kubeconfig user to kubeadm kubeconfig user. The kubeadm alpha kubeconfig user command is deprecated now. (#97583, @knight42) [SIG Cluster Lifecycle]

    • Kubeadm: the “kubeadm alpha certs” command is removed now, please use “kubeadm certs” instead. (#97706, @knight42) [SIG Cluster Lifecycle]

    • Kubeadm: the deprecated kube-dns is no longer supported as an option. If “ClusterConfiguration.dns.type” is set to “kube-dns” kubeadm will now throw an error. (#99646, @rajansandeep) [SIG Cluster Lifecycle]

    • Kubectl: The deprecated kubectl alpha debug command is removed. Use kubectl debug instead. (#98111, @pandaamanda) [SIG CLI]

    • Official support to build kubernetes with docker-machine / remote docker is removed. This change does not affect building kubernetes with docker locally. (#97935, @adeniyistephen) [SIG Release and Testing]

    • Remove deprecated --generator, --replicas, --service-generator, --service-overrides, --schedule from kubectl run Deprecate --serviceaccount, --hostport, --requests, --limits in kubectl run (#99732, @soltysh)

    • Remove the deprecated metrics “scheduling_algorithm_preemption_evaluation_seconds” and “binding_duration_seconds”, suggest to use “scheduler_framework_extension_point_duration_seconds” instead. (#96447, @chendave) [SIG Cluster Lifecycle, Instrumentation, Scheduling and Testing]

    • Removing experimental windows container hyper-v support with Docker (#97141, @wawa0210) [SIG Node and Windows]

    • Rename metrics etcd_object_counts to apiserver_storage_object_counts and mark it as stable. The original etcd_object_counts metrics name is marked as “Deprecated” and will be removed in the future. (#99785, @erain) [SIG API Machinery, Instrumentation and Testing]

    • The GA TokenRequest and TokenRequestProjection feature gates have been removed and are unconditionally enabled. Remove explicit use of those feature gates in CLI invocations. (#97148, @wawa0210) [SIG Node]

    • The PodSecurityPolicy API is deprecated in 1.21, and will no longer be served starting in 1.25. (#97171, @deads2k) [SIG Auth and CLI]

    • The batch/v2alpha1 CronJob type definitions and clients are deprecated and removed. (#96987, @soltysh) [SIG API Machinery, Apps, CLI and Testing]

    • The export query parameter (inconsistently supported by API resources and deprecated in v1.14) is fully removed. Requests setting this query parameter will now receive a 400 status response. (#98312, @deads2k) [SIG API Machinery, Auth and Testing]

    • audit.k8s.io/v1beta1 and audit.k8s.io/v1alpha1 audit policy configuration and audit events are deprecated in favor of audit.k8s.io/v1, available since v1.13. kube-apiserver invocations that specify alpha or beta policy configurations with --audit-policy-file, or explicitly request alpha or beta audit events with --audit-log-version / --audit-webhook-version must update to use audit.k8s.io/v1 and accept audit.k8s.io/v1 events prior to v1.24. (#98858, @carlory) [SIG Auth]

    • discovery.k8s.io/v1beta1 EndpointSlices are deprecated in favor of discovery.k8s.io/v1, and will no longer be served in Kubernetes v1.25. (#100472, @liggitt)

    • diskformat storage class parameter for in-tree vSphere volume plugin is deprecated as of v1.21 release. Please consider updating storageclass and remove diskformat parameter. vSphere CSI Driver does not support diskformat storageclass parameter.

      vSphere releases less than 67u3 are deprecated as of v1.21. Please consider upgrading vSphere to 67u3 or above. vSphere CSI Driver requires minimum vSphere 67u3.

      VM Hardware version less than 15 is deprecated as of v1.21. Please consider upgrading the Node VM Hardware version to 15 or above. vSphere CSI Driver recommends Node VM’s Hardware version set to at least vmx-15.

      Multi vCenter support is deprecated as of v1.21. If you have a Kubernetes cluster spanning across multiple vCenter servers, please consider moving all k8s nodes to a single vCenter Server. vSphere CSI Driver does not support Kubernetes deployment spanning across multiple vCenter servers.

      Support for these deprecations will be available till Kubernetes v1.24. (#98546, @divyenpatel)

    API Change

      1. PodAffinityTerm includes a namespaceSelector field to allow selecting eligible namespaces based on their labels.
      2. A new CrossNamespacePodAffinity quota scope API that allows restricting which namespaces allowed to use PodAffinityTerm with corss-namespace reference via namespaceSelector or namespaces fields. (#98582, @ahg-g) [SIG API Machinery, Apps, Auth and Testing]
    • Add Probe-level terminationGracePeriodSeconds field (#99375, @ehashman) [SIG API Machinery, Apps, Node and Testing]
    • Added .spec.completionMode field to Job, with accepted values NonIndexed (default) and Indexed. This is an alpha field and is only honored by servers with the IndexedJob feature gate enabled. (#98441, @alculquicondor) [SIG Apps and CLI]
    • Adds support for endPort field in NetworkPolicy (#97058, @rikatz) [SIG Apps and Network]
    • CSIServiceAccountToken graduates to Beta and enabled by default. (#99298, @zshihang)
    • Cluster admins can now turn off /debug/pprof and /debug/flags/v endpoint in kubelet by setting enableProfilingHandler and enableDebugFlagsHandler to false in the Kubelet configuration file. Options enableProfilingHandler and enableDebugFlagsHandler can be set to true only when enableDebuggingHandlers is also set to true. (#98458, @SaranBalaji90)
    • DaemonSets accept a MaxSurge integer or percent on their rolling update strategy that will launch the updated pod on nodes and wait for those pods to go ready before marking the old out-of-date pods as deleted. This allows workloads to avoid downtime during upgrades when deployed using DaemonSets. This feature is alpha and is behind the DaemonSetUpdateSurge feature gate. (#96441, @smarterclayton) [SIG Apps and Testing]
    • Enable SPDY pings to keep connections alive, so that kubectl exec and kubectl portforward won’t be interrupted. (#97083, @knight42) [SIG API Machinery and CLI]
    • FieldManager no longer owns fields that get reset before the object is persisted (e.g. “status wiping”). (#99661, @kevindelgado) [SIG API Machinery, Auth and Testing]
    • Fixes server-side apply for APIService resources. (#98576, @kevindelgado)
    • Generic ephemeral volumes are beta. (#99643, @pohly) [SIG API Machinery, Apps, Auth, CLI, Node, Storage and Testing]
    • Hugepages request values are limited to integer multiples of the page size. (#98515, @lala123912) [SIG Apps]
    • Implement the GetAvailableResources in the podresources API. (#95734, @fromanirh) [SIG Instrumentation, Node and Testing]
    • IngressClass resource can now reference a resource in a specific namespace for implementation-specific configuration (previously only Cluster-level resources were allowed). This feature can be enabled using the IngressClassNamespacedParams feature gate. (#99275, @hbagdi)
    • Jobs API has a new .spec.suspend field that can be used to suspend and resume Jobs. This is an alpha field which is only honored by servers with the SuspendJob feature gate enabled. (#98727, @adtac)
    • Kubelet Graceful Node Shutdown feature graduates to Beta and enabled by default. (#99735, @bobbypage)
    • Kubernetes is now built using go1.15.7 (#98363, @cpanato) [SIG Cloud Provider, Instrumentation, Node, Release and Testing]
    • Namespace API objects now have a kubernetes.io/metadata.name label matching their metadata.name field to allow selecting any namespace by its name using a label selector. (#96968, @jayunit100) [SIG API Machinery, Apps, Cloud Provider, Storage and Testing]
    • One new field “InternalTrafficPolicy” in Service is added. It specifies if the cluster internal traffic should be routed to all endpoints or node-local endpoints only. “Cluster” routes internal traffic to a Service to all endpoints. “Local” routes traffic to node-local endpoints only, and traffic is dropped if no node-local endpoints are ready. The default value is “Cluster”. (#96600, @maplain) [SIG API Machinery, Apps and Network]
    • PodDisruptionBudget API objects can now contain conditions in status. (#98127, @mortent) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Cluster Lifecycle and Instrumentation]
    • PodSecurityPolicy only stores “generic” as allowed volume type if the GenericEphemeralVolume feature gate is enabled (#98918, @pohly) [SIG Auth and Security]
    • Promote CronJobs to batch/v1 (#99423, @soltysh) [SIG API Machinery, Apps, CLI and Testing]
    • Promote Immutable Secrets/ConfigMaps feature to Stable. This allows to set immutable field in Secret or ConfigMap object to mark their contents as immutable. (#97615, @wojtek-t) [SIG Apps, Architecture, Node and Testing]
    • Remove support for building Kubernetes with bazel. (#99561, @BenTheElder) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scalability, Scheduling, Storage, Testing and Windows]
    • Scheduler extender filter interface now can report unresolvable failed nodes in the new field FailedAndUnresolvableNodes of ExtenderFilterResult struct. Nodes in this map will be skipped in the preemption phase. (#92866, @cofyc) [SIG Scheduling]
    • Services can specify loadBalancerClass to use a custom load balancer (#98277, @XudongLiuHarold)
    • Storage capacity tracking (= the CSIStorageCapacity feature) graduates to Beta and enabled by default, storage.k8s.io/v1alpha1/VolumeAttachment and storage.k8s.io/v1alpha1/CSIStorageCapacity objects are deprecated (#99641, @pohly)
    • Support for Indexed Job: a Job that is considered completed when Pods associated to indexes from 0 to (.spec.completions-1) have succeeded. (#98812, @alculquicondor) [SIG Apps and CLI]
    • The BoundServiceAccountTokenVolume feature has been promoted to beta, and enabled by default.
      • This changes the tokens provided to containers at /var/run/secrets/kubernetes.io/serviceaccount/token to be time-limited, auto-refreshed, and invalidated when the containing pod is deleted.
      • Clients should reload the token from disk periodically (once per minute is recommended) to ensure they continue to use a valid token. k8s.io/client-go version v11.0.0+ and v0.15.0+ reload tokens automatically.
      • By default, injected tokens are given an extended lifetime so they remain valid even after a new refreshed token is provided. The metric serviceaccount_stale_tokens_total can be used to monitor for workloads that are depending on the extended lifetime and are continuing to use tokens even after a refreshed token is provided to the container. If that metric indicates no existing workloads are depending on extended lifetimes, injected token lifetime can be shortened to 1 hour by starting kube-apiserver with --service-account-extend-token-expiration=false. (#95667, @zshihang) [SIG API Machinery, Auth, Cluster Lifecycle and Testing]
    • The EndpointSlice Controllers are now GA. The EndpointSliceController will not populate the deprecatedTopology field and will only provide topology information through the zone and nodeName fields. (#99870, @swetharepakula)
    • The Endpoints controller will now set the endpoints.kubernetes.io/over-capacity annotation to “warning” when an Endpoints resource contains more than 1000 addresses. In a future release, the controller will truncate Endpoints that exceed this limit. The EndpointSlice API can be used to support significantly larger number of addresses. (#99975, @robscott) [SIG Apps and Network]
    • The PodDisruptionBudget API has been promoted to policy/v1 with no schema changes. The only functional change is that an empty selector ({}) written to a policy/v1 PodDisruptionBudget now selects all pods in the namespace. The behavior of the policy/v1beta1 API remains unchanged. The policy/v1beta1 PodDisruptionBudget API is deprecated and will no longer be served in 1.25+. (#99290, @mortent) [SIG API Machinery, Apps, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Scheduling and Testing]
    • The EndpointSlice API is now GA. The EndpointSlice topology field has been removed from the GA API and will be replaced by a new per Endpoint Zone field. If the topology field was previously used, it will be converted into an annotation in the v1 Resource. The discovery.k8s.io/v1alpha1 API is removed. (#99662, @swetharepakula)
    • The controller.kubernetes.io/pod-deletion-cost annotation can be set to offer a hint on the cost of deleting a Pod compared to other pods belonging to the same ReplicaSet. Pods with lower deletion cost are deleted first. This is an alpha feature. (#99163, @ahg-g)
    • The kube-apiserver now resets managedFields that got corrupted by a mutating admission controller. (#98074, @kwiesmueller)
    • Topology Aware Hints are now available in alpha and can be enabled with the TopologyAwareHints feature gate. (#99522, @robscott) [SIG API Machinery, Apps, Auth, Instrumentation, Network and Testing]
    • Users might specify the kubectl.kubernetes.io/default-exec-container annotation in a Pod to preselect container for kubectl commands. (#97099, @pacoxu) [SIG CLI]

    Feature

    • Kubernetes is now built with Golang 1.16.8 (#104906, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing]
    • Update the setcap image to buster-v2.0.1 (#102377, @xmudrii) [SIG Release]
    • Add NeedResize function to kubernetes/mount-utils, user can call this function to determine if fs need to be resized (#101253, @AndyXiangLi) [SIG Storage]
    • Base image updates to mitigate kube-proxy and etcd container image CVEs
      • debian-base to buster-v1.6.0
      • debian-iptables to buster-v1.6.0 (#100976, @jindijamie) [SIG Release and Testing]
    • A client-go metric, rest_client_exec_plugin_call_total, has been added to track total calls to client-go credential plugins. (#98892, @ankeesler) [SIG API Machinery, Auth, Cluster Lifecycle and Instrumentation]
    • A new histogram metric to track the time it took to delete a job by the TTLAfterFinished controller (#98676, @ahg-g)
    • AWS cloud provider supports auto-discovering subnets without any kubernetes.io/cluster/<clusterName> tags. It also supports additional service annotation service.beta.kubernetes.io/aws-load-balancer-subnets to manually configure the subnets. (#97431, @kishorj)
    • Aborting the drain command in a list of nodes will be deprecated. The new behavior will make the drain command go through all nodes even if one or more nodes failed during the drain. For now, users can try such experience by enabling –ignore-errors flag. (#98203, @yuzhiquan)
    • Add –permit-address-sharing flag to kube-apiserver to listen with SO_REUSEADDR. While allowing to listen on wildcard IPs like 0.0.0.0 and specific IPs in parallel, it avoids waiting for the kernel to release socket in TIME_WAIT state, and hence, considerably reducing kube-apiserver restart times under certain conditions. (#93861, @sttts)
    • Add csi_operations_seconds metric on kubelet that exposes CSI operations duration and status for node CSI operations. (#98979, @Jiawei0227) [SIG Instrumentation and Storage]
    • Add migrated field into storage_operation_duration_seconds metric (#99050, @Jiawei0227) [SIG Apps, Instrumentation and Storage]
    • Add flag –lease-reuse-duration-seconds for kube-apiserver to config etcd lease reuse duration. (#97009, @lingsamuel) [SIG API Machinery and Scalability]
    • Add metric etcd_lease_object_counts for kube-apiserver to observe max objects attached to a single etcd lease. (#97480, @lingsamuel) [SIG API Machinery, Instrumentation and Scalability]
    • Add support to generate client-side binaries for new darwin/arm64 platform (#97743, @dims) [SIG Release and Testing]
    • Added ephemeral_volume_controller_create[_failures]_total counters to kube-controller-manager metrics (#99115, @pohly) [SIG API Machinery, Apps, Cluster Lifecycle, Instrumentation and Storage]
    • Added support for installing arm64 node artifacts. (#99242, @liu-cong)
    • Adds alpha feature VolumeCapacityPriority which makes the scheduler prioritize nodes based on the best matching size of statically provisioned PVs across multiple topologies. (#96347, @cofyc) [SIG Apps, Network, Scheduling, Storage and Testing]
    • Adds the ability to pass –strict-transport-security-directives to the kube-apiserver to set the HSTS header appropriately. Be sure you understand the consequences to browsers before setting this field. (#96502, @249043822) [SIG Auth]
    • Adds two new metrics to cronjobs, a histogram to track the time difference when a job is created and the expected time when it should be created, as well as a gauge for the missed schedules of a cronjob (#99341, @alaypatel07)
    • Alpha implementation of Kubectl Command Headers: SIG CLI KEP 859 enabled when KUBECTL_COMMAND_HEADERS environment variable set on the client command line. (#98952, @seans3)
    • Base-images: Update to debian-iptables:buster-v1.4.0
      • Uses iptables 1.8.5
      • base-images: Update to debian-base:buster-v1.3.0
      • cluster/images/etcd: Build etcd:3.4.13-2 image
    • CRIContainerLogRotation graduates to GA and unconditionally enabled. (#99651, @umohnani8)
    • Component owner can configure the allowlist of metric label with flag ‘–allow-metric-labels’. (#99385, @YoyinZyc) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Release]
    • Component owner can configure the allowlist of metric label with flag ‘–allow-metric-labels’. (#99738, @YoyinZyc) [SIG API Machinery, Cluster Lifecycle and Instrumentation]
    • EmptyDir memory backed volumes are sized as the the minimum of pod allocatable memory on a host and an optional explicit user provided value. (#100319, @derekwaynecarr) [SIG Node]
    • Enables Kubelet to check volume condition and log events to corresponding pods. (#99284, @fengzixu) [SIG Apps, Instrumentation, Node and Storage]
    • EndpointSliceNodeName graduates to GA and thus will be unconditionally enabled – NodeName will always be available in the v1beta1 API. (#99746, @swetharepakula)
    • Export NewDebuggingRoundTripper function and DebugLevel options in the k8s.io/client-go/transport package. (#98324, @atosatto)
    • Kube-proxy iptables: new metric sync_proxy_rules_iptables_total that exposes the number of rules programmed per table in each iteration (#99653, @aojea) [SIG Instrumentation and Network]
    • Kube-scheduler now logs plugin scoring summaries at –v=4 (#99411, @damemi) [SIG Scheduling]
    • Kubeadm now includes CoreDNS v1.8.0. (#96429, @rajansandeep) [SIG Cluster Lifecycle]
    • Kubeadm: IPv6DualStack feature gate graduates to Beta and enabled by default (#99294, @pacoxu)
    • Kubeadm: a warning to user as ipv6 site-local is deprecated (#99574, @pacoxu) [SIG Cluster Lifecycle and Network]
    • Kubeadm: add support for certificate chain validation. When using kubeadm in external CA mode, this allows an intermediate CA to be used to sign the certificates. The intermediate CA certificate must be appended to each signed certificate for this to work correctly. (#97266, @robbiemcmichael) [SIG Cluster Lifecycle]
    • Kubeadm: amend the node kernel validation to treat CGROUP_PIDS, FAIR_GROUP_SCHED as required and CFS_BANDWIDTH, CGROUP_HUGETLB as optional (#96378, @neolit123) [SIG Cluster Lifecycle and Node]
    • Kubeadm: apply the “node.kubernetes.io/exclude-from-external-load-balancers” label on control plane nodes during “init”, “join” and “upgrade” to preserve backwards compatibility with the lagacy LB mode where nodes labeled as “master” where excluded. To opt-out you can remove the label from a node. See #97543 and the linked KEP for more details. (#98269, @neolit123) [SIG Cluster Lifecycle]
    • Kubeadm: if the user has customized their image repository via the kubeadm configuration, pass the custom pause image repository and tag to the kubelet via –pod-infra-container-image not only for Docker but for all container runtimes. This flag tells the kubelet that it should not garbage collect the image. (#99476, @neolit123) [SIG Cluster Lifecycle]
    • Kubeadm: perform pre-flight validation on host/node name upon kubeadm init and kubeadm join, showing warnings on non-compliant names (#99194, @pacoxu)
    • Kubectl version changed to write a warning message to stderr if the client and server version difference exceeds the supported version skew of +/-1 minor version. (#98250, @brianpursley) [SIG CLI]
    • Kubectl: Add --use-protocol-buffers flag to kubectl top pods and nodes. (#96655, @serathius)
    • Kubectl: kubectl get will omit managed fields by default now. Users could set --show-managed-fields to true to show managedFields when the output format is either json or yaml. (#96878, @knight42) [SIG CLI and Testing]
    • Kubectl: a Pod can be preselected as default container using kubectl.kubernetes.io/default-container annotation (#99833, @mengjiao-liu)
    • Kubectl: add bash-completion for comma separated list on kubectl get (#98301, @phil9909)
    • Kubernetes is now built using go1.15.8 (#98834, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing]
    • Kubernetes is now built with Golang 1.16 (#98572, @justaugustus) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Release and Testing]
    • Kubernetes is now built with Golang 1.16.1 (#100106, @justaugustus) [SIG Cloud Provider, Instrumentation, Release and Testing]
    • Metrics can now be disabled explicitly via a command line flag (i.e. ‘–disabled-metrics=metric1,metric2’) (#99217, @logicalhan)
    • New admission controller DenyServiceExternalIPs is available. Clusters which do not need the Service externalIPs feature should enable this controller and be more secure. (#97395, @thockin)
    • Overall, enable the feature of PreferNominatedNode will improve the performance of scheduling where preemption might frequently happen, but in theory, enable the feature of PreferNominatedNode, the pod might not be scheduled to the best candidate node in the cluster. (#93179, @chendave) [SIG Scheduling and Testing]
    • Persistent Volumes formatted with the btrfs filesystem will now automatically resize when expanded. (#99361, @Novex) [SIG Storage]
    • Port the devicemanager to Windows node to allow device plugins like directx (#93285, @aarnaud) [SIG Node, Testing and Windows]
    • Removes cAdvisor JSON metrics (/stats/container, /stats//, /stats////) from the kubelet. (#99236, @pacoxu)
    • Rename metrics etcd_object_counts to apiserver_storage_object_counts and mark it as stable. The original etcd_object_counts metrics name is marked as “Deprecated” and will be removed in the future. (#99785, @erain) [SIG API Machinery, Instrumentation and Testing]
    • Sysctls graduates to General Availability and thus unconditionally enabled. (#99158, @wgahnagl)
    • The Kubernetes pause image manifest list now contains an image for Windows Server 20H2. (#97322, @claudiubelu) [SIG Windows]
    • The NodeAffinity plugin implements the PreFilter extension, offering enhanced performance for Filter. (#99213, @AliceZhang2016) [SIG Scheduling]
    • The CronJobControllerV2 feature flag graduates to Beta and set to be enabled by default. (#98878, @soltysh)
    • The EndpointSlice mirroring controller mirrors endpoints annotations and labels to the generated endpoint slices, it also ensures that updates on any of these fields are mirrored. The well-known annotation endpoints.kubernetes.io/last-change-trigger-time is skipped and not mirrored. (#98116, @aojea)
    • The RunAsGroup feature has been promoted to GA in this release. (#94641, @krmayankk) [SIG Auth and Node]
    • The ServiceAccountIssuerDiscovery feature has graduated to GA, and is unconditionally enabled. The ServiceAccountIssuerDiscovery feature-gate will be removed in 1.22. (#98553, @mtaufen) [SIG API Machinery, Auth and Testing]
    • The TTLAfterFinished feature flag is now beta and enabled by default (#98678, @ahg-g)
    • The apimachinery util/net function used to detect the bind address ResolveBindAddress() takes into consideration global IP addresses on loopback interfaces when 1) the host has default routes, or 2) there are no global IPs on those interfaces in order to support more complex network scenarios like BGP Unnumbered RFC 5549 (#95790, @aojea) [SIG Network]
    • The feature gate RootCAConfigMap graduated to GA in v1.21 and therefore will be unconditionally enabled. This flag will be removed in v1.22 release. (#98033, @zshihang)
    • The pause image upgraded to v3.4.1 in kubelet and kubeadm for both Linux and Windows. (#98205, @pacoxu)
    • Update pause container to run as pseudo user and group 65535:65535. This implies the release of version 3.5 of the container images. (#97963, @saschagrunert) [SIG CLI, Cloud Provider, Cluster Lifecycle, Node, Release, Security and Testing]
    • Update the latest validated version of Docker to 20.10 (#98977, @neolit123) [SIG CLI, Cluster Lifecycle and Node]
    • Upgrade node local dns to 1.17.0 for better IPv6 support (#99749, @pacoxu) [SIG Cloud Provider and Network]
    • Upgrades IPv6Dualstack to Beta and turns it on by default. New clusters or existing clusters are not be affected until an actor starts adding secondary Pods and service CIDRS CLI flags as described here: IPv4/IPv6 Dual-stack (#98969, @khenidak)
    • Users might specify the kubectl.kubernetes.io/default-container annotation in a Pod to preselect container for kubectl commands. (#99581, @mengjiao-liu) [SIG CLI]
    • When downscaling ReplicaSets, ready and creation timestamps are compared in a logarithmic scale. (#99212, @damemi) [SIG Apps and Testing]
    • When the kubelet is watching a ConfigMap or Secret purely in the context of setting environment variables for containers, only hold that watch for a defined duration before cancelling it. This change reduces the CPU and memory usage of the kube-apiserver in large clusters. (#99393, @chenyw1990) [SIG API Machinery, Node and Testing]
    • WindowsEndpointSliceProxying feature gate has graduated to beta and is enabled by default. This means kube-proxy will read from EndpointSlices instead of Endpoints on Windows by default. (#99794, @robscott) [SIG Network]
    • kubectl wait ensures that observedGeneration >= generation to prevent stale state reporting. An example scenario can be found on CRD updates. (#97408, @KnicKnic)

    Failing Test

    • Fixes the should receive events on concurrent watches in same order conformance test to work properly on clusters that auto-create additional configmaps in namespaces (#101950, @liggitt) [SIG API Machinery and Testing]
    • Fixed generic ephemeal volumes with OwnerReferencesPermissionEnforcement admission plugin enabled. (#101186, @jsafrane) [SIG Auth and Storage]
    • Resolves an issue with the “ServiceAccountIssuerDiscovery should support OIDC discovery” conformance test failing on clusters which are configured with issuers outside the cluster (#101725, @mtaufen) [SIG Auth and Testing]
    • Escape the special characters like [, ] and that exist in vsphere windows path (#98830, @liyanhui1228) [SIG Storage and Windows]
    • Kube-proxy: fix a bug on UDP NodePort Services where stale connection tracking entries may blackhole the traffic directed to the NodePort (#98305, @aojea)
    • Kubelet: fixes a bug in the HostPort dockershim implementation that caused the conformance test “HostPort validates that there is no conflict between pods with same hostPort but different hostIP and protocol” to fail. (#98755, @aojea) [SIG Cloud Provider, Network and Node]

    Bug or Regression

    • Fix NodeAuthenticator tests in dualstack (#104840, @ardaguclu) [SIG Auth and Testing]
    • Fix: skip case sensitivity when checking Azure NSG rules fix: ensure InstanceShutdownByProviderID return false for creating Azure VMs (#104447, @feiskyer) [SIG Cloud Provider]
    • Fixed occasional pod cgroup freeze when using cgroup v1 and systemd driver. Fixed “failed to create container … unit already exists” when using cgroup v1 and systemd driver. (#104530, @kolyshkin) [SIG CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Storage and Testing]
    • Kube-proxy: delete stale conntrack UDP entries for loadbalancer ingress IP. (#104151, @aojea) [SIG Network]
    • Metrics changes: Fix exposed buckets of scheduler_volume_scheduling_duration_seconds_bucket metric (#100720, @dntosas) [SIG Apps, Instrumentation, Scheduling and Storage]
    • Pass additional flags to subpath mount to avoid flakes in certain conditions (#104347, @mauriciopoppe) [SIG Storage]
    • When using kubectl replace (or the equivalent API call) on a Service, the caller no longer needs to do a read-modify-write cycle to fetch the allocated values for .spec.clusterIP and .spec.ports[].nodePort. Instead the API server will automatically carry these forward from the original object when the new object does not specify them. (#104673, @thockin) [SIG Network]
    • Disable aufs module for gce clusters (#103831, @lizhuqi) [SIG Cloud Provider]
    • Fix kube-apiserver metric reporting for the deprecated watch path of /api//watch/… (#104190, @wojtek-t) [SIG API Machinery and Instrumentation]
    • Fix the code is leaking the defaulting between unrelated pod instances. (#103284, @kebe7jun) [SIG CLI]
    • Fix: Provide IPv6 support for internal load balancer (#103794, @nilo19) [SIG Cloud Provider]
    • Fix: cleanup outdated routes (#102935, @nilo19) [SIG Cloud Provider]
    • Fix: delete non existing disk issue (#102083, @andyzhangx) [SIG Cloud Provider]
    • Fix: ignore not a VMSS error for VMAS nodes in reconcileBackendPools (#103997, @nilo19) [SIG Cloud Provider]
    • Fix: return empty VMAS name if using standalone VM (#103470, @nilo19) [SIG Cloud Provider]
    • Fixed a bug that scheduler extenders are not called on preemptions (#103019, @ordovicia) [SIG Scheduling]
    • Fixes an issue cleaning up CertificateSigningRequest objects with an unparseable status.certificate field (#103948, @liggitt) [SIG Apps and Auth]
    • Fixes issue with websocket-based watches of Service objects not closing correctly on timeout (#102541, @liggitt) [SIG API Machinery and Testing]
    • Fix scoring for NodeResourcesMostAllocated and NodeResourcesBalancedAllocation plugins when nodes have containers with no requests. This was leaving to under-utilization of small nodes. (#102925, @alculquicondor) [SIG Scheduling]
    • ServiceOwnsFrontendIP shouldn’t report error when the public IP doesn’t match (#102516, @nilo19) [SIG Cloud Provider]
    • Switch scheduler to generate the merge patch on pod status instead of the full pod (#103133, @marwanad) [SIG Scheduling]
    • VSphere: Fix regression during attach disk if datastore is within a storage folder or datastore cluster. (#102969, @gnufied) [SIG Cloud Provider]
    • Added jitter factor to lease controller that better smears load on kube-apiserver over time. (#101652, @marseel) [SIG API Machinery and Scalability]
    • Avoid caching the Azure VMSS instances whose network profile is nil (#100948, @feiskyer) [SIG Cloud Provider]
    • Azure: avoid setting cached Sku when updating VMSS and VMSS instances (#102005, @feiskyer) [SIG Cloud Provider]
    • Fix a bug on the endpoint slices mirroring controller where endpoint NotReadyAddresses were mirrored as Ready to the corresponding EndpointSlice (#102683, @aojea) [SIG Apps and Network]
    • Fix a bug that a preemptor pod may exist as a phantom in the scheduler. (#102498, @Huang-Wei) [SIG Scheduling]
    • Fix removing pods from podTopologyHints mapping (#101892, @aheng-ch) [SIG Node]
    • Fix resource enforcement when using systemd cgroup driver (#102147, @kolyshkin) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Storage and Testing]
    • Fix: avoid nil-pointer panic when checking the frontend IP configuration (#101739, @nilo19) [SIG Cloud Provider]
    • Fix: not tagging static public IP (#101752, @nilo19) [SIG Cloud Provider]
    • Fixed false-positive uncertain volume attachments, which led to unexpected detachment of CSI migrated volumes (#101737, @Jiawei0227) [SIG Apps and Storage]
    • Fixed garbage collection of dangling VolumeAttachments for PersistentVolumes migrated to CSI on startup of kube-controller-manager. (#102176, @timebertt) [SIG Apps and Storage]
    • Kube-proxy log now shows the “Skipping topology aware endpoint filtering since no hints were provided for zone” warning under the right conditions (#101857, @dervoeti) [SIG Network]
    • Kubeadm: remove the “ephemeral_storage” request from the etcd static pod that kubeadm deploys on stacked etcd control plane nodes. This request has caused sporadic failures on some setups due to a problem in the kubelet with cadvisor and the LocalStorageCapacityIsolation feature gate. See this issue for more details: https://github.com/kubernetes/kubernetes/issues/99305 (#102673, @jackfrancis) [SIG Cluster Lifecycle]
    • Kubeadm: when using a custom image repository for CoreDNS kubeadm now will append the “coredns” image name instead of “coredns/coredns”, thus restoring the behaviour existing before the v1.21 release. Users who rely on nested folder for the coredns image should set the “clusterConfiguration.dns.imageRepository” value including the nested path name (e.g using “registry.company.xyz/coredns” will force kubeadm to use “registry.company.xyz/coredns/coredns” image). No action is needed if using the default registry (k8s.gcr.io). (#102502, @ykakarap) [SIG Cluster Lifecycle]
    • Register/Deregister Targets in chunks for AWS TargetGroup (#101592, @M00nF1sh) [SIG Cloud Provider]
    • Respect annotation size limit for server-side apply updates to the client-side apply annotation. Also, fix opt-out of this behavior by setting the client-side apply annotation to the empty string. (#102105, @julianvmodesto) [SIG API Machinery]
    • Reverted the previous fix for portforward cleanup because it introduced a kubelet regression which can lead into segmentation faults. (#102587, @saschagrunert) [SIG API Machinery and Node]
    • Adds node event handlers to dual stack kube-proxy implementation to fix Topology Aware Hints. (#101054, @robscott) [SIG Network]
    • Azurefile: Normalize share name to not include capital letters (#100731, @kassarl) [SIG Cloud Provider and Storage]
    • EndpointSlice IP validation now matches Endpoints IP validation. (#101084, @robscott) [SIG Apps and Network]
    • Ensure service deleted when the Azure resource group has been deleted (#100944, @feiskyer) [SIG Cloud Provider]
    • Fix EndpointSlice describe panic when an Endpoint doesn’t have zone (#101025, @tnqn) [SIG CLI]
    • Fix display of Job completion mode on kubectl describe (#101198, @alculquicondor) [SIG CLI]
    • Fix: azure file inline volume namespace issue in csi migration translation (#101235, @andyzhangx) [SIG Apps, Cloud Provider, Node and Storage]
    • Fix: set “host is down” as corrupted mount (#101398, @andyzhangx) [SIG Cloud Provider and Storage]
    • Fixed a bug where startupProbe stopped working after a container’s first restart (#101093, @wzshiming) [SIG Node]
    • Fixed port-forward memory leak for long-running and heavily used connections. (#99839, @saschagrunert) [SIG API Machinery and Node]
    • Kubectl create service now respects namespace flag (#101005, @zxh326) [SIG CLI]
    • Kubelet: improve the performance when waiting for a synchronization of the node list with the kube-apiserver (#99336, @neolit123) [SIG Node]
    • No support endpointslice in linux userpace mode (#101504, @JornShen) [SIG Network]
    • Renames the timeout field for the DelegatingAuthenticationOptions to TokenRequestTimeout and set the timeout only for the token review client. Previously the timeout was also applied to watches making them reconnecting every 10 seconds. (#101102, @p0lyn0mial) [SIG API Machinery, Auth and Cloud Provider]
    • Respect ExecProbeTimeout=false for dockershim (#101127, @jackfrancis) [SIG Node and Testing]
    • Upgrades functionality of kubectl kustomize as described at https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv4.1.0 (#101177, @KnVerey) [SIG CLI]
    • AcceleratorStats will be available in the Summary API of kubelet when cri_stats_provider is used. (#96873, @ruiwen-zhao) [SIG Node]
    • All data is no longer automatically deleted when a failure is detected during creation of the volume data file on a CSI volume. Now only the data file and volume path is removed. (#96021, @huffmanca)
    • Clean ReplicaSet by revision instead of creation timestamp in deployment controller (#97407, @waynepeking348) [SIG Apps]
    • Cleanup subnet in frontend IP configs to prevent huge subnet request bodies in some scenarios. (#98133, @nilo19) [SIG Cloud Provider]
    • Client-go exec credential plugins will pass stdin only when interactive terminal is detected on stdin. This fixes a bug where previously it was checking if stdout is an interactive terminal. (#99654, @ankeesler)
    • Cloud-controller-manager: routes controller should not depend on –allocate-node-cidrs (#97029, @andrewsykim) [SIG Cloud Provider and Testing]
    • Cluster Autoscaler version bump to v1.20.0 (#97011, @towca)
    • Creating a PVC with DataSource should fail for non-CSI plugins. (#97086, @xing-yang) [SIG Apps and Storage]
    • EndpointSlice controller is now less likely to emit FailedToUpdateEndpointSlices events. (#99345, @robscott) [SIG Apps and Network]
    • EndpointSlice controllers are less likely to create duplicate EndpointSlices. (#100103, @robscott) [SIG Apps and Network]
    • EndpointSliceMirroring controller is now less likely to emit FailedToUpdateEndpointSlices events. (#99756, @robscott) [SIG Apps and Network]
    • Ensure all vSphere nodes are are tracked by volume attach-detach controller (#96689, @gnufied)
    • Ensure empty string annotations are copied over in rollbacks. (#94858, @waynepeking348)
    • Ensure only one LoadBalancer rule is created when HA mode is enabled (#99825, @feiskyer) [SIG Cloud Provider]
    • Ensure that client-go’s EventBroadcaster is safe (non-racy) during shutdown. (#95664, @DirectXMan12) [SIG API Machinery]
    • Explicitly pass KUBE_BUILD_CONFORMANCE=y in package-tarballs to reenable building the conformance tarballs. (#100571, @puerco)
    • Fix Azure file migration e2e test failure when CSIMigration is turned on. (#97877, @andyzhangx)
    • Fix CSI-migrated inline EBS volumes failing to mount if their volumeID is prefixed by aws:// (#96821, @wongma7) [SIG Storage]
    • Fix CVE-2020-8555 for Gluster client connections. (#97922, @liggitt) [SIG Storage]
    • Fix NPE in ephemeral storage eviction (#98261, @wzshiming) [SIG Node]
    • Fix PermissionDenied issue on SMB mount for Windows (#99550, @andyzhangx)
    • Fix bug that would let the Horizontal Pod Autoscaler scale down despite at least one metric being unavailable/invalid (#99514, @mikkeloscar) [SIG Apps and Autoscaling]
    • Fix cgroup handling for systemd with cgroup v2 (#98365, @odinuge) [SIG Node]
    • Fix counting error in service/nodeport/loadbalancer quota check (#97451, @pacoxu) [SIG API Machinery, Network and Testing]
    • Fix errors when accessing Windows container stats for Dockershim (#98510, @jsturtevant) [SIG Node and Windows]
    • Fix kube-proxy container image architecture for non amd64 images. (#98526, @saschagrunert)
    • Fix missing cadvisor machine metrics. (#97006, @lingsamuel) [SIG Node]
    • Fix nil VMSS name when setting service to auto mode (#97366, @nilo19) [SIG Cloud Provider]
    • Fix privileged config of Pod Sandbox which was previously ignored. (#96877, @xeniumlee)
    • Fix the panic when kubelet registers if a node object already exists with no Status.Capacity or Status.Allocatable (#95269, @SataQiu) [SIG Node]
    • Fix the regression with the slow pods termination. Before this fix pods may take an additional time to terminate - up to one minute. Reversing the change that ensured that CNI resources cleaned up when the pod is removed on API server. (#97980, @SergeyKanzhelev) [SIG Node]
    • Fix to recover CSI volumes from certain dangling attachments (#96617, @yuga711) [SIG Apps and Storage]
    • Fix: azure file latency issue for metadata-heavy workloads (#97082, @andyzhangx) [SIG Cloud Provider and Storage]
    • Fixed Cinder volume IDs on OpenStack Train (#96673, @jsafrane) [SIG Cloud Provider]
    • Fixed FibreChannel volume plugin corrupting filesystems on detach of multipath volumes. (#97013, @jsafrane) [SIG Storage]
    • Fixed a bug in kubelet that will saturate CPU utilization after containerd got restarted. (#97174, @hanlins) [SIG Node]
    • Fixed a bug that causes smaller number of conntrack-max being used under CPU static policy. (#99225, @xh4n3) (#99613, @xh4n3) [SIG Network]
    • Fixed a bug that on k8s nodes, when the policy of INPUT chain in filter table is not ACCEPT, healthcheck nodeport would not work. Added iptables rules to allow healthcheck nodeport traffic. (#97824, @hanlins) [SIG Network]
    • Fixed a bug that the kubelet cannot start on BtrfS. (#98042, @gjkim42) [SIG Node]
    • Fixed a race condition on API server startup ensuring previously created webhook configurations are effective before the first write request is admitted. (#95783, @roycaihw) [SIG API Machinery]
    • Fixed an issue with garbage collection failing to clean up namespaced children of an object also referenced incorrectly by cluster-scoped children (#98068, @liggitt) [SIG API Machinery and Apps]
    • Fixed authentication_duration_seconds metric scope. Previously, it included whole apiserver request duration which yields inaccurate results. (#99944, @marseel)
    • Fixed bug in CPUManager with race on container map access (#97427, @klueska) [SIG Node]
    • Fixed bug that caused cAdvisor to incorrectly detect single-socket multi-NUMA topology. (#99315, @iwankgb) [SIG Node]
    • Fixed cleanup of block devices when /var/lib/kubelet is a symlink. (#96889, @jsafrane) [SIG Storage]
    • Fixed no effect namespace when exposing deployment with –dry-run=client. (#97492, @masap) [SIG CLI]
    • Fixed provisioning of Cinder volumes migrated to CSI when StorageClass with AllowedTopologies was used. (#98311, @jsafrane) [SIG Storage]
    • Fixes a bug of identifying the correct containerd process. (#97888, @pacoxu)
    • Fixes add-on manager leader election to use leases instead of endpoints, similar to what kube-controller-manager does in 1.20 (#98968, @liggitt)
    • Fixes connection errors when using --volume-host-cidr-denylist or --volume-host-allow-local-loopback (#98436, @liggitt) [SIG Network and Storage]
    • Fixes problem where invalid selector on PodDisruptionBudget leads to a nil pointer dereference that causes the Controller manager to crash loop. (#98750, @mortent)
    • Fixes spurious errors about IPv6 in kube-proxy logs on nodes with IPv6 disabled. (#99127, @danwinship)
    • Fixing a bug where a failed node may not have the NoExecute taint set correctly (#96876, @howieyuen) [SIG Apps and Node]
    • GCE Internal LoadBalancer sync loop will now release the ILB IP address upon sync failure. An error in ILB forwarding rule creation will no longer leak IP addresses. (#97740, @prameshj) [SIG Cloud Provider and Network]
    • Ignore update pod with no new images in alwaysPullImages admission controller (#96668, @pacoxu) [SIG Apps, Auth and Node]
    • Improve speed of vSphere PV provisioning and reduce number of API calls (#100054, @gnufied) [SIG Cloud Provider and Storage]
    • KUBECTL_EXTERNAL_DIFF now accepts equal sign for additional parameters. (#98158, @dougsland) [SIG CLI]
    • Kube-apiserver: an update of a pod with a generic ephemeral volume dropped that volume if the feature had been disabled since creating the pod with such a volume (#99446, @pohly) [SIG Apps, Node and Storage]
    • Kube-proxy: remove deprecated –cleanup-ipvs flag of kube-proxy, and make –cleanup flag always to flush IPVS (#97336, @maaoBit) [SIG Network]
    • Kubeadm installs etcd v3.4.13 when creating cluster v1.19 (#97244, @pacoxu)
    • Kubeadm: Fixes a kubeadm upgrade bug that could cause a custom CoreDNS configuration to be replaced with the default. (#97016, @rajansandeep) [SIG Cluster Lifecycle]
    • Kubeadm: Some text in the kubeadm upgrade plan output has changed. If you have scripts or other automation that parses this output, please review these changes and update your scripts to account for the new output. (#98728, @stmcginnis) [SIG Cluster Lifecycle]
    • Kubeadm: fix a bug in the host memory detection code on 32bit Linux platforms (#97403, @abelbarrera15) [SIG Cluster Lifecycle]
    • Kubeadm: fix a bug where “kubeadm join” would not properly handle missing names for existing etcd members. (#97372, @ihgann) [SIG Cluster Lifecycle]
    • Kubeadm: fix a bug where “kubeadm upgrade” commands can fail if CoreDNS v1.8.0 is installed. (#97919, @neolit123) [SIG Cluster Lifecycle]
    • Kubeadm: fix a bug where external credentials in an existing admin.conf prevented the CA certificate to be written in the cluster-info ConfigMap. (#98882, @kvaps) [SIG Cluster Lifecycle]
    • Kubeadm: get k8s CI version markers from k8s infra bucket (#98836, @hasheddan) [SIG Cluster Lifecycle and Release]
    • Kubeadm: skip validating pod subnet against node-cidr-mask when allocate-node-cidrs is set to be false (#98984, @SataQiu) [SIG Cluster Lifecycle]
    • Kubectl logs: --ignore-errors is now honored by all containers, maintaining consistency with parallelConsumeRequest behavior. (#97686, @wzshiming)
    • Kubectl-convert: Fix no kind "Ingress" is registered for version error (#97754, @wzshiming)
    • Kubectl: Fixed panic when describing an ingress backend without an API Group (#100505, @lauchokyip) [SIG CLI]
    • Kubelet now cleans up orphaned volume directories automatically (#95301, @lorenz) [SIG Node and Storage]
    • Kubelet.exe on Windows now checks that the process running as administrator and the executing user account is listed in the built-in administrators group. This is the equivalent to checking the process is running as uid 0. (#96616, @perithompson) [SIG Node and Windows]
    • Kubelet: Fix kubelet from panic after getting the wrong signal (#98200, @wzshiming) [SIG Node]
    • Kubelet: Fix repeatedly acquiring the inhibit lock (#98088, @wzshiming) [SIG Node]
    • Kubelet: Fixed the bug of getting the number of cpu when the number of cpu logical processors is more than 64 in windows (#97378, @hwdef) [SIG Node and Windows]
    • Limits lease to have 1000 maximum attached objects. (#98257, @lingsamuel)
    • Mitigate CVE-2020-8555 for kube-up using GCE by preventing local loopback folume hosts. (#97934, @mattcary) [SIG Cloud Provider and Storage]
    • On single-stack configured (IPv4 or IPv6, but not both) clusters, Services which are both headless (no clusterIP) and selectorless (empty or undefined selector) will report ipFamilyPolicy RequireDualStack and will have entries in ipFamilies[] for both IPv4 and IPv6. This is a change from alpha, but does not have any impact on the manually-specified Endpoints and EndpointSlices for the Service. (#99555, @thockin) [SIG Apps and Network]
    • Performance regression #97685 has been fixed. (#97860, @MikeSpreitzer) [SIG API Machinery]
    • Pod Log stats for windows now reports metrics (#99221, @jsturtevant) [SIG Node, Storage, Testing and Windows]
    • Pod status updates faster when reacting on probe results. The first readiness probe will be called faster when startup probes succeeded, which will make Pod status as ready faster. (#98376, @matthyx)
    • Readjust kubelet_containers_per_pod_count buckets to only show metrics greater than 1. (#98169, @wawa0210)
    • Remove CSI topology from migrated in-tree gcepd volume. (#97823, @Jiawei0227) [SIG Cloud Provider and Storage]
    • Requests with invalid timeout parameters in the request URL now appear in the audit log correctly. (#96901, @tkashem) [SIG API Machinery and Testing]
    • Resolve a “concurrent map read and map write” crashing error in the kubelet (#95111, @choury) [SIG Node]
    • Resolves spurious Failed to list *v1.Secret or Failed to list *v1.ConfigMap messages in kubelet logs. (#99538, @liggitt) [SIG Auth and Node]
    • ResourceQuota of an entity now inclusively calculate Pod overhead (#99600, @gjkim42)
    • Return zero time (midnight on Jan. 1, 1970) instead of negative number when reporting startedAt and finishedAt of the not started or a running Pod when using dockershim as a runtime. (#99585, @Iceber)
    • Reverts breaking change to inline AzureFile volumes; referenced secrets are now searched for in the same namespace as the pod as in previous releases. (#100563, @msau42)
    • Scores from InterPodAffinity have stronger differentiation. (#98096, @leileiwan) [SIG Scheduling]
    • Specifying the KUBE_TEST_REPO environment variable when e2e tests are executed will instruct the test infrastructure to load that image from a location within the specified repo, using a predefined pattern. (#93510, @smarterclayton) [SIG Testing]
    • Static pods will be deleted gracefully. (#98103, @gjkim42) [SIG Node]
    • Sync node status during kubelet node shutdown. Adds an pod admission handler that rejects new pods when the node is in progress of shutting down. (#98005, @wzshiming) [SIG Node]
    • The calculation of pod UIDs for static pods has changed to ensure each static pod gets a unique value - this will cause all static pod containers to be recreated/restarted if an in-place kubelet upgrade from 1.20 to 1.21 is performed. Note that draining pods before upgrading the kubelet across minor versions is the supported upgrade path. (#87461, @bboreham) [SIG Node]
    • The maximum number of ports allowed in EndpointSlices has been increased from 100 to 20,000 (#99795, @robscott) [SIG Network]
    • Truncates a message if it hits the NoteLengthLimit when the scheduler records an event for the pod that indicates the pod has failed to schedule. (#98715, @carlory)
    • Updated k8s.gcr.io/ingress-gce-404-server-with-metrics-amd64 to a version that serves /metrics endpoint on a non-default port. (#97621, @vbannai) [SIG Cloud Provider]
    • Updates the commands `
      • kubectl kustomize {arg}
      • kubectl apply -k {arg} `to use same code as kustomize CLI v4.0.5 (#98946, @monopole)
    • Use force unmount for NFS volumes if regular mount fails after 1 minute timeout (#96844, @gnufied) [SIG Storage]
    • Use network.Interface.VirtualMachine.ID to get the binded VM Skip standalone VM when reconciling LoadBalancer (#97635, @nilo19) [SIG Cloud Provider]
    • Using exec auth plugins with kubectl no longer results in warnings about constructing many client instances from the same exec auth config. (#97857, @liggitt) [SIG API Machinery and Auth]
    • When a CNI plugin returns dual-stack pod IPs, kubelet will now try to respect the “primary IP family” of the cluster by picking a primary pod IP of the same family as the (primary) node IP, rather than assuming that the CNI plugin returned the IPs in the order the administrator wanted (since some CNI plugins don’t allow configuring this). (#97979, @danwinship) [SIG Network and Node]
    • When dynamically provisioning Azure File volumes for a premium account, the requested size will be set to 100GB if the request is initially lower than this value to accommodate Azure File requirements. (#99122, @huffmanca) [SIG Cloud Provider and Storage]
    • When using Containerd on Windows, the C:\Windows\System32\drivers\etc\hosts file will now be managed by kubelet. (#83730, @claudiubelu)
    • VolumeBindingArgs now allow BindTimeoutSeconds to be set as zero, while the value zero indicates no waiting for the checking of volume binding operation. (#99835, @chendave) [SIG Scheduling and Storage]
    • kubectl exec and kubectl attach now honor the --quiet flag which suppresses output from the local binary that could be confused by a script with the remote command output (all non-failure output is hidden). In addition, print inline with exec and attach the list of alternate containers when we default to the first spec.container. (#99004, @smarterclayton) [SIG CLI]

    Other (Cleanup or Flake)

    • Kube-apiserver: sets an upper-bound on the lifetime of idle keep-alive connections and time to read the headers of incoming requests (#103958, @liggitt) [SIG API Machinery and Node]
    • Client-go: reduce verbosity of “Starting/Stopping reflector” messages to 3 again (#102788, @pohly) [SIG API Machinery]
    • Update the Debian images to pick up CVE fixes in the base images:
      • Update the debian-base image to v1.7.0
      • Update the debian-iptables image to v1.6.1 (#102340, @cpanato) [SIG API Machinery and Testing]
    • APIs for kubelet annotations and labels from k8s.io/kubernetes/pkg/kubelet/apis are now moved under k8s.io/kubelet/pkg/apis/ (#98931, @michaelbeaumont)
    • Apiserver_request_duration_seconds is promoted to stable status. (#99925, @logicalhan) [SIG API Machinery, Instrumentation and Testing]
    • Bump github.com/Azure/go-autorest/autorest to v0.11.12 (#97033, @patrickshan) [SIG API Machinery, CLI, Cloud Provider and Cluster Lifecycle]
    • Clients required to use go1.15.8+ or go1.16+ if kube-apiserver has the goaway feature enabled to avoid unexpected data race condition. (#98809, @answer1991)
    • Delete deprecated service.beta.kubernetes.io/azure-load-balancer-mixed-protocols mixed procotol annotation in favor of the MixedProtocolLBService feature (#97096, @nilo19) [SIG Cloud Provider]
    • EndpointSlice generation is now incremented when labels change. (#99750, @robscott) [SIG Network]
    • Featuregate AllowInsecureBackendProxy graduates to GA and unconditionally enabled. (#99658, @deads2k)
    • Increase timeout for pod lifecycle test to reach pod status=ready (#96691, @hh)
    • Increased CSINodeIDMaxLength from 128 bytes to 192 bytes. (#98753, @Jiawei0227)
    • Kube-apiserver: The OIDC authenticator no longer waits 10 seconds before attempting to fetch the metadata required to verify tokens. (#97693, @enj) [SIG API Machinery and Auth]
    • Kube-proxy: Traffic from the cluster directed to ExternalIPs is always sent directly to the Service. (#96296, @aojea) [SIG Network and Testing]
    • Kubeadm: change the default image repository for CI images from ‘gcr.io/kubernetes-ci-images’ to ‘gcr.io/k8s-staging-ci-images’ (#97087, @SataQiu) [SIG Cluster Lifecycle]
    • Kubectl: The deprecated kubectl alpha debug command is removed. Use kubectl debug instead. (#98111, @pandaamanda) [SIG CLI]
    • Kubelet command line flags related to dockershim are now showing deprecation message as they will be removed along with dockershim in future release. (#98730, @dims)
    • Official support to build kubernetes with docker-machine / remote docker is removed. This change does not affect building kubernetes with docker locally. (#97618, @jherrera123) [SIG Release and Testing]
    • Process start time on Windows now uses current process information (#97491, @jsturtevant) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Windows]
    • Resolves flakes in the Ingress conformance tests due to conflicts with controllers updating the Ingress object (#98430, @liggitt) [SIG Network and Testing]
    • The AttachVolumeLimit feature gate (GA since v1.17) has been removed and now unconditionally enabled. (#96539, @ialidzhikov)
    • The CSINodeInfo feature gate that is GA since v1.17 is unconditionally enabled, and can no longer be specified via the --feature-gates argument. (#96561, @ialidzhikov) [SIG Apps, Auth, Scheduling, Storage and Testing]
    • The apiserver_request_total metric is promoted to stable status and no longer has a content-type dimensions, so any alerts/charts which presume the existence of this will fail. This is however, unlikely to be the case since it was effectively an unbounded dimension in the first place. (#99788, @logicalhan)
    • The default delegating authorization options now allow unauthenticated access to healthz, readyz, and livez. A system:masters user connecting to an authz delegator will not perform an authz check. (#98325, @deads2k) [SIG API Machinery, Auth, Cloud Provider and Scheduling]
    • The deprecated feature gates CSIDriverRegistry, BlockVolume and CSIBlockVolume are now unconditionally enabled and can no longer be specified in component invocations. (#98021, @gavinfish) [SIG Storage]
    • The deprecated feature gates RotateKubeletClientCertificate, AttachVolumeLimit, VolumePVCDataSource and EvenPodsSpread are now unconditionally enabled and can no longer be specified in component invocations. (#97306, @gavinfish) [SIG Node, Scheduling and Storage]
    • The e2e suite can be instructed not to wait for pods in kube-system to be ready or for all nodes to be ready by passing --allowed-not-ready-nodes=-1 when invoking the e2e.test program. This allows callers to run subsets of the e2e suite in scenarios other than perfectly healthy clusters. (#98781, @smarterclayton) [SIG Testing]
    • The feature gates WindowsGMSA and WindowsRunAsUserName that are GA since v1.18 are now removed. (#96531, @ialidzhikov) [SIG Node and Windows]
    • The new -gce-zones flag on the e2e.test binary instructs tests that check for information about how the cluster interacts with the cloud to limit their queries to the provided zone list. If not specified, the current behavior of asking the cloud provider for all available zones in multi zone clusters is preserved. (#98787, @smarterclayton) [SIG API Machinery, Cluster Lifecycle and Testing]
    • Update cri-tools to v1.20.0 (#97967, @rajibmitra) [SIG Cloud Provider]
    • Windows nodes on GCE will take longer to start due to dependencies installed at node creation time. (#98284, @pjh) [SIG Cloud Provider]
    • apiserver_storage_objects (a newer version of etcd_object_counts) is promoted and marked as stable. (#100082, @logicalhan)

    app-operator 5.2.0

    Changed

    • Reject App CRs with version labels with the legacy 1.0.0 value.
    • Validate .spec.catalog using Catalog CRs instead of AppCatalog CRs.

    Fixed

    • Fix creating AppCatalog CRs in appcatalogsync resource.

    aws-cni 1.9.0

    cluster-operator 3.10.0

    Changed

    • Introducing v1alpha3 CR’s.

    external-dns 2.5.0

    Changed

    • Upgrade upstream external-dns from v0.8.0 to v0.9.0. The new release brings a lot of smaller improvements and bug fixes.

    cert-exporter 1.8.0

    Added

    • Add new cert_exporter_certificate_cr_not_after metric. This metric exports the status.notAfter field of cert-manager Certificate CR.

    Changed

    • Remove static certificate source label from cert_exporter_secret_not_after (static value secret) and cert_exporter_not_after (static value file) metrics.

    cert-manager 2.11.0

    Changed

    • Upgrade to upstream v1.5.4 (#191).
    • Add metadata to enable metrics scraping (#181).
    • Fix startubjob PSP (#191)
    • Upgrade to upstream v1.5.3 (#184). This is the first version compatible with Kubernetes 1.22.
    • Add metadata to enable metrics scraping (#181).
    • Update to upstream v1.4.2 (#174). This deprecates v1alpha2, v1alpha3 and v1beta1 versions of cert-manager.io and acme.cert-manager.io CRDs. Further information can be found in the upstream release notes of cert-manager.
    • Increase resource requests for the ClusterIssuer and CRD installation Jobs (#174) to prevent timeouts.

    chart-operator 2.19.0

    Removed

    • Remove tillermigration resource now Helm 3 migration is complete.

    Changed

    • Increase memory limit for deploying large charts in workload clusters.

    cluster-autoscaler 1.21.0-gs2

    Changed

    • Fix RBAC for cluster autoscaler 1.21.
    • Updated cluster-autoscaler to version 1.21.0.
    • Use new node selector node-role.kubernetes.io/master in place of deprecated one kubernetes.io/role.
    • Prepare helm values to configuration management.
    • Update architect-orb to v4.0.0.

    Added

    • Add VerticalPodAutoscaler resource to adjust limits automatically.

    external-dns 2.5.0

    Changed

    • Upgrade upstream external-dns from v0.8.0 to v0.9.0. The new release brings a lot of smaller improvements and bug fixes.

    kube-state-metrics 1.4.0

    Changed

    • Migrate to configuration management.
    • Update architect-orb to v4.0.0.

    metrics-server 1.5.0

    Changed

    • Bumped API version for RoleBinding to v1 as it was using a deprecated version (removed in 1.22).

    net-exporter 1.10.3

    Changed

    • Prepare helm values to configuration management.
    • Update architect-orb to v4.0.0.

    node-exporter 1.8.0

    Changed

    • Migrate to configuration management.
    • Update architect-orb to v4.0.0.

    aws-ebs-csi-driver 2.3.1

    Fixed

    • Enable permissions for ebs volume resizing by default.

    Changed

    • Bump aws-ebs-csi-driver version to v1.2.0
  • This release provides a security fix for CVE-2021-25741.

    Change details

    kubernetes 1.20.11

    Bug or Regression

    • Kube-proxy: delete stale conntrack UDP entries for loadbalancer ingress IP. (#104152, @aojea) [SIG Network]
    • Metrics changes: Fix exposed buckets of scheduler_volume_scheduling_duration_seconds_bucket metric (#100720, @dntosas) [SIG Apps, Instrumentation, Scheduling and Storage]
    • Pass additional flags to subpath mount to avoid flakes in certain conditions (#104348, @mauriciopoppe) [SIG Storage]
    • When using kubectl replace (or the equivalent API call) on a Service, the caller no longer needs to do a read-modify-write cycle to fetch the allocated values for .spec.clusterIP and .spec.ports[].nodePort. Instead the API server will automatically carry these forward from the original object when the new object does not specify them. (#104674, @thockin) [SIG Network]

    Other (Cleanup or Flake)

    • Kube-apiserver: sets an upper-bound on the lifetime of idle keep-alive connections and time to read the headers of incoming requests (#103958, @liggitt) [SIG API Machinery and Node]

    Dependencies

    Added

    Nothing has changed.

    Changed

    Nothing has changed.

    Removed

    Nothing has changed.

  • This release provides a bug fix for ebs-csi-driver to enable it to resize EBS volumes.

    Change details

    aws-ebs-csi-driver 2.2.1

    Fixed

    • Enable permissions for ebs volume resizing by default.
  • This release adds support to aws-operator to comply with the following additional AWS S3 policies:

    • s3-bucket-public-read-prohibited
    • s3-bucket-ssl-requests-only
    • s3-bucket-public-write-prohibited
    • s3-bucket-server-side-encryption-enabled
    • s3-bucket-logging-enabled.

    Change details

    aws-operator 10.7.1

    Added

    • Add security settings to S3 bucket to comply with aws policies s3-bucket-public-read-prohibited,s3-bucket-ssl-requests-only,s3-bucket-public-write-prohibited,s3-bucket-server-side-encryption-enabled,s3-bucket-logging-enabled, aws-operator will need additonal permissions s3:PutBucketPublicAccessBlock and s3:PutBucketPolicy.
  • This release provides a bug fix for ebs-csi-driver to enable it to resize EBS volumes.

    Change details

    aws-ebs-csi-driver 2.2.1

    Fixed

    • Enable permissions for ebs volume resizing by default.
  • This release provides stability improvements and bug fixes for various components.

    Highlights

    • Kubernetes 1.20.9;
    • Kiam 4.1

    Change details

    kubernetes 1.20.9

    Feature

    • Kubernetes 1.20.x is now built using Go 1.15.14 (#103677, @puerco) [SIG Cloud Provider, Instrumentation, Release and Testing]
    • Updates the following images to pick up CVE fixes:
      • debian to v1.8.0
      • debian-iptables to v1.6.5
      • setcap to v2.0.3 (#103235, @thejoycekung) [SIG API Machinery, Release and Testing]

    Bug or Regression

    • Fix scoring for NodeResourcesMostAllocated and NodeResourcesBalancedAllocation plugins when nodes have containers with no requests. This was leaving to under-utilization of small nodes. (#102925, @alculquicondor) [SIG Scheduling]
    • Switch scheduler to generate the merge patch on pod status instead of the full pod (#103133, @marwanad) [SIG Scheduling]
    • VSphere: Fix regression during attach disk if datastore is within a storage folder or datastore cluster. (#102999, @gnufied) [SIG Cloud Provider]

    Dependencies

    Added

    Nothing has changed.

    Changed

    • sigs.k8s.io/structured-merge-diff/v4: v4.0.3 → v4.1.2

    Removed

    Nothing has changed.

    etcd 3.4.16

    etcd server

    Package fileutil

    Metrics

    Dependency

    Go

    calico 3.15.5

    Bug fixes

    • Fix that calico/node would fail to set NetworkUnavailable to false for etcd clusters with mismatched nodenames node #949 (@caseydavenport)
    • Fixes a bug where IPv6 networks were not handled properly by the failsafe rules felix #2748 (@mgleung)
    • Fix that, after a netlink read failure, Felix would tight loop reading from a closed channel. Restart the event poll in that case. felix #2713 (@fasaxc)

    Other changes

    • FailsafeInboundHostPorts & FailsafeOutboundHostPorts now support restricting to specific cidrs. New format :: felix #2721 (@mgleung)

    app-operator 5.1.0

    Changed

    • Create AppCatalogEntry CRs into the same namespace of Catalog CR.
    • Include chart.keywords, chart.description and chart.upstreamChartVersion in AppCatalogEntry CRs.
    • Create AppCatalog CRs from Catalog CRs for compatibility with existing app-operator releases.
    • Prepare helm values to configuration management.
    • Use Catalog CRs in App controller.
    • Reconcile to Catalog CRs instead of AppCatalog.
    • Get Chart CRD from the GitHub resources.
    • Get metadata constants from k8smetadata library not apiextensions.

    Fixed

    • For the chart CR watcher get the kubeconfig secret from the chart-operator app CR to avoid hardcoding it.
    • Quote namespace in helm templates to handle numeric workload cluster IDs.

    aws-ebs-csi-driver 2.2.0

    Added

    • CRD for snapshot-controller.

    Changed

    • Update aws-ebs-csi-driver to v1.1.1.
    • Reduce default log level to 2.
    • Default volume resizing.

    kiam 2.0.0

    Changed

    • Upgrade kiam version to 4.1.
    • Update RBAC API version from v1beta1 to v1.
    • Add kind: Issuer and group: cert-manager.io to Certificate templates.

    cert-manager 2.8.0

    Changed

    • Label deployments with giantswarm.io/monitoring_basic_sli: "true". (#171)
    • Migrate values file structure to match config repo. (#172)

    coredns 1.6.0

    Changed

    • Make targetCPUUtilizationPercentage in HPA configurable.
    • Update coredns to upstream version 1.8.3.
    • Increase maximum replica count to 50 when using horizontal pod autoscaling.

    kube-state-metrics 1.3.1

    Changed

    • Set docker.io as the default registry

    metrics-server 1.3.0

    Added

    • Added new configuration value extraArgs.

    net-exporter 1.10.2

    Changed

    • Allow to customize dns service.
    • Only check pod existence on dial errors. Check pod deletion directly by IP instead of listing pods and searching.

    external-dns 2.4.0

    Changed

    • Upgrade upstream external-dns from v0.7.6 to v0.8.0.
    • Allow to configure the minimum interval between two consecutive synchronizations triggered from kubernetes events through externalDNS.minEventSyncInterval.

    cert-exporter 1.7.1

    Fixed

    • Fix configuration version in Chart.yaml.

    chart-operator 2.18.0

    Added

    • Add releasemaxhistory resource which ensures we retry at a reduced rate when there are repeated failed upgrades.
    • Proxy support in helm template.

    Changed

    • Upgrade Helm release when failed even if version or values have not changed to handle situations like failed webhooks where we should retry.
    • Prepare helm values to configuration management.
    • Update architect-orb to v3.0.0.
    • [CAPI] Add tolerations to start on NotReady nodes for installing CNI.
    • [CAPI] Create giantswarm-critical priority class.
    • [CAPI] Use host network to allow installing CNI packaged as an app.

    Fixed

    • Improve status message when helm release has failed max number of attempts.

    cluster-operator 3.9.0

    Changed

    Use app-operator-konfigure configmap for the app-operator per workload cluster.

  • This release provides support for Kubernetes 1.20. It also enables the Container Storage Interface (CSI) and the automatic termination of unhealthy nodes by default.

    Highlights

    • Kubernetes 1.20 support;
    • Container Storage Interface (CSI) enabled by default;
    • gp3 is now the default storage class instead of gp2;
    • Automatic termination of unhealthy nodes enabled by default;
    • Security fixes:
      • 49 Linux CVEs;
      • 5 openssl CVEs;
      • 5 nvidia-drivers CVEs;
      • 1 runc CVE;
      • 1 containerd CVE;
      • 1 Kubernetes CVE.

    Warning: From AWS workload cluster release v15.0.0, the automatic termination of unhealthy nodes is enabled by default. For more information about the feature and information how to disable it, please follow the official documentation.

    Change details

    app-operator 4.4.0

    Added

    • Add support for skip CRD flag when installing Helm releases;
    • Emit events when config maps and secrets referenced in App CRs are updated;
    • Cache k8sclient, helmclient for later use;
    • Apply the namespaceConfig to the desired chart;
    • Install apps in CAPI Workload Clusters;
    • Apply compatibleProvider, namespace metadata validation based on the relevant AppCatalogEntry CR;
    • Add annotations from Helm charts to AppCatalogEntry CRs;
    • Enable Vertical Pod Autoscaler.

    Fixed

    • Updated OperatorKit to v4.3.1 for Kubernetes 1.20 support;
    • Restore chart-operator when it had been deleted;
    • Use backoff in chart CR watcher to wait until kubeconfig secret exists;

    Changed

    • Updated Helm to v3.5.3;
    • Replace status webhook with chart CR status watcher;
    • Sort AppCatalogEntry CRs by version and created timestamp;
    • Watch cluster namespace for per workload cluster instances of app-operator.

    cluster-operator 3.8.0

    Changed

    • Adjust helm chart to be used with config-controller.

    Fixed

    • Updated OperatorKit to v4.3.1 for Kubernetes 1.20 support;
    • Fix clusterIPRange value in configmap;
    • Fix kubeconfig resource to search secrets in all namespaces.

    aws-operator 10.6.1

    Added

    • S3 vpc endpoint to AWS CNI subnet;
    • Enabled EBS CSI migration.

    Changed

    • Upgrade k8scloudconfig to v10.8.1 which includes a change to better determine if memory eviction thresholds are crossed;
    • Update Flatcar AMI’s to the latest stable releases;
    • Enabled EBS CSI migration;
    • Avoid TCCPN stack failure by checking if a control-plane tag exists before adding it;
    • Look up cloud tags in all namespaces;
    • Find certs in all namespaces;
    • Enable terminate unhealthy node feature by default;
    • Add node termination counter per cluster metric.

    Removed

    • Removed default storage-class annotation, EBS CSI driver is taking over.

    containerlinux 2765.2.6

    Security fixes

    Bug fixes

    • Update-engine sent empty requests when restarted before a pending reboot (Flatcar#388) motd login prompt list of failed services: The output of “systemctl list-units –state=failed –no-legend” contains a bullet point which is not expected and ended up being taken as the unit name of failed units which was previously on the start of the line. Filtered the bullet point out to stay compatible with the old behavior in case upstream would remove the bullet point again. (coreos-overlay#1042)
    • The Linux kernel IOMMU-related crash introduced in the 5.10.37 update got fixed through the 5.10.38 update (Flatcar#400)
    • Fix the patch to update DefaultTasksMax in systemd (coreos-overlay#971)
    • GCE: The old interface name ens4v1 which was replaced by eth0 due to a broken udev rule was restored, but now as alternative interface name, and eth0 will stay the primary name for consistency across cloud environments. (init#38)

    Changes

    • The virtio network interfaces got predictable interface names as alternative interface names, and thus these names can also be used to match for a specific interface in case there is more than one and the eth0 and eth1 name assignment is not stable. (init#38)

    Updates

    kubernetes 1.20.8

    What’s New (Major Themes)

    Dockershim deprecation

    Docker as an underlying runtime is being deprecated. Docker-produced images will continue to work in your cluster with all runtimes, as they always have. The Kubernetes community has written a blog post about this in detail with a dedicated FAQ page for it.

    External credential provider for client-go

    The client-go credential plugins can now be passed in the current cluster information via the KUBERNETES_EXEC_INFO environment variable. Learn more about this on client-go credential plugins documentation.

    CronJob controller v2 is available through feature gate

    An alternative implementation of the CronJob controller is now available as an alpha feature in this release, which has experimental performance improvement by using informers instead of polling. While this will be the default behavior in the future, you can try them in this release through a feature gate.

    PID Limits graduates to General Availability

    PID Limits features are now generally available on both SupportNodePidsLimit (node-to-pod PID isolation) and SupportPodPidsLimit (ability to limit PIDs per pod), after being enabled-by-default in beta stage for a year.

    API Priority and Fairness graduates to Beta

    Initially introduced in 1.18, Kubernetes 1.20 now enables API Priority and Fairness (APF) by default. This allows kube-apiserver to categorize incoming requests by priority levels.

    IPv4/IPv6 run

    IPv4/IPv6 dual-stack has been reimplemented for 1.20 to support dual-stack Services, based on user and community feedback. If your cluster has dual-stack enabled, you can create Services which can use IPv4, IPv6, or both, and you can change this setting for existing Services. Details are available in updated IPv4/IPv6 dual-stack docs, which cover the nuanced array of options.

    We expect this implementation to progress from alpha to beta and GA in coming releases, so we’re eager to have you comment about your dual-stack experiences in #k8s-dual-stack or in enhancements #563.

    go1.15.5

    go1.15.5 has been integrated into the Kubernetes project as of this release, including other infrastructure related updates on this effort.

    CSI Volume Snapshot graduates to General Availability

    CSI Volume Snapshot moves to GA in the 1.20 release. This feature provides a standard way to trigger volume snapshot operations in Kubernetes and allows Kubernetes users to incorporate snapshot operations in a portable manner on any Kubernetes environment regardless of supporting underlying storage providers. Additionally, these Kubernetes snapshot primitives act as basic building blocks that unlock the ability to develop advanced, enterprise-grade, storage administration features for Kubernetes: including application or cluster level backup solutions. Note that snapshot support will require Kubernetes distributors to bundle the Snapshot controller, Snapshot CRDs, and validation webhook. In addition, a CSI driver supporting the snapshot functionality must also be deployed on the cluster.

    Non-recursive Volume Ownership (FSGroup) graduates to Beta

    By default, the fsgroup setting, if specified, recursively updates permissions for every file in a volume on every mount. This can make mount, and pod startup, very slow if the volume has many files. This setting enables a pod to specify a PodFSGroupChangePolicy that indicates that volume ownership and permissions will be changed only when permission and ownership of the root directory do not match with expected permissions on the volume.

    CSIDriver policy for FSGroup graduates to Beta

    The FSGroup’s CSIDriver Policy is now beta in 1.20. This allows CSIDrivers to explicitly indicate if they want Kubernetes to manage permissions and ownership for their volumes via fsgroup.

    Security Improvements for CSI Drivers (Alpha)

    In 1.20, we introduce a new alpha feature CSIServiceAccountToken. This feature allows CSI drivers to impersonate the pods that they mount the volumes for. This improves the security posture in the mounting process where the volumes are ACL’ed on the pods’ service account without handing out unnecessary permissions to the CSI drivers’ service account. This feature is especially important for secret-handling CSI drivers, such as the secrets-store-csi-driver. Since these tokens can be rotated and short-lived, this feature also provides a knob for CSI drivers to receive NodePublishVolume RPC calls periodically with the new token. This knob is also useful when volumes are short-lived, e.g. certificates.

    Introducing Graceful Node Shutdown (Alpha)

    The GracefulNodeShutdown feature is now in Alpha. This allows kubelet to be aware of node system shutdowns, enabling graceful termination of pods during a system shutdown. This feature can be enabled through feature gate.

    Runtime log sanitation

    Logs can now be configured to use runtime protection from leaking sensitive data. Details for this experimental feature is available in documentation.

    Pod resource metrics

    On-demand metrics calculation is now available through /metrics/resources. When enabled, the endpoint will report the requested resources and the desired limits of all running pods.

    Introducing RootCAConfigMap

    RootCAConfigMap graduates to Beta, separating from BoundServiceAccountTokenVolume. The kube-root-ca.crt ConfigMap is now available to every namespace, by default. It contains the Certificate Authority bundle for verify kube-apiserver connections.

    kubectl debug graduates to Beta

    kubectl alpha debug graduates from alpha to beta in 1.20, becoming kubectl debug. kubectl debug provides support for common debugging workflows directly from kubectl. Troubleshooting scenarios supported in this release of kubectl include: Troubleshoot workloads that crash on startup by creating a copy of the pod that uses a different container image or command. Troubleshoot distroless containers by adding a new container with debugging tools, either in a new copy of the pod or using an ephemeral container. (Ephemeral containers are an alpha feature that are not enabled by default.) Troubleshoot on a node by creating a container running in the host namespaces and with access to the host’s filesystem. Note that as a new builtin command, kubectl debug takes priority over any kubectl plugin named “debug”. You will need to rename the affected plugin. Invocations using kubectl alpha debug are now deprecated and will be removed in a subsequent release. Update your scripts to use kubectl debug instead of kubectl alpha debug! For more information about kubectl debug, see Debugging Running Pods on the Kubernetes website, kubectl help debug, or reach out to SIG CLI by visiting #sig-cli or commenting on enhancement #1441.

    Removing deprecated flags in kubeadm

    kubeadm applies a number of deprecations and removals of deprecated features in this release. More details are available in the Urgent Upgrade Notes and Kind / Deprecation sections.

    Pod Hostname as FQDN graduates to Beta

    Previously introduced in 1.19 behind a feature gate, SetHostnameAsFQDN is now enabled by default. More details on this behavior are available in documentation for DNS for Services and Pods

    TokenRequest / TokenRequestProjection graduates to General Availability

    Service account tokens bound to a pod is now a stable feature. The feature gates will be removed in 1.21 release. For more information, refer to notes below on the changelogs.

    RuntimeClass feature graduates to General Availability.

    The node.k8s.io API groups are promoted from v1beta1 to v1. v1beta1 is now deprecated and will be removed in a future release, please start using v1. (#95718, @SergeyKanzhelev) [SIG Apps, Auth, Node, Scheduling and Testing]

    Cloud Controller Manager now exclusively shipped by Cloud Provider

    Kubernetes will no longer ship an instance of the Cloud Controller Manager binary. Each Cloud Provider is expected to ship their own instance of this binary. Details for a Cloud Provider to create an instance of such a binary can be found here. Anyone with questions on building a Cloud Controller Manager should reach out to SIG Cloud Provider. Questions about the Cloud Controller Manager on a Managed Kubernetes solution should go to the relevant Cloud Provider. Questions about the Cloud Controller Manager on a non managed solution can be brought up with SIG Cloud Provider.

    Important Security Information

    This release contains changes that address the following vulnerabilities:

    CVE-2021-25735: Validating Admission Webhook does not observe some previous fields

    A security issue was discovered in kube-apiserver that could allow node updates to bypass a Validating Admission Webhook. You are only affected by this vulnerability if you run a Validating Admission Webhook for Nodes that denies admission based at least partially on the old state of the Node object.

    Note: This only impacts validating admission plugins that rely on old values in certain fields, and does not impact calls from kubelet that go through the built-in NodeRestriction admission plugin.

    Affected Versions:

    • kube-apiserver v1.20.0 - v1.20.5
    • kube-apiserver v1.19.0 - v1.19.9
    • kube-apiserver <= v1.18.17

    Fixed Versions:

    • kube-apiserver v1.21.0
    • kube-apiserver v1.20.6
    • kube-apiserver v1.19.10
    • kube-apiserver v1.18.18

    This vulnerability was reported by Rogerio Bastos & Ari Lima from RedHat

    CVSS Rating: Medium (6.5) CVSS:3.0/AV:N/AC:L/PR:H/UI:N/S:U/C:N/I:H/A:H

    Deprecation

    • Docker support in the kubelet is now deprecated and will be removed in a future release. The kubelet uses a module called “dockershim” which implements CRI support for Docker and it has seen maintenance issues in the Kubernetes community. We encourage you to evaluate moving to a container runtime that is a full-fledged implementation of CRI (v1alpha1 or v1 compliant) as they become available. (#94624, @dims) [SIG Node]
    • Kubeadm: deprecate self-hosting support. The experimental command “kubeadm alpha self-hosting” is now deprecated and will be removed in a future release. (#95125, @neolit123) [SIG Cluster Lifecycle]
    • Kubeadm: graduate the “kubeadm alpha certs” command to a parent command “kubeadm certs”. The command “kubeadm alpha certs” is deprecated and will be removed in a future release. Please migrate. (#94938, @yagonobre) [SIG Cluster Lifecycle]
    • Kubeadm: remove the deprecated “kubeadm alpha kubelet config enable-dynamic” command. To continue using the feature please defer to the guide for “Dynamic Kubelet Configuration” at k8s.io. This change also removes the parent command “kubeadm alpha kubelet” as there are no more sub-commands under it for the time being. (#94668, @neolit123) [SIG Cluster Lifecycle]
    • Kubeadm: remove the deprecated –kubelet-config flag for the command “kubeadm upgrade node” (#94869, @neolit123) [SIG Cluster Lifecycle]
    • Kubectl: deprecate –delete-local-data (#95076, @dougsland) [SIG CLI, Cloud Provider and Scalability]
    • Kubelet’s deprecated endpoint metrics/resource/v1alpha1 has been removed, please adopt metrics/resource. (#94272, @RainbowMango) [SIG Instrumentation and Node]
    • Removes deprecated scheduler metrics DeprecatedSchedulingDuration, DeprecatedSchedulingAlgorithmPredicateEvaluationSecondsDuration, DeprecatedSchedulingAlgorithmPriorityEvaluationSecondsDuration (#94884, @arghya88) [SIG Instrumentation and Scheduling]
    • Scheduler alpha metrics binding_duration_seconds and scheduling_algorithm_preemption_evaluation_seconds are deprecated, Both of those metrics are now covered as part of framework_extension_point_duration_seconds, the former as a PostFilter the latter and a Bind plugin. The plan is to remove both in 1.21 (#95001, @arghya88) [SIG Instrumentation and Scheduling]
    • Support controlplane as a valid EgressSelection type in the EgressSelectorConfiguration API. Master is deprecated and will be removed in v1.22. (#95235, @andrewsykim) [SIG API Machinery]
    • The v1alpha1 PodPreset API and admission plugin has been removed with no built-in replacement. Admission webhooks can be used to modify pods on creation. (#94090, @deads2k) [SIG API Machinery, Apps, CLI, Cloud Provider, Scalability and Testing]

    API Change

    • We have added a new Priority & Fairness rule that exempts all probes (/readyz, /healthz, /livez) to prevent restarting of “healthy” kube-apiserver instance(s) by kubelet. (#101112, @tkashem) [SIG API Machinery]
    • Fixes using server-side apply with APIService resources (#100714, @kevindelgado) [SIG API Machinery, Apps and Testing]
    • Regenerate protobuf code to fix CVE-2021-3121 (#100501, @joelsmith) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node and Storage]
    • Kubernetes is now built using go1.15.8 (#98962, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing]
    • TokenRequest and TokenRequestProjection features have been promoted to GA. This feature allows generating service account tokens that are not visible in Secret objects and are tied to the lifetime of a Pod object. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection for details on configuring and using this feature. The TokenRequest and TokenRequestProjection feature gates will be removed in v1.21.
      • kubeadm’s kube-apiserver Pod manifest now includes the following flags by default “–service-account-key-file”, “–service-account-signing-key-file”, “–service-account-issuer”. (#93258, @zshihang) [SIG API Machinery, Auth, Cluster Lifecycle, Storage and Testing]
    • A new nofuzz go build tag now disables gofuzz support. Release binaries enable this. (#92491, @BenTheElder) [SIG API Machinery]
    • Add WindowsContainerResources and Annotations to CRI-API UpdateContainerResourcesRequest (#95741, @katiewasnothere) [SIG Node]
    • Add a serving and terminating condition to the EndpointSlice API. serving tracks the readiness of endpoints regardless of their terminating state. This is distinct from ready since ready is only true when pods are not terminating. terminating is true when an endpoint is terminating. For pods this is any endpoint with a deletion timestamp. (#92968, @andrewsykim) [SIG Apps and Network]
    • Add dual-stack Services (alpha). This is a BREAKING CHANGE to an alpha API. It changes the dual-stack API wrt Service from a single ipFamily field to 3 fields: ipFamilyPolicy (SingleStack, PreferDualStack, RequireDualStack), ipFamilies (a list of families assigned), and clusterIPs (inclusive of clusterIP). Most users do not need to set anything at all, defaulting will handle it for them. Services are single-stack unless the user asks for dual-stack. This is all gated by the “IPv6DualStack” feature gate. (#91824, @khenidak) [SIG API Machinery, Apps, CLI, Network, Node, Scheduling and Testing]
    • Add support for hugepages to downward API (#86102, @derekwaynecarr) [SIG API Machinery, Apps, CLI, Network, Node, Scheduling and Testing]
    • Adds kubelet alpha feature, GracefulNodeShutdown which makes kubelet aware of node system shutdowns and result in graceful termination of pods during a system shutdown. (#96129, @bobbypage) [SIG Node]
    • AppProtocol is now GA for Endpoints and Services. The ServiceAppProtocol feature gate will be deprecated in 1.21. (#96327, @robscott) [SIG Apps and Network]
    • Automatic allocation of NodePorts for services with type LoadBalancer can now be disabled by setting the (new) parameter Service.spec.allocateLoadBalancerNodePorts=false. The default is to allocate NodePorts for services with type LoadBalancer which is the existing behavior. (#92744, @uablrek) [SIG Apps and Network]
    • Certain fields on Service objects will be automatically cleared when changing the service’s type to a mode that does not need those fields. For example, changing from type=LoadBalancer to type=ClusterIP will clear the NodePort assignments, rather than forcing the user to clear them. (#95196, @thockin) [SIG API Machinery, Apps, Network and Testing]
    • Document that ServiceTopology feature is required to use service.spec.topologyKeys. (#96528, @andrewsykim) [SIG Apps]
    • EndpointSlice has a new NodeName field guarded by the EndpointSliceNodeName feature gate.
      • EndpointSlice topology field will be deprecated in an upcoming release.
      • EndpointSlice “IP” address type is formally removed after being deprecated in Kubernetes 1.17.
      • The discovery.k8s.io/v1alpha1 API is deprecated and will be removed in Kubernetes 1.21. (#96440, @robscott) [SIG API Machinery, Apps and Network]
    • External facing API podresources is now available under k8s.io/kubelet/pkg/apis/ (#92632, @RenaudWasTaken) [SIG Node and Testing]
    • Fewer candidates are enumerated for preemption to improve performance in large clusters. (#94814, @adtac)
    • Fix conversions for custom metrics. (#94481, @wojtek-t) [SIG API Machinery and Instrumentation]
    • GPU metrics provided by kubelet are now disabled by default. (#95184, @RenaudWasTaken)
    • If BoundServiceAccountTokenVolume is enabled, cluster admins can use metric serviceaccount_stale_tokens_total to monitor workloads that are depending on the extended tokens. If there are no such workloads, turn off extended tokens by starting kube-apiserver with flag --service-account-extend-token-expiration=false (#96273, @zshihang) [SIG API Machinery and Auth]
    • Introduce alpha support for exec-based container registry credential provider plugins in the kubelet. (#94196, @andrewsykim) [SIG Node and Release]
    • Introduces a metric source for HPAs which allows scaling based on container resource usage. (#90691, @arjunrn) [SIG API Machinery, Apps, Autoscaling and CLI]
    • Kube-apiserver now deletes expired kube-apiserver Lease objects:
      • The feature is under feature gate APIServerIdentity.
      • A flag is added to kube-apiserver: identity-lease-garbage-collection-check-period-seconds (#95895, @roycaihw) [SIG API Machinery, Apps, Auth and Testing]
    • Kube-controller-manager: volume plugins can be restricted from contacting local and loopback addresses by setting --volume-host-allow-local-loopback=false, or from contacting specific CIDR ranges by setting --volume-host-cidr-denylist (for example, --volume-host-cidr-denylist=127.0.0.1/28,feed::/16) (#91785, @mattcary) [SIG API Machinery, Apps, Auth, CLI, Network, Node, Storage and Testing]
    • Migrate scheduler, controller-manager and cloud-controller-manager to use LeaseLock (#94603, @wojtek-t) [SIG API Machinery, Apps, Cloud Provider and Scheduling]
    • Modify DNS-1123 error messages to indicate that RFC 1123 is not followed exactly (#94182, @mattfenwick) [SIG API Machinery, Apps, Auth, Network and Node]
    • Move configurable fsgroup change policy for pods to beta (#96376, @gnufied) [SIG Apps and Storage]
    • New flag is introduced, i.e. –topology-manager-scope=container|pod. The default value is the “container” scope. (#92967, @cezaryzukowski) [SIG Instrumentation, Node and Testing]
    • New parameter defaultingType for PodTopologySpread plugin allows to use k8s defined or user provided default constraints (#95048, @alculquicondor) [SIG Scheduling]
    • NodeAffinity plugin can be configured with AddedAffinity. (#96202, @alculquicondor) [SIG Node, Scheduling and Testing]
    • Promote RuntimeClass feature to GA. Promote node.k8s.io API groups from v1beta1 to v1. (#95718, @SergeyKanzhelev) [SIG Apps, Auth, Node, Scheduling and Testing]
    • Reminder: The labels “failure-domain.beta.kubernetes.io/zone” and “failure-domain.beta.kubernetes.io/region” are deprecated in favor of “topology.kubernetes.io/zone” and “topology.kubernetes.io/region” respectively. All users of the “failure-domain.beta…” labels should switch to the “topology…” equivalents. (#96033, @thockin) [SIG API Machinery, Apps, CLI, Cloud Provider, Network, Node, Scheduling, Storage and Testing]
    • Server Side Apply now treats LabelSelector fields as atomic (meaning the entire selector is managed by a single writer and updated together), since they contain interrelated and inseparable fields that do not merge in intuitive ways. (#93901, @jpbetz) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Storage and Testing]
    • Services will now have a clusterIPs field to go with clusterIP. clusterIPs[0] is a synonym for clusterIP and will be synchronized on create and update operations. (#95894, @thockin) [SIG Network]
    • The ServiceAccountIssuerDiscovery feature gate is now Beta and enabled by default. (#91921, @mtaufen) [SIG Auth]
    • The status of v1beta1 CRDs without “preserveUnknownFields:false” now shows a violation, “spec.preserveUnknownFields: Invalid value: true: must be false”. (#93078, @vareti)
    • The usage of mixed protocol values in the same LoadBalancer Service is possible if the new feature gate MixedProtocolLBService is enabled. The feature gate is disabled by default. The user has to enable it for the API Server. (#94028, @janosi) [SIG API Machinery and Apps]
    • This PR will introduce a feature gate CSIServiceAccountToken with two additional fields in CSIDriverSpec. (#93130, @zshihang) [SIG API Machinery, Apps, Auth, CLI, Network, Node, Storage and Testing]
    • Users can try the CronJob controller v2 using the feature gate. This will be the default controller in future releases. (#93370, @alaypatel07) [SIG API Machinery, Apps, Auth and Testing]
    • VolumeSnapshotDataSource moves to GA in 1.20 release (#95282, @xing-yang) [SIG Apps]
    • WinOverlay feature graduated to beta (#94807, @ksubrmnn) [SIG Windows]

    Feature

    • Kubernetes is now built using Go 1.15.13 (#102786, @thejoycekung) [SIG Cloud Provider, Instrumentation, Release and Testing]

    • Kubernetes is now built using go1.15.11 (#101192, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing]

    • Kubernetes is now built using go1.15.12 (#101845, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing]

    • AWS cloudprovider supports auto-discovering subnets without any kubernetes.io/cluster/ tags. It also supports additional service annotation service.beta.kubernetes.io/aws-load-balancer-subnets to manually configure the subnets. (#97431, @kishorj) [SIG Cloud Provider]

    • Kubernetes is now built using go1.15.10 (#100375, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing]

    • A new metric apiserver_request_filter_duration_seconds has been introduced that measures request filter latency in seconds. (#95207, @tkashem) [SIG API Machinery and Instrumentation]

    • A new set of alpha metrics are reported by the Kubernetes scheduler under the /metrics/resources endpoint that allow administrators to easily see the resource consumption (requests and limits for all resources on the pods) and compare it to actual pod usage or node capacity. (#94866, @smarterclayton) [SIG API Machinery, Instrumentation, Node and Scheduling]

    • Add –experimental-logging-sanitization flag enabling runtime protection from leaking sensitive data in logs (#96370, @serathius) [SIG API Machinery, Cluster Lifecycle and Instrumentation]

    • Add a StorageVersionAPI feature gate that makes API server update storageversions before serving certain write requests. This feature allows the storage migrator to manage storage migration for built-in resources. Enabling internal.apiserver.k8s.io/v1alpha1 API and APIServerIdentity feature gate are required to use this feature. (#93873, @roycaihw) [SIG API Machinery, Auth and Testing]

    • Add a metric for time taken to perform recursive permission change (#95866, @JornShen) [SIG Instrumentation and Storage]

    • Add a new vSphere metric: cloudprovider_vsphere_vcenter_versions. Its content shows vCenter hostnames with the associated server version. (#94526, @Danil-Grigorev) [SIG Cloud Provider and Instrumentation]

    • Add a new flag to set priority for the kubelet on Windows nodes so that workloads cannot overwhelm the node thereby disrupting kubelet process. (#96051, @ravisantoshgudimetla) [SIG Node and Windows]

    • Add feature to size memory backed volumes (#94444, @derekwaynecarr) [SIG Storage and Testing]

    • Add foreground cascading deletion to kubectl with the new kubectl delete foreground|background|orphan option. (#93384, @zhouya0)

    • Add metrics for azure service operations (route and loadbalancer). (#94124, @nilo19) [SIG Cloud Provider and Instrumentation]

    • Add network rule support in Azure account creation. (#94239, @andyzhangx)

    • Add node_authorizer_actions_duration_seconds metric that can be used to estimate load to node authorizer. (#92466, @mborsz) [SIG API Machinery, Auth and Instrumentation]

    • Add pod_ based CPU and memory metrics to Kubelet’s /metrics/resource endpoint (#95839, @egernst) [SIG Instrumentation, Node and Testing]

    • Added get-users and delete-user to the kubectl config subcommand (#89840, @eddiezane) [SIG CLI]

    • Added counter metric “apiserver_request_self” to count API server self-requests with labels for verb, resource, and subresource. (#94288, @LogicalShark) [SIG API Machinery, Auth, Instrumentation and Scheduling]

    • Added new k8s.io/component-helpers repository providing shared helper code for (core) components. (#92507, @ingvagabund) [SIG Apps, Node, Release and Scheduling]

    • Adds create ingress command to kubectl (#78153, @amimof) [SIG CLI and Network]

    • Adds a headless service on node-local-cache addon. (#88412, @stafot) [SIG Cloud Provider and Network]

    • Allow cross-compilation of kubernetes on different platforms. (#94403, @bnrjee) [SIG Release]

    • Azure: Support multiple services sharing one IP address (#94991, @nilo19) [SIG Cloud Provider]

    • CRDs: For structural schemas, non-nullable null map fields will now be dropped and defaulted if a default is available. null items in the list will continue being preserved, and fail validation if not nullable. (#95423, @apelisse) [SIG API Machinery]

    • Changed: default “Accept: /” header added to HTTP probes. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#http-probes (https://github.com/kubernetes/website/pull/24756) (#95641, @fonsecas72) [SIG Network and Node]

    • Client-go credential plugins can now be passed in the current cluster information via the KUBERNETES_EXEC_INFO environment variable. (#95489, @ankeesler) [SIG API Machinery and Auth]

    • Command to start network proxy changes from ‘KUBE_ENABLE_EGRESS_VIA_KONNECTIVITY_SERVICE ./cluster/kube-up.sh’ to ‘KUBE_ENABLE_KONNECTIVITY_SERVICE=true ./hack/kube-up.sh’ (#92669, @Jefftree) [SIG Cloud Provider]

    • Configure AWS LoadBalancer health check protocol via service annotations. (#94546, @kishorj)

    • DefaultPodTopologySpread graduated to Beta. The feature gate is enabled by default. (#95631, @alculquicondor) [SIG Scheduling and Testing]

    • E2e test for PodFsGroupChangePolicy (#96247, @saikat-royc) [SIG Storage and Testing]

    • Ephemeral containers now apply the same API defaults as initContainers and containers (#94896, @wawa0210) [SIG Apps and CLI]

    • Graduate the Pod Resources API to G.A Introduces the pod_resources_endpoint_requests_total metric which tracks the total number of requests to the pod resources API (#92165, @RenaudWasTaken) [SIG Instrumentation, Node and Testing]

    • In dual-stack bare-metal clusters, you can now pass dual-stack IPs to kubelet --node-ip. eg: kubelet --node-ip 10.1.0.5,fd01::0005. This is not yet supported for non-bare-metal clusters.

      In dual-stack clusters where nodes have dual-stack addresses, hostNetwork pods will now get dual-stack PodIPs. (#95239, @danwinship) [SIG Network and Node]

    • Introduce api-extensions category which will return: mutating admission configs, validating admission configs, CRDs and APIServices when used in kubectl get, for example. (#95603, @soltysh) [SIG API Machinery]

    • Introduces a new GCE specific cluster creation variable KUBE_PROXY_DISABLE. When set to true, this will skip over the creation of kube-proxy (whether the daemonset or static pod). This can be used to control the lifecycle of kube-proxy separately from the lifecycle of the nodes. (#91977, @varunmar) [SIG Cloud Provider]

    • Kube-apiserver now maintains a Lease object to identify itself:

      • The feature is under feature gate APIServerIdentity.
      • Two flags are added to kube-apiserver: identity-lease-duration-seconds, identity-lease-renew-interval-seconds (#95533, @roycaihw) [SIG API Machinery]
    • Kube-apiserver: The timeout used when making health check calls to etcd can now be configured with --etcd-healthcheck-timeout. The default timeout is 2 seconds, matching the previous behavior. (#93244, @Sh4d1) [SIG API Machinery]

    • Kube-apiserver: added support for compressing rotated audit log files with --audit-log-compress (#94066, @lojies) [SIG API Machinery and Auth]

    • Kubeadm now prints warnings instead of throwing errors if the current system time is outside of the NotBefore and NotAfter bounds of a loaded certificate. (#94504, @neolit123)

    • Kubeadm: Add a preflight check that the control-plane node has at least 1700MB of RAM (#93275, @xlgao-zju) [SIG Cluster Lifecycle]

    • Kubeadm: add the “–cluster-name” flag to the “kubeadm alpha kubeconfig user” to allow configuring the cluster name in the generated kubeconfig file (#93992, @prabhu43) [SIG Cluster Lifecycle]

    • Kubeadm: add the “–kubeconfig” flag to the “kubeadm init phase upload-certs” command to allow users to pass a custom location for a kubeconfig file. (#94765, @zhanw15) [SIG Cluster Lifecycle]

    • Kubeadm: make etcd pod request 100m CPU, 100Mi memory and 100Mi ephemeral_storage by default (#94479, @knight42) [SIG Cluster Lifecycle]

    • Kubeadm: make the command “kubeadm alpha kubeconfig user” accept a “–config” flag and remove the following flags:

      • apiserver-advertise-address / apiserver-bind-port: use either localAPIEndpoint from InitConfiguration or controlPlaneEndpoint from ClusterConfiguration.
      • cluster-name: use clusterName from ClusterConfiguration
      • cert-dir: use certificatesDir from ClusterConfiguration (#94879, @knight42) [SIG Cluster Lifecycle]
    • Kubectl create now supports creating ingress objects. (#94327, @rikatz) [SIG CLI and Network]

    • Kubectl rollout history sts/sts-name –revision=some-revision will start showing the detailed view of the sts on that specified revision (#86506, @dineshba) [SIG CLI]

    • Kubectl: Previously users cannot provide arguments to a external diff tool via KUBECTL_EXTERNAL_DIFF env. This release now allow users to specify args to KUBECTL_EXTERNAL_DIFF env. (#95292, @dougsland) [SIG CLI]

    • Kubemark now supports both real and hollow nodes in a single cluster. (#93201, @ellistarn) [SIG Scalability]

    • Kubernetes E2E test image manifest lists now contain Windows images. (#77398, @claudiubelu) [SIG Testing and Windows]

    • Kubernetes is now built using go1.15.2

      • build: Update to k/repo-infra@v0.1.1 (supports go1.15.2)

      • build: Use go-runner:buster-v2.0.1 (built using go1.15.1)

      • bazel: Replace –features with Starlark build settings flag

      • hack/lib/util.sh: some bash cleanups

        • switched one spot to use kube::logging
        • make kube::util::find-binary return an error when it doesn’t find anything so that hack scripts fail fast instead of with “binary not found errors”.
        • this required deleting some genfeddoc stuff. the binary no longer exists in k/k repo since we removed federation/, and I don’t see it in https://github.com/kubernetes-sigs/kubefed/ either. I’m assuming that it’s gone for good now.
      • bazel: output go_binary rule directly from go_binary_conditional_pure

        From: @mikedanese: Instead of aliasing. Aliases are annoying in a number of ways. This is specifically bugging me now because they make the action graph harder to analyze programmatically. By using aliases here, we would need to handle potentially aliased go_binary targets and dereference to the effective target.

        The comment references an issue with pure = select(...) which appears to be resolved considering this now builds.

      • make kube::util::find-binary not dependent on bazel-out/ structure

        Implement an aspect that outputs go_build_mode metadata for go binaries, and use that during binary selection. (#94449, @justaugustus) [SIG Architecture, CLI, Cluster Lifecycle, Node, Release and Testing]

    • Kubernetes is now built using go1.15.5

    • New default scheduling plugins order reduces scheduling and preemption latency when taints and node affinity are used (#95539, @soulxu) [SIG Scheduling]

    • Only update Azure data disks when attach/detach (#94265, @andyzhangx) [SIG Cloud Provider]

    • Promote SupportNodePidsLimit to GA to provide node-to-pod PID isolation. Promote SupportPodPidsLimit to GA to provide the ability to limit PIDs per pod. (#94140, @derekwaynecarr)

    • SCTP support in API objects (Pod, Service, NetworkPolicy) is now GA. Note that this has no effect on whether SCTP is enabled on nodes at the kernel level, and note that some cloud platforms and network plugins do not support SCTP traffic. (#95566, @danwinship) [SIG Apps and Network]

    • Scheduler now ignores Pod update events if the resourceVersion of old and new Pods are identical. (#96071, @Huang-Wei) [SIG Scheduling]

    • Scheduling Framework: expose Run[Pre]ScorePlugins functions to PreemptionHandle which can be used in PostFilter extension point. (#93534, @everpeace) [SIG Scheduling and Testing]

    • SelectorSpreadPriority maps to PodTopologySpread plugin when DefaultPodTopologySpread feature is enabled (#95448, @alculquicondor) [SIG Scheduling]

    • Send GCE node startup scripts' logs to console and journal. (#95311, @karan)

    • SetHostnameAsFQDN has been graduated to Beta and therefore it is enabled by default. (#95267, @javidiaz) [SIG Node]

    • Support [service.beta.kubernetes.io/azure-pip-ip-tags] annotations to allow customers to specify ip-tags to influence public-ip creation in Azure [Tag1=Value1, Tag2=Value2, etc.] (#94114, @MarcPow) [SIG Cloud Provider]

    • Support custom tags for cloud provider managed resources (#96450, @nilo19) [SIG Cloud Provider]

    • Support customize load balancer health probe protocol and request path (#96338, @nilo19) [SIG Cloud Provider]

    • Support for Windows container images (OS Versions: 1809, 1903, 1909, 2004) was added to the pause:3.4 image. (#91452, @claudiubelu) [SIG Node, Release and Windows]

    • Support multiple standard load balancers in one cluster (#96111, @nilo19) [SIG Cloud Provider]

    • The beta RootCAConfigMap feature gate is enabled by default and causes kube-controller-manager to publish a “kube-root-ca.crt” ConfigMap to every namespace. This ConfigMap contains a CA bundle used for verifying connections to the kube-apiserver. (#96197, @zshihang) [SIG API Machinery, Apps, Auth and Testing]

    • The kubelet_runtime_operations_duration_seconds metric buckets were set to 0.005 0.0125 0.03125 0.078125 0.1953125 0.48828125 1.220703125 3.0517578125 7.62939453125 19.073486328125 47.6837158203125 119.20928955078125 298.0232238769531 and 745.0580596923828 seconds (#96054, @alvaroaleman) [SIG Instrumentation and Node]

    • There is a new pv_collector_total_pv_count metric that counts persistent volumes by the volume plugin name and volume mode. (#95719, @tsmetana) [SIG Apps, Instrumentation, Storage and Testing]

    • Volume snapshot e2e test to validate PVC and VolumeSnapshotContent finalizer (#95863, @RaunakShah) [SIG Cloud Provider, Storage and Testing]

    • Warns user when executing kubectl apply/diff to a resource currently being deleted. (#95544, @SaiHarshaK) [SIG CLI]

    • kubectl alpha debug has graduated to beta and is now kubectl debug. (#96138, @verb) [SIG CLI and Testing]

    • kubectl debug gains support for changing container images when copying a pod for debugging, similar to how kubectl set image works. See kubectl help debug for more information. (#96058, @verb) [SIG CLI]

    Documentation

    • Fake dynamic client: document that List does not preserve TypeMeta in UnstructuredList (#95117, @andrewsykim) [SIG API Machinery]
    • Kubelet: remove alpha warnings for CNI flags. (#94508, @andrewsykim) [SIG Network and Node]
    • Updates docs and guidance on cloud provider InstancesV2 and Zones interface for external cloud providers:
      • removes experimental warning for InstancesV2
      • document that implementation of InstancesV2 will disable calls to Zones
      • deprecate Zones in favor of InstancesV2 (#96397, @andrewsykim) [SIG Cloud Provider]

    Failing Test

    • Fixes the should receive events on concurrent watches in same order conformance test to work properly on clusters that auto-create additional configmaps in namespaces (#101950, @liggitt) [SIG API Machinery and Testing]
    • Fix handing special characters in the volume path on Windows (#99008, @yujuhong) [SIG Storage]
    • Kube-proxy: fix a bug on UDP NodePort Services where stale conntrack entries may blackhole the traffic directed to the NodePort. (#98305, @aojea) [SIG Network]
    • Kubelet: the HostPort implementation in dockershim was not taking into consideration the HostIP field, causing that the same HostPort can not be used with different IP addresses. This bug causes the conformance test “HostPort validates that there is no conflict between pods with same hostPort but different hostIP and protocol” to fail. (#98838, @aojea) [SIG Network and Node]
    • Resolves an issue running Ingress conformance tests on clusters which use finalizers on Ingress objects to manage releasing load balancer resources (#96742, @spencerhance) [SIG Network and Testing]
    • The Conformance test “validates that there is no conflict between pods with same hostPort but different hostIP and protocol” now validates the connectivity to each hostPort, in addition to the functionality. (#96627, @aojea) [SIG Scheduling and Testing]

    Bug or Regression

    • Added jitter factor to lease controller that better smears load on kube-apiserver over time. (#101652, @marseel) [SIG API Machinery and Scalability]

    • Avoid caching the Azure VMSS instances whose network profile is nil (#100948, @feiskyer) [SIG Cloud Provider]

    • Azure: avoid setting cached Sku when updating VMSS and VMSS instances (#102005, @feiskyer) [SIG Cloud Provider]

    • Fix a bug on the endpoint slices mirroring controller where endpoint NotReadyAddresses were mirrored as Ready to the corresponding EndpointSlice (#102683, @aojea) [SIG Apps and Network]

    • Fix a bug that a preemptor pod may exist as a phantom in the scheduler. (#102498, @Huang-Wei) [SIG Scheduling]

    • Fix errors when accessing Windows container stats for Dockershim (#98510, @jsturtevant) [SIG Node and Windows]

    • Fix removing pods from podTopologyHints mapping (#101896, @aheng-ch) [SIG Node]

    • Fix: avoid nil-pointer panic when checking the frontend IP configuration (#101739, @nilo19) [SIG Cloud Provider]

    • Fix: delete non existing disk issue (#102083, @andyzhangx) [SIG Cloud Provider]

    • Fixed false-positive uncertain volume attachments, which led to unexpected detachment of CSI migrated volumes (#101737, @Jiawei0227) [SIG Apps and Storage]

    • Fixed garbage collection of dangling VolumeAttachments for PersistentVolumes migrated to CSI on startup of kube-controller-manager. (#102176, @timebertt) [SIG Apps and Storage]

    • Improve speed of vSphere PV provisioning and reduce number of API calls (#102350, @gnufied) [SIG Cloud Provider and Storage]

    • Kubeadm: remove the “ephemeral_storage” request from the etcd static pod that kubeadm deploys on stacked etcd control plane nodes. This request has caused sporadic failures on some setups due to a problem in the kubelet with cadvisor and the LocalStorageCapacityIsolation feature gate. See this issue for more details: https://github.com/kubernetes/kubernetes/issues/99305 (#102673, @jackfrancis) [SIG Cluster Lifecycle]

    • Register/Deregister Targets in chunks for AWS TargetGroup (#101592, @M00nF1sh) [SIG Cloud Provider]

    • Respect annotation size limit for server-side apply updates to the client-side apply annotation. Also, fix opt-out of this behavior by setting the client-side apply annotation to the empty string. (#102105, @julianvmodesto) [SIG API Machinery]

    • Reverted the previous fix for portforward cleanup because it introduced a kubelet regression which can lead into segmentation faults. (#102586, @saschagrunert) [SIG API Machinery and Node]

    • ServiceOwnsFrontendIP shouldn’t report error when the public IP doesn’t match (#102516, @nilo19) [SIG Cloud Provider]

    • Azurefile: Normalize share name to not include capital letters (#100731, @kassarl) [SIG Cloud Provider and Storage]

    • EndpointSlice IP validation now matches Endpoints IP validation. (#101084, @robscott) [SIG Apps and Network]

    • EndpointSlice controllers are less likely to create duplicate EndpointSlices. (#101763, @aojea) [SIG Apps and Network]

    • Ensure service deleted when the Azure resource group has been deleted (#100944, @feiskyer) [SIG Cloud Provider]

    • Fix panic in JSON logging format caused by missing Duration encoder (#101158, @serathius) [SIG API Machinery, Cluster Lifecycle and Instrumentation]

    • Fix smb mount PermissionDenied issue on Windows (#99550, @andyzhangx) [SIG Cloud Provider, Storage and Windows]

    • Fix: azure file inline volume namespace issue in csi migration translation (#101235, @andyzhangx) [SIG Apps, Cloud Provider, Node and Storage]

    • Fix: not tagging static public IP (#101752, @nilo19) [SIG Cloud Provider]

    • Fix: set “host is down” as corrupted mount (#101398, @andyzhangx) [SIG Cloud Provider and Storage]

    • Fixed a bug where startupProbe stopped working after a container’s first restart (#101093, @wzshiming) [SIG Node]

    • Fixed port-forward memory leak for long-running and heavily used connections. (#99839, @saschagrunert) [SIG API Machinery and Node]

    • Kubectl create service now respects namespace flag (#101005, @zxh326) [SIG CLI]

    • Kubelet: improve the performance when waiting for a synchronization of the node list with the kube-apiserver (#99336, @neolit123) [SIG Node]

    • No support endpointslice in linux userpace mode (#101503, @JornShen) [SIG Network]

    • Renames the timeout field for the DelegatingAuthenticationOptions to TokenRequestTimeout and set the timeout only for the token review client. Previously the timeout was also applied to watches making them reconnecting every 10 seconds. (#101103, @p0lyn0mial) [SIG API Machinery, Auth and Cloud Provider]

    • Respect ExecProbeTimeout=false for dockershim (#101126, @jackfrancis) [SIG Node and Testing]

    • Fix priority expander falling back to a random choice even though there is a higher priority option to choose

    • Clone kubernetes/kubernetes in update-vendor.sh shallowly, instead of fetching all revisions

    • Speed up binpacking by reducing the number of PreFilter calls (call once per pod instead of pods * nodes times)

    • Speed up finding unneeded nodes by 5x+ in very large clusters by reducing the number of PreFilter calls

    • Expose --max-nodes-total as a metric

    • Errors in IncreaseSize changed from type apiError to cloudProviderError

    • Make build-in-docker and test-in-docker work on Linux systems with SELinux enabled

    • Fix an error where existing nodes were not considered as destinations while finding place for pods in scale-down simulations

    • Remove redundant log lines and reduce severity around parsing kubeEnv

    • Don’t treat nodes created by virtual kubelet as nodes from non-autoscaled node groups

    • Remove redundant logging around calculating node utilization

    • Add configurable --network and --rm flags for docker in Makefile

    • Subtract DaemonSet pods' requests from node allocatable in the denominator while computing node utilization

    • Include taints by condition when determining if a node is unready/still starting

    • Fix update-vendor.sh to work on OSX and zsh

    • Add best-effort eviction for DaemonSet pods while scaling down non-empty nodes

    • Add build support for ARM64

    • Regenerate list of EC2 instances

    • Fix pricing endpoint in AWS China Region

    • Avoid systemd-logind loading configuration warning (#97950, @wzshiming) [SIG Node]

    • Count pod overhead against an entity’s ResourceQuota (#99600, @gjkim42) [SIG API Machinery and Node]

    • EndpointSlice controller is now less likely to emit FailedToUpdateEndpointSlices events. (#100113, @robscott) [SIG Apps and Network]

    • EndpointSliceMirroring controller is now less likely to emit FailedToUpdateEndpointSlices events. (#100143, @robscott) [SIG Apps and Network]

    • Ensure only one LoadBalancer rule is created when HA mode is enabled (#99825, @feiskyer) [SIG Cloud Provider]

    • Fix kubelet from panic after getting the wrong signal (#98200, @wzshiming) [SIG Node]

    • Fix repeatedly acquire the inhibit lock (#98088, @wzshiming) [SIG Node]

    • Fixed bug that caused cAdvisor to incorrectly detect single-socket multi-NUMA topology. (#99207, @iwankgb) [SIG Node]

    • Fixing a bug where a failed node may not have the NoExecute taint set correctly (#98168, @CKchen0726) [SIG Apps and Node]

    • Kubelet now cleans up orphaned volume directories automatically (#95301, @lorenz) [SIG Node and Storage]

    • Resolves spurious Failed to list *v1.Secret or Failed to list *v1.ConfigMap messages in kubelet logs. (#99538, @liggitt) [SIG Auth and Node]

    • Sync node status during kubelet node shutdown. Adds an pod admission handler that rejects new pods when the node is in progress of shutting down. (#98005, @wzshiming) [SIG Node]

    • We will no longer automatically delete all data when a failure is detected during creation of the volume data file on a CSI volume. Now we will only remove the data file and volume path. (#96021, @huffmanca) [SIG Storage]

    • Aggregate errors when putting vmss (#98350, @nilo19) [SIG Cloud Provider]

    • Avoid marking node as Ready until node has synced with API servers at least once (#97995, @ehashman) [SIG Node]

    • Cleanup subnet in frontend IP configs to prevent huge subnet request bodies in some scenarios. (#98132, @nilo19) [SIG Cloud Provider]

    • Fix CSI-migrated inline EBS volumes failing to mount if their volumeID is prefixed by aws:// (#96821, @wongma7) [SIG Storage]

    • Fix azure file migration issue (#97877, @andyzhangx) [SIG Auth, Cloud Provider and Storage]

    • Fix kubectl-convert import known versions (#97754, @wzshiming) [SIG CLI and Testing]

    • Fix the description of command line flags that can override –config (#98786, @changshuchao) [SIG Scheduling]

    • Fix the panic when kubelet registers if a node object already exists with no Status.Capacity or Status.Allocatable (#97803, @TeddyAndrieux) [SIG Node]

    • Fix the regression with the slow pods termination. Before this fix pods may take an additional time to terminate - up to one minute. Reversing the change that ensured that CNI resources cleaned up when the pod is removed on API server. (#97980, @SergeyKanzhelev) [SIG Node]

    • Fix to recover CSI volumes from certain dangling attachments (#96617, @yuga711) [SIG Apps and Storage]

    • Fixed a bug that the kubelet cannot start on BtrfS. (#98014, @gjkim42) [SIG Node]

    • Fixed an issue with garbage collection failing to clean up namespaced children of an object also referenced incorrectly by cluster-scoped children (#98068, @liggitt) [SIG API Machinery and Apps]

    • Fixed provisioning of Cinder volumes migrated to CSI when StorageClass with AllowedTopologies was used. (#98311, @jsafrane) [SIG Storage]

    • Fixes a panic in the disruption budget controller for PDB objects with invalid selectors (#98775, @ialidzhikov) [SIG Apps]

    • Fixes connection errors when using --volume-host-cidr-denylist or --volume-host-allow-local-loopback (#98436, @liggitt) [SIG Network and Storage]

    • Kubeadm: get k8s CI version markers from k8s infra bucket (#98836, @hasheddan) [SIG Cluster Lifecycle and Release]

    • Kubelet should ignore cgroup driver check on Windows node. (#98383, @pacoxu) [SIG Node]

    • Make podTopologyHints protected by lock (#95111, @choury) [SIG Node]

    • Static pods will be deleted gracefully. (#98103, @gjkim42) [SIG Node]

    • Truncates a message if it hits the NoteLengthLimit when the scheduler records an event for the pod that indicates the pod has failed to schedule. (#98715, @carlory) [SIG Scheduling]

    • Warning about using a deprecated volume plugin is logged only once. (#96751, @jsafrane) [SIG Storage]

    • Fix Azure file share not deleted issue when the namespace is deleted (#97417, @andyzhangx) [SIG Cloud Provider and Storage]

    • Fix counting error in service/nodeport/loadbalancer quota check (#97826, @pacoxu) [SIG API Machinery and Network]

    • Fix missing cadvisor machine metrics. (#97006, @lingsamuel) [SIG Node]

    • Fix: azure file latency issue for metadata-heavy workloads (#97082, @andyzhangx) [SIG Cloud Provider and Storage]

    • Fixed bug in CPUManager with race on container map access (#97427, @klueska) [SIG Node]

    • GCE Internal LoadBalancer sync loop will now release the ILB IP address upon sync failure. An error in ILB forwarding rule creation will no longer leak IP addresses. (#97740, @prameshj) [SIG Cloud Provider and Network]

    • Kubeadm: avoid detection of the container runtime for commands that do not need it (#97847, @pacoxu) [SIG Cluster Lifecycle]

    • Performance regression #97685 has been fixed. (#97860, @MikeSpreitzer) [SIG API Machinery]

    • Use network.Interface.VirtualMachine.ID to get the binded VM Skip standalone VM when reconciling LoadBalancer (#97639, @nilo19) [SIG Cloud Provider]

    • AcceleratorStats will be available in the Summary API of kubelet when cri_stats_provider is used. (#97018, @ruiwen-zhao) [SIG Node]

    • Fixed FibreChannel volume plugin corrupting filesystems on detach of multipath volumes. (#97013, @jsafrane) [SIG Storage]

    • Fixed a bug in kubelet that will saturate CPU utilization after containerd got restarted. (#97175, @hanlins) [SIG Node]

    • Kubeadm now installs version 3.4.13 of etcd when creating a cluster with v1.19 (#97284, @pacoxu) [SIG Cluster Lifecycle]

    • Kubeadm: Fixes a kubeadm upgrade bug that could cause a custom CoreDNS configuration to be replaced with the default. (#97016, @rajansandeep) [SIG Cluster Lifecycle]

    • Add kubectl wait –ignore-not-found flag (#90969, @zhouya0) [SIG CLI]

    • Added support to kube-proxy for externalTrafficPolicy=Local setting via Direct Server Return (DSR) load balancers on Windows. (#93166, @elweb9858) [SIG Network]

    • Alter wording to describe pods using a pvc (#95635, @RaunakShah) [SIG CLI]

    • An issues preventing volume expand controller to annotate the PVC with volume.kubernetes.io/storage-resizer when the PVC StorageClass is already updated to the out-of-tree provisioner is now fixed. (#94489, @ialidzhikov) [SIG API Machinery, Apps and Storage]

    • Azure ARM client: don’t segfault on empty response and http error (#94078, @bpineau) [SIG Cloud Provider]

    • Azure armclient backoff step defaults to 1 (no retry). (#94180, @feiskyer)

    • Azure: fix a bug that kube-controller-manager would panic if wrong Azure VMSS name is configured (#94306, @knight42) [SIG Cloud Provider]

    • Both apiserver_request_duration_seconds metrics and RequestReceivedTimestamp fields of an audit event now take into account the time a request spends in the apiserver request filters. (#94903, @tkashem)

    • Build/lib/release: Explicitly use ‘–platform’ in building server images

      When we switched to go-runner for building the apiserver, controller-manager, and scheduler server components, we no longer reference the individual architectures in the image names, specifically in the ‘FROM’ directive of the server image Dockerfiles.

      As a result, server images for non-amd64 images copy in the go-runner amd64 binary instead of the go-runner that matches that architecture.

      This commit explicitly sets the ‘–platform=linux/${arch}’ to ensure we’re pulling the correct go-runner arch from the manifest list.

      Before: FROM ${base_image}

      After: FROM --platform=linux/${arch} ${base_image} (#94552, @justaugustus) [SIG Release]

    • Bump node-problem-detector version to v0.8.5 to fix OOM detection in with Linux kernels 5.1+ (#96716, @tosi3k) [SIG Cloud Provider, Scalability and Testing]

    • CSIDriver object can be deployed during volume attachment. (#93710, @Jiawei0227) [SIG Apps, Node, Storage and Testing]

    • Ceph RBD volume expansion now works even when ceph.conf was not provided. (#92027, @juliantaylor)

    • Change plugin name in fsgroupapplymetrics of csi and flexvolume to distinguish different driver (#95892, @JornShen) [SIG Instrumentation, Storage and Testing]

    • Change the calculation of pod UIDs so that static pods get a unique value - will cause all containers to be killed and recreated after in-place upgrade. (#87461, @bboreham) [SIG Node]

    • Change the mount way from systemd to normal mount except ceph and glusterfs intree-volume. (#94916, @smileusd) [SIG Apps, Cloud Provider, Network, Node, Storage and Testing]

    • Changes to timeout parameter handling in 1.20.0-beta.2 have been reverted to avoid breaking backwards compatibility with existing clients. (#96727, @liggitt) [SIG API Machinery and Testing]

    • Clear UDP conntrack entry on endpoint changes when using nodeport (#71573, @JacobTanenbaum) [SIG Network]

    • Cloud node controller: handle empty providerID from getProviderID (#95342, @nicolehanjing) [SIG Cloud Provider]

    • Disable watchcache for events (#96052, @wojtek-t) [SIG API Machinery]

    • Disabled LocalStorageCapacityIsolation feature gate is honored during scheduling. (#96092, @Huang-Wei) [SIG Scheduling]

    • Do not fail sorting empty elements. (#94666, @soltysh) [SIG CLI]

    • Dual-stack: make nodeipam compatible with existing single-stack clusters when dual-stack feature gate become enabled by default (#90439, @SataQiu) [SIG API Machinery]

    • Duplicate owner reference entries in create/update/patch requests now get deduplicated by the API server. The client sending the request now receives a warning header in the API response. Clients should stop sending requests with duplicate owner references. The API server may reject such requests as early as 1.24. (#96185, @roycaihw) [SIG API Machinery and Testing]

    • Endpoint slice controller now mirrors parent’s service label to its corresponding endpoint slices. (#94443, @aojea)

    • Ensure getPrimaryInterfaceID not panic when network interfaces for Azure VMSS are null (#94355, @feiskyer) [SIG Cloud Provider]

    • Exposes and sets a default timeout for the SubjectAccessReview client for DelegatingAuthorizationOptions (#95725, @p0lyn0mial) [SIG API Machinery and Cloud Provider]

    • Exposes and sets a default timeout for the TokenReview client for DelegatingAuthenticationOptions (#96217, @p0lyn0mial) [SIG API Machinery and Cloud Provider]

    • Fix CVE-2020-8555 for Quobyte client connections. (#95206, @misterikkit) [SIG Storage]

    • Fix IP fragmentation of UDP and TCP packets not supported issues on LoadBalancer rules (#96464, @nilo19) [SIG Cloud Provider]

    • Fix a bug that DefaultPreemption plugin is disabled when using (legacy) scheduler policy. (#96439, @Huang-Wei) [SIG Scheduling and Testing]

    • Fix a bug where loadbalancer deletion gets stuck because of missing resource group. (#93962, @phiphi282)

    • Fix a concurrent map writes error in kubelet (#93773, @knight42) [SIG Node]

    • Fix a panic in kubectl debug when a pod has multiple init or ephemeral containers. (#94580, @kiyoshim55)

    • Fix a regression where kubeadm bails out with a fatal error when an optional version command line argument is supplied to the “kubeadm upgrade plan” command (#94421, @rosti) [SIG Cluster Lifecycle]

    • Fix azure disk attach failure for disk size bigger than 4TB (#95463, @andyzhangx) [SIG Cloud Provider]

    • Fix azure disk data loss issue on Windows when unmount disk (#95456, @andyzhangx) [SIG Cloud Provider and Storage]

    • Fix azure file migration panic (#94853, @andyzhangx) [SIG Cloud Provider]

    • Fix bug in JSON path parser where an error occurs when a range is empty (#95933, @brianpursley) [SIG API Machinery]

    • Fix client-go prometheus metrics to correctly present the API path accessed in some environments. (#74363, @aanm) [SIG API Machinery]

    • Fix detach azure disk issue when vm not exist (#95177, @andyzhangx) [SIG Cloud Provider]

    • Fix etcd_object_counts metric reported by kube-apiserver (#94773, @tkashem) [SIG API Machinery]

    • Fix incorrectly reported verbs for kube-apiserver metrics for CRD objects (#93523, @wojtek-t) [SIG API Machinery and Instrumentation]

    • Fix k8s.io/apimachinery/pkg/api/meta.SetStatusCondition to update ObservedGeneration (#95961, @KnicKnic) [SIG API Machinery]

    • Fix kubectl SchemaError on CRDs with schema using x-kubernetes-preserve-unknown-fields on array types. (#94888, @sttts) [SIG API Machinery]

    • Fix memory leak in kube-apiserver when underlying time goes forth and back. (#96266, @chenyw1990) [SIG API Machinery]

    • Fix missing csi annotations on node during parallel csinode update. (#94389, @pacoxu) [SIG Storage]

    • Fix network_programming_latency metric reporting for Endpoints/EndpointSlice deletions, where we don’t have correct timestamp (#95363, @wojtek-t) [SIG Network and Scalability]

    • Fix paging issues when Azure API returns empty values with non-empty nextLink (#96211, @feiskyer) [SIG Cloud Provider]

    • Fix pull image error from multiple ACRs using azure managed identity (#96355, @andyzhangx) [SIG Cloud Provider]

    • Fix race condition on timeCache locks. (#94751, @auxten)

    • Fix regression on kubectl port-forward when TCP and UCP services were configured on the same port. (#94728, @amorenoz)

    • Fix scheduler cache snapshot when a Node is deleted before its Pods (#95130, @alculquicondor) [SIG Scheduling]

    • Fix the cloudprovider_azure_api_request_duration_seconds metric buckets to correctly capture the latency metrics. Previously, the majority of the calls would fall in the “+Inf” bucket. (#94873, @marwanad) [SIG Cloud Provider and Instrumentation]

    • Fix vSphere volumes that could be erroneously attached to wrong node (#96224, @gnufied) [SIG Cloud Provider and Storage]

    • Fix verb & scope reporting for kube-apiserver metrics (LIST reported instead of GET) (#95562, @wojtek-t) [SIG API Machinery and Testing]

    • Fix vSphere detach failure for static PVs (#95447, @gnufied) [SIG Cloud Provider and Storage]

    • Fix: azure disk resize error if source does not exist (#93011, @andyzhangx) [SIG Cloud Provider]

    • Fix: detach azure disk broken on Azure Stack (#94885, @andyzhangx) [SIG Cloud Provider]

    • Fix: resize Azure disk issue when it’s in attached state (#96705, @andyzhangx) [SIG Cloud Provider]

    • Fix: smb valid path error (#95583, @andyzhangx) [SIG Storage]

    • Fix: use sensitiveOptions on Windows mount (#94126, @andyzhangx) [SIG Cloud Provider and Storage]

    • Fixed a bug causing incorrect formatting of kubectl describe ingress. (#94985, @howardjohn) [SIG CLI and Network]

    • Fixed a bug in client-go where new clients with customized Dial, Proxy, GetCert config may get stale HTTP transports. (#95427, @roycaihw) [SIG API Machinery]

    • Fixed a bug that prevents kubectl to validate CRDs with schema using x-kubernetes-preserve-unknown-fields on object fields. (#96369, @gautierdelorme) [SIG API Machinery and Testing]

    • Fixed a bug that prevents the use of ephemeral containers in the presence of a validating admission webhook. (#94685, @verb) [SIG Node and Testing]

    • Fixed a bug where aggregator_unavailable_apiservice metrics were reported for deleted apiservices. (#96421, @dgrisonnet) [SIG API Machinery and Instrumentation]

    • Fixed a bug where improper storage and comparison of endpoints led to excessive API traffic from the endpoints controller (#94112, @damemi) [SIG Apps, Network and Testing]

    • Fixed a regression which prevented pods with docker/default seccomp annotations from being created in 1.19 if a PodSecurityPolicy was in place which did not allow runtime/default seccomp profiles. (#95985, @saschagrunert) [SIG Auth]

    • Fixed bug in reflector that couldn’t recover from “Too large resource version” errors with API servers 1.17.0-1.18.5 (#94316, @janeczku) [SIG API Machinery]

    • Fixed bug where kubectl top pod output is not sorted when –sort-by and –containers flags are used together (#93692, @brianpursley) [SIG CLI]

    • Fixed kubelet creating extra sandbox for pods with RestartPolicyOnFailure after all containers succeeded (#92614, @tnqn) [SIG Node and Testing]

    • Fixes an issue proxying to ipv6 pods without specifying a port (#94834, @liggitt) [SIG API Machinery and Network]

    • Fixes code generation for non-namespaced create subresources fake client test. (#96586, @Doude) [SIG API Machinery]

    • Fixes high CPU usage in kubectl drain (#95260, @amandahla) [SIG CLI]

    • For vSphere Cloud Provider, If VM of worker node is deleted, the node will also be deleted by node controller (#92608, @lubronzhan) [SIG Cloud Provider]

    • Gracefully delete nodes when their parent scale set went missing (#95289, @bpineau) [SIG Cloud Provider]

    • HTTP/2 connection health check is enabled by default in all Kubernetes clients. The feature should work out-of-the-box. If needed, users can tune the feature via the HTTP2_READ_IDLE_TIMEOUT_SECONDS and HTTP2_PING_TIMEOUT_SECONDS environment variables. The feature is disabled if HTTP2_READ_IDLE_TIMEOUT_SECONDS is set to 0. (#95981, @caesarxuchao) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Node]

    • If the user specifies an invalid timeout in the request URL, the request will be aborted with an HTTP 400.

      • If the user specifies a timeout in the request URL that exceeds the maximum request deadline allowed by the apiserver, the request will be aborted with an HTTP 400. (#96061, @tkashem) [SIG API Machinery, Network and Testing]
    • If we set SelectPolicy MinPolicySelect on scaleUp behavior or scaleDown behavior,Horizontal Pod Autoscaler doesn’t automatically scale the number of pods correctly (#95647, @JoshuaAndrew) [SIG Apps and Autoscaling]

    • Ignore apparmor for non-linux operating systems (#93220, @wawa0210) [SIG Node and Windows]

    • Ignore root user check when windows pod starts (#92355, @wawa0210) [SIG Node and Windows]

    • Improve error messages related to nodePort endpoint changes conntrack entries cleanup. (#96251, @ravens) [SIG Network]

    • In dual-stack clusters, kubelet will now set up both IPv4 and IPv6 iptables rules, which may fix some problems, eg with HostPorts. (#94474, @danwinship) [SIG Network and Node]

    • Increase maximum IOPS of AWS EBS io1 volume to current maximum (64,000). (#90014, @jacobmarble)

    • Ipvs: ensure selected scheduler kernel modules are loaded (#93040, @cmluciano) [SIG Network]

    • K8s.io/apimachinery: runtime.DefaultUnstructuredConverter.FromUnstructured now handles converting integer fields to typed float values (#93250, @liggitt) [SIG API Machinery]

    • Kube-proxy now trims extra spaces found in loadBalancerSourceRanges to match Service validation. (#94107, @robscott) [SIG Network]

    • Kubeadm ensures “kubeadm reset” does not unmount the root “/var/lib/kubelet” directory if it is mounted by the user. (#93702, @thtanaka)

    • Kubeadm now makes sure the etcd manifest is regenerated upon upgrade even when no etcd version change takes place (#94395, @rosti) [SIG Cluster Lifecycle]

    • Kubeadm now warns (instead of error out) on missing “ca.key” files for root CA, front-proxy CA and etcd CA, during “kubeadm join –control-plane” if the user has provided all certificates, keys and kubeconfig files which require signing with the given CA keys. (#94988, @neolit123)

    • Kubeadm: add missing “–experimental-patches” flag to “kubeadm init phase control-plane” (#95786, @Sh4d1) [SIG Cluster Lifecycle]

    • Kubeadm: avoid a panic when determining if the running version of CoreDNS is supported during upgrades (#94299, @zouyee) [SIG Cluster Lifecycle]

    • Kubeadm: ensure the etcd data directory is created with 0700 permissions during control-plane init and join (#94102, @neolit123) [SIG Cluster Lifecycle]

    • Kubeadm: fix coredns migration should be triggered when there are newdefault configs during kubeadm upgrade (#96907, @pacoxu) [SIG Cluster Lifecycle]

    • Kubeadm: fix the bug that kubeadm tries to call ‘docker info’ even if the CRI socket was for another CR (#94555, @SataQiu) [SIG Cluster Lifecycle]

    • Kubeadm: for Docker as the container runtime, make the “kubeadm reset” command stop containers before removing them (#94586, @BedivereZero) [SIG Cluster Lifecycle]

    • Kubeadm: make the kubeconfig files for the kube-controller-manager and kube-scheduler use the LocalAPIEndpoint instead of the ControlPlaneEndpoint. This makes kubeadm clusters more reseliant to version skew problems during immutable upgrades: https://kubernetes.io/docs/setup/release/version-skew-policy/#kube-controller-manager-kube-scheduler-and-cloud-controller-manager (#94398, @neolit123) [SIG Cluster Lifecycle]

    • Kubeadm: relax the validation of kubeconfig server URLs. Allow the user to define custom kubeconfig server URLs without erroring out during validation of existing kubeconfig files (e.g. when using external CA mode). (#94816, @neolit123) [SIG Cluster Lifecycle]

    • Kubectl: print error if users place flags before plugin name (#92343, @knight42) [SIG CLI]

    • Kubelet: assume that swap is disabled when /proc/swaps does not exist (#93931, @SataQiu) [SIG Node]

    • New Azure instance types do now have correct max data disk count information. (#94340, @ialidzhikov) [SIG Cloud Provider and Storage]

    • Port mapping now allows the same containerPort of different containers to different hostPort without naming the mapping explicitly. (#94494, @SergeyKanzhelev)

    • Print go stack traces at -v=4 and not -v=2 (#94663, @soltysh) [SIG CLI]

    • Recreate EndpointSlices on rapid Service creation. (#94730, @robscott)

    • Reduce volume name length for vSphere volumes (#96533, @gnufied) [SIG Storage]

    • Remove ready file and its directory (which is created during volume SetUp) during emptyDir volume TearDown. (#95770, @jingxu97) [SIG Storage]

    • Reorganized iptables rules to fix a performance issue (#95252, @tssurya) [SIG Network]

    • Require feature flag CustomCPUCFSQuotaPeriod if setting a non-default cpuCFSQuotaPeriod in kubelet config. (#94687, @karan) [SIG Node]

    • Resolves a regression in 1.19+ with workloads targeting deprecated beta os/arch labels getting stuck in NodeAffinity status on node startup. (#96810, @liggitt) [SIG Node]

    • Resolves non-deterministic behavior of the garbage collection controller when ownerReferences with incorrect data are encountered. Events with a reason of OwnerRefInvalidNamespace are recorded when namespace mismatches between child and owner objects are detected. The kubectl-check-ownerreferences tool can be run prior to upgrading to locate existing objects with invalid ownerReferences.

      • A namespaced object with an ownerReference referencing a uid of a namespaced kind which does not exist in the same namespace is now consistently treated as though that owner does not exist, and the child object is deleted.
      • A cluster-scoped object with an ownerReference referencing a uid of a namespaced kind is now consistently treated as though that owner is not resolvable, and the child object is ignored by the garbage collector. (#92743, @liggitt) [SIG API Machinery, Apps and Testing]
    • Skip [k8s.io/kubernetes@v1.19.0/test/e2e/storage/testsuites/base.go:162]: Driver azure-disk doesn’t support snapshot type DynamicSnapshot – skipping skip [k8s.io/kubernetes@v1.19.0/test/e2e/storage/testsuites/base.go:185]: Driver azure-disk doesn’t support ntfs – skipping (#96144, @qinpingli) [SIG Storage and Testing]

    • StatefulSet Controller now waits for PersistentVolumeClaim deletion before creating pods. (#93457, @ymmt2005)

    • StreamWatcher now calls HandleCrash at appropriate sequence. (#93108, @lixiaobing1)

    • Support the node label node.kubernetes.io/exclude-from-external-load-balancers (#95542, @nilo19) [SIG Cloud Provider]

    • The AWS network load balancer attributes can now be specified during service creation (#95247, @kishorj) [SIG Cloud Provider]

    • The /debug/api_priority_and_fairness/dump_requests path at an apiserver will no longer return a phantom line for each exempt priority level. (#93406, @MikeSpreitzer) [SIG API Machinery]

    • The kube-apiserver will no longer serve APIs that should have been deleted in GA non-alpha levels. Alpha levels will continue to serve the removed APIs so that CI doesn’t immediately break. (#96525, @deads2k) [SIG API Machinery]

    • The kubelet recognizes the –containerd-namespace flag to configure the namespace used by cadvisor. (#87054, @changyaowei) [SIG Node]

    • Unhealthy pods covered by PDBs can be successfully evicted if enough healthy pods are available. (#94381, @michaelgugino) [SIG Apps]

    • Update Calico to v3.15.2 (#94241, @lmm) [SIG Cloud Provider]

    • Update default etcd server version to 3.4.13 (#94287, @jingyih) [SIG API Machinery, Cloud Provider, Cluster Lifecycle and Testing]

    • Update max azure data disk count map (#96308, @andyzhangx) [SIG Cloud Provider and Storage]

    • Update the PIP when it is not in the Succeeded provisioning state during the LB update. (#95748, @nilo19) [SIG Cloud Provider]

    • Update the frontend IP config when the service’s pipName annotation is changed (#95813, @nilo19) [SIG Cloud Provider]

    • Update the route table tag in the route reconcile loop (#96545, @nilo19) [SIG Cloud Provider]

    • Use NLB Subnet CIDRs instead of VPC CIDRs in Health Check SG Rules (#93515, @t0rr3sp3dr0) [SIG Cloud Provider]

    • Users will see increase in time for deletion of pods and also guarantee that removal of pod from api server would mean deletion of all the resources from container runtime. (#92817, @kmala) [SIG Node]

    • Very large patches may now be specified to kubectl patch with the --patch-file flag instead of including them directly on the command line. The --patch and --patch-file flags are mutually exclusive. (#93548, @smarterclayton) [SIG CLI]

    • Volume binding: report UnschedulableAndUnresolvable status instead of an error when bound PVs not found (#95541, @cofyc) [SIG Apps, Scheduling and Storage]

    • Warn instead of fail when creating Roles and ClusterRoles with custom verbs via kubectl (#92492, @eddiezane) [SIG CLI]

    • When creating a PVC with the volume.beta.kubernetes.io/storage-provisioner annotation already set, the PV controller might have incorrectly deleted the newly provisioned PV instead of binding it to the PVC, depending on timing and system load. (#95909, @pohly) [SIG Apps and Storage]

    • [kubectl] Fail when local source file doesn’t exist (#90333, @bamarni) [SIG CLI]

    Other (Cleanup or Flake)

    • Update the Debian images to pick up CVE fixes in the base images:
      • Update the debian-base image to v1.7.0
      • Update the debian-iptables image to v1.6.1 (#102341, @cpanato) [SIG API Machinery and Testing]
    • Kubeadm: change the default image repository for CI images from ‘gcr.io/kubernetes-ci-images’ to ‘gcr.io/k8s-staging-ci-images’ (#97087, @SataQiu) [SIG Cluster Lifecycle]
    • Resolves flakes in the Ingress conformance tests due to conflicts with controllers updating the Ingress object (#98430, @liggitt) [SIG Network and Testing]
    • Handle slow CronJob lister in CronJob controller v2 and improve memory footprint. (#96443, @alaypatel07) [SIG Apps]
    • –redirect-container-streaming is no longer functional. The flag will be removed in v1.22 (#95935, @tallclair) [SIG Node]
    • A new metric requestAbortsTotal has been introduced that counts aborted requests for each group, version, verb, resource, subresource and scope. (#95002, @p0lyn0mial) [SIG API Machinery, Cloud Provider, Instrumentation and Scheduling]
    • API priority and fairness metrics use snake_case in label names (#96236, @adtac) [SIG API Machinery, Cluster Lifecycle, Instrumentation and Testing]
    • Add fine-grained debugging to intra-pod conformance test to troubleshoot networking issues for potentially unhealthy nodes when running conformance or sonobuoy tests. (#93837, @jayunit100)
    • Add the following metrics:
      • network_plugin_operations_total
      • network_plugin_operations_errors_total (#93066, @AnishShah)
    • Adds a bootstrapping ClusterRole, ClusterRoleBinding and group for /metrics, /livez/, /readyz/, & /healthz/- endpoints. (#93311, @logicalhan) [SIG API Machinery, Auth, Cloud Provider and Instrumentation]
    • AdmissionReview objects sent for the creation of Namespace API objects now populate the namespace attribute consistently (previously the namespace attribute was empty for Namespace creation via POST requests, and populated for Namespace creation via server-side-apply PATCH requests) (#95012, @nodo) [SIG API Machinery and Testing]
    • Applies translations on all command descriptions (#95439, @HerrNaN) [SIG CLI]
    • Base-images: Update to debian-iptables:buster-v1.3.0
      • Uses iptables 1.8.5
      • base-images: Update to debian-base:buster-v1.2.0
      • cluster/images/etcd: Build etcd:3.4.13-1 image
        • Uses debian-base:buster-v1.2.0 (#94733, @justaugustus) [SIG API Machinery, Release and Testing]
    • Changed: default “Accept-Encoding” header removed from HTTP probes. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#http-probes (#96127, @fonsecas72) [SIG Network and Node]
    • Client-go header logging (at verbosity levels >= 9) now masks Authorization header contents (#95316, @sfowl) [SIG API Machinery]
    • Decrease warning message frequency on setting volume ownership for configmap/secret. (#92878, @jvanz)
    • Enhance log information of verifyRunAsNonRoot, add pod, container information (#94911, @wawa0210) [SIG Node]
    • Fix func name NewCreateCreateDeploymentOptions (#91931, @lixiaobing1) [SIG CLI]
    • Fix kubelet to properly log when a container is started. Previously, kubelet may log that container is dead and was restarted when it was actually started for the first time. This behavior only happened on pods with initContainers and regular containers. (#91469, @rata)
    • Fixes the message about no auth for metrics in scheduler. (#94035, @zhouya0) [SIG Scheduling]
    • Generators for services are removed from kubectl (#95256, @Git-Jiro) [SIG CLI]
    • Introduce kubectl-convert plugin. (#96190, @soltysh) [SIG CLI and Testing]
    • Kube-scheduler now logs processed component config at startup (#96426, @damemi) [SIG Scheduling]
    • Kubeadm: Separate argument key/value in log msg (#94016, @mrueg) [SIG Cluster Lifecycle]
    • Kubeadm: remove the CoreDNS check for known image digests when applying the addon (#94506, @neolit123) [SIG Cluster Lifecycle]
    • Kubeadm: update the default pause image version to 1.4.0 on Windows. With this update the image supports Windows versions 1809 (2019LTS), 1903, 1909, 2004 (#95419, @jsturtevant) [SIG Cluster Lifecycle and Windows]
    • Kubectl: the generator flag of kubectl autoscale has been deprecated and has no effect, it will be removed in a feature release (#92998, @SataQiu) [SIG CLI]
    • Lock ExternalPolicyForExternalIP to default, this feature gate will be removed in 1.22. (#94581, @knabben) [SIG Network]
    • Mask ceph RBD adminSecrets in logs when logLevel >= 4. (#95245, @sfowl)
    • Remove offensive words from kubectl cluster-info command. (#95202, @rikatz)
    • Remove support for “ci/k8s-master” version label in kubeadm, use “ci/latest” instead. See kubernetes/test-infra#18517. (#93626, @vikkyomkar)
    • Remove the dependency of csi-translation-lib module on apiserver/cloud-provider/controller-manager (#95543, @wawa0210) [SIG Release]
    • Scheduler framework interface moved from pkg/scheduler/framework/v1alpha to pkg/scheduler/framework (#95069, @farah) [SIG Scheduling, Storage and Testing]
    • Service.beta.kubernetes.io/azure-load-balancer-disable-tcp-reset is removed. All Standard load balancers will always enable tcp resets. (#94297, @MarcPow) [SIG Cloud Provider]
    • Stop propagating SelfLink (deprecated in 1.16) in kube-apiserver (#94397, @wojtek-t) [SIG API Machinery and Testing]
    • Strip unnecessary security contexts on Windows (#93475, @ravisantoshgudimetla) [SIG Node, Testing and Windows]
    • To ensure the code be strong, add unit test for GetAddressAndDialer (#93180, @FreeZhang61) [SIG Node]
    • UDP and SCTP protocols can left stale connections that need to be cleared to avoid services disruption, but they can cause problems that are hard to debug. Kubernetes components using a loglevel greater or equal than 4 will log the conntrack operations and its output, to show the entries that were deleted. (#95694, @aojea) [SIG Network]
    • Update CNI plugins to v0.8.7 (#94367, @justaugustus) [SIG Cloud Provider, Network, Node, Release and Testing]
    • Update cri-tools to v1.19.0 (#94307, @xmudrii) [SIG Cloud Provider]
    • Update etcd client side to v3.4.13 (#94259, @jingyih) [SIG API Machinery and Cloud Provider]
    • Users will now be able to configure all supported values for AWS NLB health check interval and thresholds for new resources. (#96312, @kishorj) [SIG Cloud Provider]
    • V1helpers.MatchNodeSelectorTerms now accepts just a Node and a list of Terms (#95871, @damemi) [SIG Apps, Scheduling and Storage]
    • vSphere: improve logging message on node cache refresh event (#95236, @andrewsykim) [SIG Cloud Provider]
    • MatchNodeSelectorTerms function moved to k8s.io/component-helpers (#95531, @damemi) [SIG Apps, Scheduling and Storage]
    • kubectl api-resources now prints the API version (as ‘API group/version’, same as output of kubectl api-versions). The column APIGROUP is now APIVERSION (#95253, @sallyom) [SIG CLI]
    • kubectl get ingress now prefers the networking.k8s.io/v1 over extensions/v1beta1 (deprecated since v1.14). To explicitly request the deprecated version, use kubectl get ingress.v1beta1.extensions. (#94309, @liggitt) [SIG API Machinery and CLI]

    Dependencies

    Added
    • cloud.google.com/go/firestore: v1.1.0
    • github.com/Azure/go-autorest: v14.2.0+incompatible
    • github.com/armon/go-metrics: f0300d1
    • github.com/armon/go-radix: 7fddfc3
    • github.com/bketelsen/crypt: 5cbc8cc
    • github.com/form3tech-oss/jwt-go: v3.2.2+incompatible
    • github.com/fvbommel/sortorder: v1.0.1
    • github.com/hashicorp/consul/api: v1.1.0
    • github.com/hashicorp/consul/sdk: v0.1.1
    • github.com/hashicorp/errwrap: v1.0.0
    • github.com/hashicorp/go-cleanhttp: v0.5.1
    • github.com/hashicorp/go-immutable-radix: v1.0.0
    • github.com/hashicorp/go-msgpack: v0.5.3
    • github.com/hashicorp/go-multierror: v1.0.0
    • github.com/hashicorp/go-rootcerts: v1.0.0
    • github.com/hashicorp/go-sockaddr: v1.0.0
    • github.com/hashicorp/go-uuid: v1.0.1
    • github.com/hashicorp/go.net: v0.0.1
    • github.com/hashicorp/logutils: v1.0.0
    • github.com/hashicorp/mdns: v1.0.0
    • github.com/hashicorp/memberlist: v0.1.3
    • github.com/hashicorp/serf: v0.8.2
    • github.com/jmespath/go-jmespath/internal/testify: v1.5.1
    • github.com/mitchellh/cli: v1.0.0
    • github.com/mitchellh/go-testing-interface: v1.0.0
    • github.com/mitchellh/gox: v0.4.0
    • github.com/mitchellh/iochan: v1.0.0
    • github.com/pascaldekloe/goe: 57f6aae
    • github.com/posener/complete: v1.1.1
    • github.com/ryanuber/columnize: 9b3edd6
    • github.com/sean-/seed: e2103e2
    • github.com/subosito/gotenv: v1.2.0
    • github.com/willf/bitset: d5bec33
    • gopkg.in/ini.v1: v1.51.0
    • gopkg.in/yaml.v3: 9f266ea
    • rsc.io/quote/v3: v3.1.0
    • rsc.io/sampler: v1.3.0
    Changed
    Removed
    • github.com/armon/consul-api: eb2c6b5
    • github.com/go-ini/ini: v1.9.0
    • github.com/ugorji/go: v1.1.4
    • github.com/xlab/handysort: fb3537e
    • github.com/xordataexchange/crypt: b2862e3
    • vbom.ml/util: db5cfe1

    aws-ebs-csi-driver 2.1.0

    Changed

    • Update aws-ebs-csi-driver to v1.1.0.

    aws-cni 1.8.0

    Changes since v1.7.10:

    • Bug - Use symmetric return path for non-VPC traffic - alternate solution (#1475, @kishorj)
    • Bug - Gracefully handle failed ENI SG update (#1341, @jayanthvn)
    • Bug - Fix CNI crashing when there is no available IP addresses (#1499, @M00nF1sh)
    • Bug - Use primary ENI SGs if SG is null for Custom networking (#1259, @jayanthvn)
    • Bug - Don’t cache dynamic VPC IPv4 CIDR info (#1113, @anguslees)
    • Improvement - Address Excessive API Server calls from CNI Pods (#1419, @achevuru)
    • Improvement - refine ENI tagging logic (#1482, @M00nF1sh)
    • Improvement - Change tryAssignIPs to assign up to configured WARM_IP_TARGET (#1279, @jacksontj)
    • Improvement - Use regional STS endpoint (#1332, @nithu0115)
    • Improvement - Update containernetworking dependencies (#1200, @mogren)
    • Improvement - Split Calico manifest into two (#1410, @caseydavenport)
    • Improvement - Update Calico manifest to support ARM & AMD (#1282, @jayanthvn)
    • Improvement - Auto gen of AWS CNI, metrics helper and calico artifacts through helm (#1271, @jayanthvn)
    • Improvement - Refactor EC2 Metadata IMDS code (#1225, @anguslees)
    • Improvement - Unnecessary logging for each CNI invocation (#1469, @jayanthvn)
    • Improvement - New instance types (#1463, @jayanthvn)
    • Improvement - Use ‘exec’ ENTRYPOINTs (#1432, @anguslees)
    • Improvement - Fix logging texts for ENI cleanup (#1209, @mogren)
    • Improvement - Remove Duplicated vlan IPTable rules (#1208, @mogren)
    • Improvement - Minor code cleanup (#1198, @mogren)
    • HelmChart - Adding flags to support overriding container runtime endpoint. (#1443, @haouc)
    • HelmChart - Add podLabels to amazon-vpc-cni chart (#1440, @haouc)
    • HelmChart - Add workflow to sync aws-vpc-cni helm chart to eks-charts (#1430, @fawadkhaliq)
    • Testing - Remove validation of VPC CIDRs from ip rules (#1476, @kishorj)
    • Testing - Updated agent version (#1474, @cgchinmay)
    • Testing - Fix for CI failure (#1470, @achevuru)
    • Testing - Binary for mtu and veth prefix check (#1458, @cgchinmay)
    • Testing - add test to verify cni-metrics-helper puts metrics to CW (#1461, @abhipth)
    • Testing - add e2e test for security group for pods (#1459, @abhipth)
    • Testing - Added Test cases for EnvVars check on CNI daemonset (#1431, @cgchinmay)
    • Testing - add test to verify host networking setup & cleanup (#1457, @abhipth)
    • Testing - Runners failing because of docker permissions (#1456, @jayanthvn)
    • Testing - decouple test helper input struct from netlink library (#1455, @abhipth)
    • Testing - add custom networking e2e test suite (#1445, @abhipth)
    • Testing - add integration test for ipamd env variables (#1453, @abhipth)
    • Testing - add agent for testing pod networking (#1448, @abhipth)
    • Testing - fix format of commited code to fix unit test step (#1449, @abhipth)
    • Testing - Unblocks Github Action Integration Tests (#1435, @couralex6)
    • Testing - add warm ENI/IP target integration tests (#1438, @abhipth)
    • Testing - add service connectivity test (#1436, @abhipth)
    • Testing - add network connectivity test (#1424, @abhipth)
    • Testing - add ginkgo automation framework (#1416, @abhipth)
    • Testing - Add some test coverage to allocating ENIs (#1234, @mogren)
    • Testing - Add some minimal tests to metrics (#1228, @mogren)

    Changes since v1.7.9:

    • Improvement - Multi card support - Prevent route override for primary ENI across multi-cards ENAs (#1396 , @jayanthvn)

    Changes since v1.7.8:

    • Improvement - Adds http timeout to aws sessions (#1370 by couralex6)
    • Improvement - Switch calico to be deployed with the Tigera operator (#1297 by tmjd)
    • Improvement - Update calico to v3.17.1 (#1328 by lwr20)
    • Improvement - update plugins to v0.9.0 (#1362 by fr0stbyte)
    • Improvement - update github.com/containernetworking/plugins to v0.9.0 (#1350 by fr0stbyte)
    • Bug - Fix regex match for getting primary interface (#1311 by Jayanthvn)
    • Bug - Output to stderr when no log file path is passed (#1275 by couralex6)
    • Bug - Fix deletion of hostVeth rule for pods using security group (#1376 by SaranBalaji90)

    cluster-autoscaler 1.20.3

    Changed

    • Allow users to set container resources;
    • Update cluster-autoscaler to version 1.20.0.
  • This release fixes the issue which caused Kubernetes nodes to lose network connectivity in certain situations by reverting Flatcar Container Linux to an older version.

    Warning: The nginx app needs to be updated to v1.14.0+ because a new version of external-dns is included in this release.

    Change details

    • revert Flatcar Container Linux to v2605.12.0.
  • This release fixes the issue which prevented clusters being created on some management clusters with the AWS v14.2.0 release.

    Warning: The nginx app needs to be updated to v1.14.0+ because a new version of external-dns is included in this release.

    Change details

    aws-operator 10.3.1

    Added

    • Backport China Flatcar AMIs.

    cert-operator 1.0.1

    Fixed

    • Add list permission for cluster.x-k8s.io.

    Changed

    • Update Kubernetes dependencies to 1.18 versions.
    • Reconcile CertConfigs based on their cert-operator.giantswarm.io/version label.

    Removed

    • Stop using the VersionBundle version.

    Added

    • Add network policy resource.
    • Added lookup for nodepool clusters in other namespaces than default.