Last modified August 28, 2023
Securing your cluster with RBAC and PSS
Two of the most central mechanisms to secure your cluster in Kubernetes are Role Based Access Control (RBAC) and Pod Security Standards (PSS). Together, they allow you to create fine-grained roles and policies to manage access control for users and software running on your cluster. Both are enabled by default on Giant Swarm clusters.
Role based access control
The RBAC API defines both roles and bindings on either namespace or cluster level. Like any other Kubernetes API object, these can be defined by writing YAML (or JSON) manifests.
Note that to apply these manifests, you need a user with higher level access than the access you want to set up. When in doubt, use a Cluster Admin account to apply RBAC manifests.
The Role and ClusterRole resource
ClusterRole allow a cluster administrator to define a reusable set of permissions. While a
Role is used to grant permissions within a single namespace, a
ClusterRole can grant the same permissions, but can be bound on a cluster-scope.
If you want to grant access to a resource that is not namespaced you have to use a
As an example, the following
Role grants read access to pods in the
kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: namespace: default name: pod-reader rules: - apiGroups: [""] # "" indicates the core API group resources: - pods verbs: - get - watch - list
As you can see, the list of
resources accessible only contains the value
verbs entries define what the subject can do to pods. The three verbs listed in the example above allow for read-only access. Other verbs available for use are
On to another example. With a
ClusterRole like the following you can grant read access to secrets.
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: # "namespace" omitted since ClusterRoles are not namespaced name: secret-reader rules: - apiGroups: [""] resources: - secrets verbs: - get - watch - list
ClusterRole is not namespaced, you can use it either in a particular namespace or across all namespaces, depending on how it is bound. This will be explained in the next section.
You can also create ClusterRoles by combining other ClusterRoles using an
aggregationRule. The permissions of aggregated ClusterRoles are controller-managed, and filled in by unioning the rules of any
ClusterRole that matches the provided label selector.
You can use this to create meta-ClusterRoles that you fill with sub-ClusterRoles like following:
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: monitoring aggregationRule: clusterRoleSelectors: - matchLabels: rbac.example.com/aggregate-to-monitoring: "true" rules:  # Rules are automatically filled in by the controller manager.
Now any ClusterRole you create that carries the
rbac.example.com/aggregate-to-monitoring: "true" label will automatically get added to the
monitoring ClusterRole, e.g.:
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: monitoring-endpoints labels: rbac.example.com/aggregate-to-monitoring: "true" rules: - apiGroups: [""] Resources: ["services", "endpoints", "pods"] verbs: ["get", "list", "watch"]
You can further use this feature to extend already existing ClusterRoles, if you feel an existing role is too restrictive or want to include access to new custom resources. The default ClusterRoles (
view) are already prepared to be extended through aggregation rules.
For example we could extend the
admin ClusterRole to include access to some custom PodSecurityPolicy we created:
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: aggregate-admin labels: # Add these permissions to the "admin" default ClusterRole. rbac.authorization.k8s.io/aggregate-to-admin: "true" rules: - apiGroups: - policy resources: - podsecuritypolicies resourceNames: - admin-psp verbs: - use
After applying above
ClusterRole to the cluster, everyone with the ClusterRole
admin will be able to use the
Binding a Role and ClusterRole
With bindings you can bind a
ClusterRole to subjects like users, groups, or service accounts. There are again two different resource types available:
RoleBindinggrants access within a certain namespace and can bind either a
ClusterRoleBindinggrants access cluster-wide and can only bind a
RoleBinding can either reference a
Role or a
ClusterRole, but will only grant permissions on the namespace it is applied to.
RoleBinding binds the
Role named “pod-reader” to the user “jane” within the “default” namespace. This grants “jane” read access to pods in the “default” namespace.
kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: read-pods namespace: default subjects: - kind: User name: jane apiGroup: rbac.authorization.k8s.io roleRef: kind: Role name: pod-reader apiGroup: rbac.authorization.k8s.io
In contrast, binding a
ClusterRole (instead of a
Role) with a
RoleBinding allows cluster administrators to define a set of common roles for the entire cluster, then reuse them within multiple namespaces.
For instance, even though the following
RoleBinding refers to a
ClusterRole, user “dave” will only be able read secrets in the “development” namespace (the namespace of the
kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: read-secrets namespace: development # This only grants permissions within the "development" namespace. subjects: - kind: User name: dave apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: secret-reader apiGroup: rbac.authorization.k8s.io
ClusterRoleBinding may be used to grant permission at the cluster level and in all namespaces. The following
ClusterRoleBinding allows any user in the group “manager” to read secrets in any namespace.
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: read-secrets-global subjects: - kind: Group name: manager apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: secret-reader apiGroup: rbac.authorization.k8s.io
Referring to subjects
Bindings can refer to subjects that are either single users, groups, or service accounts. The latter are needed to grant API access (and with PSPs also Pod privileges) to certain Pods, e.g. for monitoring and logging agents.
For a detailed explanation of how to refer to subjects in Bindings we defer to the official RBAC documentation.
Default Roles and Role Bindings
Your Kubernetes cluster comes by default with a set of Roles and Cluster Roles as well as some default Bindings. These are automatically reconciled and thus cannot be changed or deleted.
You can use the
ClusterRole resources to create bindings for your users. Following example would grant all users in the group
mynamespace-admin full permissions to resources in the
mynamespace namespace. See how it references a
ClusterRole named “admin”, which comes with Kubernetes by default.
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: mynamespace-admin namespace: mynamespace subjects: - kind: Group name: mynamespace-admin apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: admin apiGroup: rbac.authorization.k8s.io
A super-user role binding
One of the most important default Role Bindings is for the “cluster-admin” role, which depicts a super-user in the cluster. By default it is bound to the
system:masters group. Thus, if you need cluster admin access to your Kubernetes cluster, you need to create user credentials (e.g. by creating a key pair with gsctl or the Giant Swarm Web UI) that include that group.
For a complete overview of default roles and bindings we defer to the official RBAC documentation.
Warning: Be careful assigning super-user as a default role. Giving
cluster-admin role to every user means letting them perform any action in the cluster. As an analogy, it is like you giving root access to every user in a Linux system. Consequently think twice which role your users will have in the system. For Kubernetes, it translates in selecting a username and group name properly. In case you use authentication based on certs, common name and organization would be those respectively. If you are using an external authentication system then be sure it returns the correct user and group name to the Kubernetes API.
Verifying if you Have Access
If you are not sure if your (or another) user is allowed to do a certain action, you can verify that with the
kubectl auth can-i command.
$ kubectl auth can-i create deployments \ --namespace production yes
You can also impersonate users to check their access:
$ kubectl auth can-i create deployments \ --namespace production \ --as jane no
or impersonate a Service Account to check if it is set up right:
$ kubectl auth can-i use podsecuritypolicies/privileged \ -n kube-system \ --as=system:serviceaccount:kube-system:calico-node yes
You can also verify access for a whole group:
$ kubectl auth can-i get pods --as-group=system:masters yes
Pod Security Standards (PSS)
Note: Prior to Kubernetes v1.25, pod-level security configuration was controlled by Pod Security Policies (PSP). PSPs were a Kubernetes API resource which allowed cluster administrators to restrict the values a pod could use for certain security-relevant configuration. Due to limitations of the specification, the PSP type was deprecated in v1.21 and removed in v1.25.
The Kubernetes maintainers publish a set of policies called Pod Security Standards, which describe acceptable pod configurations for different levels of risk. The policies apply to several pod and container specification-level fields which have security implications for the workload, and are grouped into three increasingly restrictive levels.
For example, the
baseline policy level limits the capabilities a pod can use to a limited subset (~14 possibilities). The
restricted level takes this policy further and requires each pod to explicitly drop all capabilities in its Pod spec, and allows adding back only a single capability (
NET_BIND_SERVICE). The least restrictive level,
privileged, is a no-op policy which does not perform any validation or impose any rules. Refer to the official Pod Security Standards docs for more information.
The Pod Security Standards provide only a set of suggested policies intended to be enforced by other implementations of actual controls which validate pods against the PSS rules.
Pod Security Admission (PSA)
The Kubernetes API includes a built-in admission controller called Pod Security Admission, which a specific implementation of a technical control for the Pod Security Standards. It is an admission controller which is built into the API server, and can be configured using labels on cluster namespaces.
Due to perceived limitations in the initial implementation of PSA and to keep feature parity with PSPs for our customers, Giant Swarm uses an external admission controller to enforce these policies instead of PSA.
To learn more about builtin PSA, please refer to the upstream Kubernetes Pod Security Admission documentation. We’ve also written a blog post outlining our decision not to use PSA.
Pod Security Standards with Kyverno
Instead of the Pod Security Admission controller, Giant Swarm clusters use Kyverno along with a set of Kyverno ClusterPolicies which map to the Kubernetes Pod Security Standards.
(outside API Server)") A --> |Enforced by| C("Pod Security Admission (PSA)
(inside API Server)")
By default, Giant Swarm clusters enforce the
restricted level Pod Security Standards. This level aligns with our “secure by default” principle, and is intended to ensure our clusters are hardened according to community best practices.
Cluster administrators have complete flexibility and control over their policy enforcement, and may choose to ease their security requirements or enforce additional policies to suit their risk tolerance and business needs.
A detailed guide to working with Kyverno PSS Policies and PolicyExceptions is available as a standalone resource.
User management with Giant Swarm
If you are managing users through the Giant Swarm REST API or its clients,
gsctl or the Web UI, you can specify a Common Name Prefix and Organizations when you create key-pairs of kubeconfigs.
Using common name as username in Kubernetes
Setting a Common Name Prefix results in a username like the following:
<cn-prefix> is a username of your choice and
<cluster-domain> is your cluster’s domain, e.g.
When binding roles to a user you need to use the full username mentioned above.
Using organizations as groups in Kubernetes
Organizations you set when creating key-pairs or kubeconfigs get mapped to groups inside Kubernetes.
You can then assign roles to a whole group of users.
A user can be part of multiple groups and thus be assigned multiple roles, too.
There is only a single predefined user group inside Kubernetes. Members of the
system:masters group will directly be assigned the default
cluster-admin role inside your cluster, which is allowed to do anything. We recommend only using this kind of user, when bootstrapping security settings for the cluster.
<cn-prefix> defaults to the email you sign in with at Giant Swarm. The default organization is empty. Thus, a user that is created without any additional CN Prefix and/or Organizations won’t have any rights on the cluster unless you bind a role to their specific username.
Bootstrapping and managing access rights
For bootstrapping and managing access rights on your Kubernetes cluster you first need a user that already has all rights that you want to give out. The easiest way to get such a user is creating a user that is in the
system:masters group and thus carries Cluster Admin rights.
To create such a user with
gsctl for a cluster with id
w6wn8 just run following command:
gsctl create kubeconfig \ -c w6wn8 \ -d "Access Management" \ --ttl 1d \ --certificate-organizations "system:masters"
This will create a kubeconfig with a Cluster Admin user that is valid for a day (as we don’t want to yield that power for too long).
With this user we can now start creating Roles and Bindings for our users and apps. For this we will look at some examples.
Note that in these examples we are assuming you are creating users through Giant Swarm’s API.
Giving admin access to a specific namespace
There is a default admin role defined inside Kubernetes. You can bind that role to a user or group of users for a specific namespace with a Role Binding similar to following.
kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: development-admin namespace: development subjects: - kind: User name: jane.w6wn8.k8s.example.eu-central-1.aws.gigantic.io apiGroup: rbac.authorization.k8s.io - kind: Group name: dev-admin apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: admin apiGroup: rbac.authorization.k8s.io
The above YAML gives admin rights to the
development namespace to both the user with CN Prefix
jane and all users within the
You could create kubeconfigs for such users like following.
gsctl create kubeconfig -c w6wn8 -d "Jane" --cn-prefix "jane"
gsctl create kubeconfig -c w6wn8 -d "Marc" --cn-prefix "marc" --certificate-organizations "dev-admin"
Note that even when we don’t need a specific username for giving Marc Admin rights, we should still set a CN Prefix so we can identify Marc’s actions on the cluster.
Giving read access to the whole cluster
There is a default view role defined inside Kubernetes. You can bind that role to a user or group of users for the whole cluster with a Cluster Role Binding similar to following.
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cluster-viewer subjects: - kind: User name: jane.w6wn8.k8s.example.eu-central-1.aws.gigantic.io apiGroup: rbac.authorization.k8s.io - kind: Group name: cluster-view apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: view apiGroup: rbac.authorization.k8s.io
Above YAML gives view rights to the whole cluster to both the user with CN Prefix
jane and all users within the
Let’s assume we already created both users from the example above. As Jane’s username has not changed, she automatically gets the new rights using her existing credentials.
However, Marc is only part of the
dev-admin group, so if we want to give him view access to the cluster we need to give him new credentials.
gsctl create kubeconfig \ -c w6wn8 \ -d "Marc" \ --cn-prefix "marc" \ --certificate-organizations "dev-admin, cluster-view"
With the above Marc would now be part of both groups and thus be bound by both bindings.
Running applications that need API access
Applications running inside your cluster that need access to the Kubernetes API need the right permissions bound to them. For this the Pods need to use a Service Account.
The typical process looks like following example:
1. Create a Service Account for your app
apiVersion: v1 kind: ServiceAccount metadata: name: fluentd namespace: logging
2. Add the Service Account to your app
This is done by adding a line referncing the Service Account to the Pod spec of your Deployment or Daemonset. To be sure you have the right place in the YAML you can put it right above the line
containers: at the same intendation level. The section should look similar to following:
[...] spec: template: metadata: name: fluentd labels: component: fluentd spec: serviceAccountName: fluentd containers: [...]
3. Create a Role for your app
This role should be very specific to the needs of your app and not allow anything more than what the app needs to work. In this example fluentd needs a Cluster Role that can get, watch, and list Pods and Namespaces in the whole cluster.
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluentd rules: - apiGroups: [""] # core API group resources: ["pods", "namespaces"] verbs: ["get", "watch", "list"]
4. Bind the Role to the Service Account
Now we bind the role we created above to the Service Account we’re using.
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluentd subjects: - kind: ServiceAccount name: fluentd namespace: logging roleRef: kind: ClusterRole name: fluentd apiGroup: rbac.authorization.k8s.io
You can revoke access from any user or group of users by either completely removing the binding(s) they are part of or, in case of bindings that bind to several subjects, removing them specifically from the list of subjects in that binding.
Note that bindings that come with the cluster by default like
system:masters cannot be removed as they are reconciled. We highly recommend you only create short-lived users (i.e. certificates with a TTL of a day or less) for this kind of access.