Last modified October 6, 2017

Securing your Cluster with RBAC and PSP

Two of the most central mechanisms to secure your cluster in Kubernetes are RBAC (Role Based Access Control) and PSP (Pod Security Policies). Together, these two allow you to create fine-grained roles and policies to manage access control for users and software running on your cluster. Both are enabled by default on Giant Swarm clusters.

Role Based Access Control

The RBAC API defines both roles and bindings on either namespace or cluster level. Like any other Kubernetes API object, these can be defined by writing YAML (or JSON) manifests.

Note that to apply these manifests, you need a user with higher level access than the access you want to set up. When in doubt, use a Cluster Admin account to apply RBAC manifests.

Roles & Cluster Roles

While a Role is used to grant permissions within a single namespace, a ClusterRole can grant the same permissions, but can be bound on a cluster-scope.

Note: If you want to grant access to resources that are not namespaced you always need a ClusterRole.

The following Role grants read access to pods in the default namespace.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  namespace: default
  name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

With a ClusterRole like the following you can grant read access to secrets. You can do this in any particular namespace or across all namespaces, depending on how the ClusterRole is bound.

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  # "namespace" omitted since ClusterRoles are not namespaced
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]

Role & Cluster Role Bindings

With bindings you can bind Roles or Cluster Roles to subjects (users, groups, or service accounts).

A RoleBinding grants access within a certain namespace, while a ClusterRoleBinding grants cluster-wide access.

A RoleBinding can either reference a Role or a ClusterRole, but will only grant permissions on the namespace it is applied to.

The following RoleBinding grants the “pod-reader” role to the user “jane” within the “default” namespace. This allows “jane” to read pods in the “default” namespace.

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: read-pods
  namespace: default
subjects:
- kind: User
  name: jane
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Binding Cluster Roles with Role Bindings allows cluster adminstrators to define a set of common roles for the entire cluster, then reuse them within multiple namespaces.

For instance, even though the following RoleBinding refers to a ClusterRole, “dave” (the subject) will only be able read secrets in the “development” namespace (the namespace of the RoleBinding).

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: read-secrets
  namespace: development # This only grants permissions within the "development" namespace.
subjects:
- kind: User
  name: dave
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io

Finally, a ClusterRoleBinding may be used to grant permission at the cluster level and in all namespaces. The following ClusterRoleBinding allows any user in the group “manager” to read secrets in any namespace.

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: read-secrets-global
subjects:
- kind: Group
  name: manager
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io

Referring to Subjects

Bindings can refer to subjects that are either single users, groups, or service accounts. The latter are needed to grant API access (and with PSPs also Pod privileges) to certain Pods, e.g. for monitoring and logging agents.

For a detailed explanation of how to refer to subjects in Bindings we defer to the official RBAC documentation.

Default Roles and Role Bindings

Your Kubernetes cluster comes by default with a set of Roles and Cluster Roles as well as some default Bindings. These are automatically reconcilliated and thus cannot be changed or deleted.

You can use the Roles and Cluster Roles to create bindings for your users. Following example would give all users in the group mynamespace-admin admin access to the mynamespace namespace.

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: mynamespace-admin
  namespace: mynamespace
subjects:
- kind: Group
  name: mynamespace-admin
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: admin
  apiGroup: rbac.authorization.k8s.io

One of the most important default Role Bindings is the Cluster Admin role, which depicts a super-user in the cluster. By default it is bound to the system:masters group. Thus, if you need cluster admin access to your Kubernetes cluster, just use a user (e.g. by creating a key pair with gsctl or happa) that carries that group.

For a complete overview of default roles and bindings we defer to the official RBAC documentation.

Verifiyng if you have Access

If you are not sure if your (or another) user is allowed to do a certain action, you can verify that with the kubectl auth can-i command.

$ kubectl auth can-i create deployments --namespace production
yes

You can also impersonate users to check their access:

$ kubectl auth can-i create deployments --namespace production --as jane
no

or impersonate a Service Account to check if it is set up right:

$ kubectl auth can-i use podsecuritypolicies/privileged -n kube-system --as=system:serviceaccount:kube-system:calico-node
yes

You can also verfiy access for a whole group:

$ kubectl auth can-i get pods --as-group=system:masters
yes

Pod Security Policies

A PodSecurityPolicy object defines a set of conditions that a pod must run with in order to be accepted into the system. It governs the ability to make requests on a Pod that affect the SecurityContext that will be applied to a Pod and container.

By default, Giant Swarm clusters come with two PSPs defined - a privileged policy that allows almost any Security Context and a restricted policy that mainly restricts users from running privileged containers and mount host paths as volumes.

Additionally, there are two respective cluster roles defined - privileged-psp-user and restricted-psp-user.

By default all authenticated users are assigned the restricted-psp-user role.

If you need to run privileged containers, you need to either use a cluster admin user or bind the privileged-psp-user role to your desired user or group.

Keep in mind that Pods that need extensive Security Contexts need to run with a Service Account that has a similarly extensive PSP bound to it. You can either bind the already existing privileged-psp-user role to it, or create a PSP that caters specifically to your desired Security Context.

For details on PSPs we defer to the official PSP documentation.

User Management with Giant Swarm

If you are managing users through the Giant Swarm API or its clients, gsctl or Happa, you can specify a Common Name Prefix and Organizations when you create key-pairs of kubeconfigs.

Using Common Name as Username in Kubernetes

Setting a Common Name Prefix results in a username like the following:

<cn-prefix>.user.api.<cluster-domain>

where <cn-prefix> is a username of your choice and <cluster-domain> is your cluster’s domain, e.g. rlbz2.k8s.ginger.eu-central-1.aws.gigantic.io.

When binding roles to a user you need to use the full username mentioned above.

Using Organizations as Groups in Kubernetes

Organizations you set when creating key-pairs or kubeconfigs get mapped to groups inside Kubernetes.

You can then assign roles to a whole group of users.

A user can be part of multiple groups and thus be assigned multiple roles, too.

There is only a single predefined user group inside Kubernetes. Members of the system:masters group will directly be assigned the default cluster-admin role inside your cluster, which is allowed to do anything. We recommend only using this kind of user, when bootstrapping security settings for the cluster.

Further Reading