Last modified May 25, 2023

Handling cloud provider credentials

In order to manage workload clusters in your cloud provider accounts/subscriptions, the Giant Swarm controllers require some configuration so they are able to act on your behalf.

In this article we explain how to provide this configuration via the Management API.

Terminology

As we are dealing with two cloud providers here which use different vocabulary in their own worlds, let’s first establish some terminology for this article.

Account (AWS) or Subscription (Azure): we use the term account here for both.
Credentials: a set of details which enable a controller or a Giant Swarm staff member to work with your cloud provider account on your behalf. This article is mostly about providing these credentials in the right place and format.

Credential secrets explained

Credentials are stored in the management cluster in the form of a Secret resource.

In the metadata, credential secrets must provide the giantswarm.io/managed-by: credentiald and app: credentiald labels. We also recommend to choose a resource name starting with credential-.

The data part contains the cloud provider credentials that should be used. The format here differs slightly between cloud providers.

AWS
Azure

Here is an example for AWS:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: credential-example
  namespace: my-namespace
  labels:
    app: credentiald
    giantswarm.io/managed-by: credentiald
data:
  aws.admin.arn: YXJuOmF3czppYW06OjEyMzQ1Njc4OTA6cm9sZS9HaWFudFN3YXJtQWRtaW4=
  aws.awsoperator.arn: YXJuOmF3czppYW06OjEyMzQ1Njc4OTA6cm9sZS9HaWFudFN3YXJtQVdTT3BlcmF0b3I=

In above AWS example, these are two fields which both indicate AWS IAM role identifiers (ARNs). As typical with opaque secrets, the value is encoded in base64. Here is how the two example strings would look like when decoded:

Key	Example value (decoded)
`aws.admin.arn`	`arn:aws:iam::1234567890:role/GiantSwarmAdmin`
`aws.awsoperator.arn`	`arn:aws:iam::1234567890:role/GiantSwarmAWSOperator`

The first of the two identifies the IAM role to be assumed by Giant Swarm staff for operations tasks. The second one is used by aws-operator, which is the software in charge of automatic cluster management. Easily visible, they include the numeric AWS account ID and also the IAM role name to be assumed.

We provide detailed documentation regarding how to configure these roles in your AWS account.

Here is an example for a credential secret on Azure:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: credential-example
  namespace: my-namespace
  labels:
    app: credentiald
    giantswarm.io/managed-by: credentiald
data:
data:
  azure.azureoperator.clientid: MTIzNGFiY2QtYWIxMi0xMmFiLWNkMzQtYWJjZDEyMzRhYmNkCg==
  azure.azureoperator.subscriptionid: NTY3OGFiY2QtYWI1Ni01NmFiLWNkNTYtYWJjZDU2NzhhYmNkCg==
  azure.azureoperator.tenantid: MTIzNGRlZmctZGUxMi0xMmRlLWRlMzQtZGVmZzEyMzRkZWZnCg==
  azure.azureoperator.clientsecret: YWJjZC1lZmdoaWprbG1uTE9QUTEyMzQ1Njd+ODlSU3R1dnd4Cg==

Here the data part contains four fields which specify the IDs of Client, Subscription and Tenant as well as the Client Secret.

Decoded example:

Key	Example value (decoded)
`azure.azureoperator.clientid`	`1234abcd-ab12-12ab-cd34-abcd1234abcd`
`azure.azureoperator.subscriptionid`	`5678abcd-ab56-56ab-cd56-abcd5678abcd`
`azure.azureoperator.tenantid`	`1234defg-de12-12de-de34-defg1234defg`
`azure.azureoperator.clientsecret`	`abcd-efghijklmnLOPQ1234567~89RStuvwx`

In order for azure-operator to manage workload clusters using these credentials, it will need access to your Subscription using a Service Principal.

We provide a detailed guide prepare and obtain this data.

Referencing credentials in the cluster resource

When creating a workload cluster, depending on the cloud provider, there will be an AWSCluster or AzureCluster resource created, containing the provider-specific configuration of your workload cluster.

If you create your cluster manifest manually or via the kubectl gs template cluster command, you are free to adapt the provider cluster resource to match your exact requirements before submitting the manifest to the API (e. g. via kubectl apply).

In AWS, it is possible to reference your credential secret directly. The custom resource definition (CRD) provides an attribute .spec.provider.credentialSecret. The two sub-attributes name has to match the credential secret’s resource name. The namespace sub-attribute accordingly must match the secret’s namespace.

Note that this attribute has to be set before submitting the manifest to the Management API. Otherwise, if the AWSCluster resource gets submitted without any .spec.provider.credentialSecret in place, our admission controllers will fill in default values. Read on to understand the defaulting logic.

On Azure, the AzureCluster field .spec.identityRef is set by the azure-operator. It references credential secrets indirectly by creating an org-credential based on the organization your workload cluster is created in.

Defaulting

If the cluster resource does not reference a provider credential secret to use (as explained above), our admission controller fills in some defaults, in this logical order

An organization’s credential secret is used, if it exists
Otherwise the installation’s default secret is used

Both options are explained in further detail below.

Organization default credentials

Any workload cluster in Giant Swarm belongs to an organization as encoded in the giantswarm.io/organization label of the cluster’s Cluster resource.

For each organization you can also deposit a credential secret which will be used automatically for any new workload cluster that does not specify credentials explicitly (as explained above). To be looked up as the organization’s default credentials, your Secret resource must match the following requirements:

The label giantswarm.io/organization must be present, with the value matching the organization name
There must not be any other credential secret matching the same giantswarm.io/organization label value.

This partial example shows the required metadata:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: credential-example
  namespace: org-example
  labels:
    app: credentiald
    giantswarm.io/managed-by: credentiald
    giantswarm.io/organization: acme
data:
  ...

Installation default credentials

As a last resort, if credentials are not referenced explicitly and no organization default credentials are found, our defaulting will use the installation default credentials.

The installation default credentials are typically configured for you by Giant Swarm, using credentials that you provide. They are placed in the namespace giantswarm and thus normally not accessible to customers directly.

The idea of the default credentials is to enable every organization to create clusters, even if they don’t have specific credentials configured. This can be a viable use case if all tenants are trusted and should be allowed to use the same cloud provider account.

Choosing the right method for you

The defaulting options as well as the option to reference credentials explicitly per workload cluster should offer the right method for each use case. To sum up the different methods and reasons to use them:

If you want to run all workload clusters in the same cloud provider account, use the installation default credentials. As a result you won’t have to worry about this detail during cluster creation at all.
If you plan to run each organization’s clusters in a different cloud provider account, use the organization default credentials mechanism. Just make sure to set up the organization’s credentials as early as possible, to prevent any cluster running via the installation default credentials.
On AWS, you can set the cloud provider credentials explicitly for more flexibility. Whether you want to rely on organization level or installation level defaults as a fall back is up to you.