Last modified February 6, 2023

Creating a base template for workload clusters

Add a CAPx (CAPI) Workload Cluster template (cluster App based)

Our CAPx (CAPI provider-specific clusters) are delivered by Giant Swarm as a set of two applications. The first one is an App Custom Resource(CR) with a Cluster instance definition, while the second one is an App CR containing all the default applications needed for a cluster to run correctly. As such, creating a CAPx cluster means that you need to deliver two configured App CRs to the Management Cluster.

Adding definitions can be done on two levels: shared cluster template and version-specific template, see create shared template base and create versioned base.

IMPORTANT, CAPx configuration utilizes the App Platform Configuration Levels, in the following manner:

• cluster templates provide default configuration via App’ config field,
• cluster instances provide custom configuration via App’ extraConfig field, which is overlaid on top of config. The file set with higher priority will prevail in case of colliding config values.

See more about this approach here.

Choose bases

In order to avoid code duplication, it is advised to utilize the bases and overlays of Kustomize in order to configure the cluster.

Create shared cluster template base

IMPORTANT: shared cluster template base should not serve as a standalone base for cluster creation, it is there only to abstract App CRs that are common to all clusters versions, and to provide basic configuration for default apps App. It is then used as a base for other bases, which provide an overlay with a specific configuration. This is to avoid code duplication across bases.

Bear in mind, this is not a complete guide of how to create a perfect base, but rather a mere summary of basic steps needed to move forward. Hence, instructions here will not always be precise in telling you what to change, as this can strongly depend on the resources involved, how much of them you would like to include in a base, etc.

We provide a command to create bases for CAPI clusters in an easy way. A base can be created by running:

kubectl gs gitops add base --provider capa

This command will create the folder structure and the specific folder for the provider capa, including the template files in the template folder. The structure would be:

bases
└── clusters
    └── capa
        └── template
            ├── cluster_config.yaml
            ├── cluster.yaml
            ├── default_apps_config.yaml
            ├── default_apps.yaml
            └── kustomization.yaml

The current possible values for the providers can be checked in the command reference

Create versioned base (optional)

IMPORTANT, versioned cluster template bases use a shared cluster template base and overlay it with a preferably generic configuration for a given cluster version. Versioning comes from the fact that values.yaml schema may change over multiple releases, and although minor differences can be handled on the extraConfig level, it is advised for the bases to follow major values.yaml schema versions to avoid confusion.

There is one example of a versioned base in the gitops-template repository for capo v0.6.0 as major changes were introduced to the values.yaml in the cluster-openstack v0.6.0 release.

IMPORTANT, despite the below instructions referencing kubectl-gs for templating configuration, kubectl-gs generates configuration for the most recent schema only. If you configure a base for older versions of the cluster app, it is advised to check what is generated against the version-specific values.yaml.

In this example we are creating a custom version for capa base:

1. Create a directory structure:

   mkdir -p bases/clusters/capa/v0.21.0
   

2. Use the kubectl gs template cluster to template cluster resources, see an example for the capa provider below. Use arbitrary values for the mandatory fields, we will configure them later in our process:

   kubectl gs template cluster \
   --name mywcl \
   --organization myorg \
   --provider capa | yq -s '.metadata.name' 
   

3. This creates four files but we are only interested in the cluster userconfig file mywcl-userconfig.yml. We will extract the values and create a new cluster-config.yaml file in our version folder:

   cat mywcl-userconfig.yml | yq eval '.data.values' > bases/clusters/capa/v0.21.0/cluster_config.yaml
   

   The content of the file will be something like this:

   aws: {}
   bastion: {}
   clusterName: mywcl
   controlPlane:
     replicas: 3
   machinePools:
     machine-pool0:
       availabilityZones:
       - eu-central-1a
       instanceType: m5.xlarge
       maxSize: 10
       minSize: 3
       rootVolumeSizeGB: 300
   network:
     availabilityZoneUsageLimit: 3
   organization: myorg
   

4. Replace mywcl, myorg values from the previous step with variables:

   sed -i "s/myorg/${organization}/g" bases/clusters/capa/0.21.0/cluster_config.yaml
   sed -i "s/mywcl/${cluster_name}/g" bases/clusters/capa/0.21.0/cluster_config.yaml
   

5. Check cluster_config.yaml against the version-specific values.yaml, and tweak it if necessary to match the expected schema. At this point you may also provide extra configuration, like additional availability zones, node pools, etc. If you used kubectl gs template to get the values, this should be aligned with the latest version. If you were trying to create a different version, you might need to check proper values for that version.

6. Create a patch for the cluster App CR to provide the newly created configuration. For this create a file patch_config.yaml with this content:

   apiVersion: application.giantswarm.io/v1alpha1
   kind: App
   metadata:
     name: ${cluster_name}
     namespace: org-${organization}
   spec:
     extraConfigs:
       - kind: configMap
         name: ${cluster_name}-config
         namespace: org-${organization}
         priority: 55
   

7. Create the kustomization.yaml, referencing the template, and generating the ConfigMap out of cluster_config.yaml:

   apiVersion: kustomize.config.k8s.io/v1beta1
   configMapGenerator:
     - files:
       - values=cluster_config.yaml
       name: ${cluster_name}-config
       namespace: org-${organization}
   generatorOptions:
     disableNameSuffixHash: true
   kind: Kustomization
   patchesStrategicMerge:
     - patch_config.yaml
   resources:
     - ../template
   

With these steps you have created a new base, built upon the original template which you can use to create workload clusters with the kubectl gs GitOps add workload-cluster --base command as it is explained in the workload cluster creation tutorial.