Last modified May 30, 2018

Prepare an AWS account to run Giant Swarm guest clusters

As detailed in the Architecture docs, the guest clusters (the clusters running your Kubernetes workloads) in a Giant Swarm installation are running in an AWS account separate from the host cluster. This gives great flexibility depending on the requirements and the usage scenario. For example, it allows the host cluster to be running in an AWS account owned by Giant Swarm, while guest clusters operate in different AWS accounts each, depending on a customer’s department using them.

Overview

In order to run Giant Swarm guest clusters, an AWS account needs to fulfill these requirements:

  • Service limits set according to requirements
  • IAM role to be assumed by our aws-operator software
  • IAM role to be assumed by Giant Swarm staff

Each Giant Swarm guest cluster belongs to an organization within Giant Swarm. This organization will later be configured with information about the two IAM roles mentioned above.

Increase service limits in AWS

A number of limits apply to an AWS account initially, which are described in the AWS Service Limits documentation. The following overview lists the limits that have to be adjusted in order to use the account to operate Giant Swarm guest clusters.

Adjusting a service limit requires a support case in the AWS Support Center, where a specific entry form is provided for this type of case. Each limit type requires a separate case. When creating these, make sure to be logged in to the AWS account you want to adjust the limits for, and always select the correct region.

The screenshot below shows the entry form.

Screenshot

These are the limit increases to be requested, grouped by limit type:

  • VPC
    • VPCs per region: 50
    • NAT Gateway per Availability Zone per region: 50
  • Elastic IP
    • New VPC Elastic IP Address Limit per region: 50
  • Elastic Load Balancers
    • Application and Classic Load Balancers per region: 100
  • Auto Scaling
    • Auto Scaling Groups per region: 250
    • Launch Configurations per region: 500
  • EC2 Instances
    • m3.large per region: 250
    • m4.2xlarge per region: 250
    • m5.2xlarge per region: 250
    • other instance types to be used as workers: increase accordingly

When requesting a service limit increase, you will be asked for a description of your use case. You can use this text for the purpose:

We intend to run multiple Kubernetes clusters in this account, potentially used by various globally distributed teams. We will be creating and deleting new clusters frequently.

Each cluster needs its own VPC for security/isolation reasons and its own Elastic IP address for the NAT gateway.

Each cluster has at least 1 Auto Scaling Group, but can contain multiple ASGs if multiple instance types are requested as cluster nodes. If we count with 50 clusters with up to 5 EC2 instances as worker nodes each, we need up to 250 ASGs. For updating the ASGs in a rolling manner we need to duplicate the ASGs for a short time during update, hence the 500 Launch Configurations.

The number of EC2 instances used as worker nodes is supposed to be scaled dynamically based on traffic, hence the high numbers of EC2 instances requested.

Create an IAM role for aws-operator

Giant Swarm’s service creating and maintaining your guest clusters is called aws-operator. It is running in the host cluster. In order to handle resources in your AWS account, it needs to assume a prepared IAM role in your AWS account. Here we explain all the required steps to set up this role.

1. Determine the host cluster’s AWS account ID

First you need to know in which AWS account the Giant Swarm host cluster is (or will be) running. As Giant Swarm’s customer you might have already decided this yourself. Normally you yourself own this account.

Please have the ID of the account selected to run the host cluster at hand, as you will need it in step 2.

2. Basic role setup

First, log in to the AWS console for your AWS account. Then open the IAM section of the AWS console and go to the Roles subsection.

Now hit the Create role button. In the following screen, when asked to Select type of trusted entity chose Another AWS account.

AWS IAM console: Create role

The Account ID you enter is the ID of the AWS account as determined in step 1.

It is important that the Require external ID and Require MFA options remain unchecked!

Proceed to the next step to set up permissions.

3. Permissions setup

In the Attach permissions policies section, hit the Create policy button.

Paste the following JSON code into the JSON editor field and then hit the Review policy button.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:*",
                "cloudwatch:*",
                "cloudformation:*",
                "elasticloadbalancing:*",
                "events:*",
                "kms:*",
                "ec2:*",
                "ecr:*",
                "logs:*",
                "route53:*",
                "route53domains:*",
                "s3:*",
                "sts:AssumeRole",
                "sts:DecodeAuthorizationMessage",
                "sts:GetFederationToken",
                "trustedadvisor:*",
                "iam:AddRoleToInstanceProfile",
                "iam:AttachRolePolicy",
                "iam:CreatePolicyVersion",
                "iam:CreatePolicy",
                "iam:CreateInstanceProfile",
                "iam:CreateRole",
                "iam:DeletePolicyVersion",
                "iam:GetAccount*",
                "iam:GetRole",
                "iam:GetUser",
                "iam:GetRolePolicy",
                "iam:GetInstanceProfile",
                "iam:GetServiceLinkedRoleDeletionStatus",
                "iam:GetUserPolicy",
                "iam:GetUser",
                "iam:DeleteInstanceProfile",
                "iam:DeleteRole",
                "iam:DeletePolicy",
                "iam:DeleteRolePolicy",
                "iam:DetachRolePolicy",
                "iam:DeleteServiceLinkedRole",
                "iam:ListInstanceProfilesForRole",
                "iam:ListAttachedRolePolicies",
                "iam:ListAttachedUserPolicies",
                "iam:ListRolePolicies",
                "iam:ListPolicies",
                "iam:ListRoles",
                "iam:PassRole",
                "iam:PutRolePolicy",
                "iam:UpdateAssumeRolePolicy",
                "iam:UpdateRoleDescription",
                "iam:RemoveRoleFromInstanceProfile"
            ],
            "Resource": "*"
        }
    ]
}

In the next step you have to assign a name to the policy. Please use the name

GiantSwarmAWSOperatorPolicy

4. Attach policy to role

Once you created the policy, let’s turn back to the point in role creation called Attach permissions policies. Here you can now hit the Refresh button to load all existing policies, then enter GiantSwarmAWSOperatorPolicy into the search field to select the policy you just created. Check the box in the row containing that policy.

AWS IAM console: Attach policy

Then proceed to the next step.

5. Name the role

The last step of role creation requires you to set a name for the role. Please set the name to GiantSwarmAWSOperator.

AWS IAM console: Review

You may also set a description for team members to better understand the reasons for the existence of this role. It could be helpful to also paste a link to this guide into the field for reference.

6. Get the role’s ARN

After creating the new role, you should have a list of all IAM roles in front of you. From that list, open the GiantSwarmAWSOperator role you just created.

The role details screen shows the Role ARN, which is a unique identifier for the role. It should look like

arn:aws:iam::<YOUR_ACCOUNT_ID>:role/GiantSwarmAWSOperator

Please copy the exact ARN from the screen, as you will have to provide it to us later.

Create an IAM role for Giant Swarm staff

The second IAM role to be created is similar to the one before, but in this case it is used by Giant Swarm support staff. The main differences will be that this role must have Giant Swarm’s account as a trusted entity, instead of the account running the host cluster, and it can have multi-factor authentication enabled.

1. Basic role setup

  • Like above, go to the Roles subsection of the AWS console and select Create role. When asked to Select type of trusted entity chose Another AWS account.

  • In Account ID enter the value 084190472784.

  • As above, do not enable Require external ID.

  • Different from above, instead, we strongly recommended to check the option Require MFA (multi factor authorization). This adds an extra authentication step for users to assume the role, which increases security.

2. Permission setup

Select Create policy to create another policy. Use the same JSON policy code as before. This time, call the policy

GiantSwarmAdminPolicy

Background: We ask you to create two distinct, but identical, IAM roles at this point. This enables you to later adjust permissions independently if the need arises.

3. Attach policy to role

Attach the new GiantSwarmAdminPolicy policy to the role you are creating.

4. Name the role

Name this role

GiantSwarmAdmin

accordingly.

5. Get the role’s ARN

From the conformation screen, copy the exact ARN again. It should be in the form of

arn:aws:iam::<YOUR_ACCOUNT_ID>:role/GiantSwarmAdmin

Configure the Giant Swarm organization

In the previous sections, we explained how to create two IAM roles in the AWS account that’s going to run the Giant Swarm guest clusters.

Giant Swarm guest clusters are owned by organizations, which allows to control access to clusters, since only members of the owner organization have access to the management functions of a cluster.

In order to run a guest cluster in your AWS account, the organization owning your cluster has to know about the roles you just created.

If you have direct access to the Giant Swarm API, please set the credentials of your organization via the API instead. See the documentation of the POST /v4/organizations/{org}/credentials/ endpoint for details.

In case you work with a Giant Swarm partner, it might be that you don’t have access to the Giant Swarm API. In that case, please hand over the role ARNs for the GiantSwarmAWSOperator role and the GiantSwarmAdmin role to your partner contact.

Further reading