Last modified December 13, 2022

Using persistent volumes on AWS with EFS

If your cluster is running in the cloud on Amazon Web Services (AWS) the most common way to store data is using EBS volumes with the dynamic provisioner. Sometimes EBS is not the optimal solution.

The advantages of using EFS over EBS are:

EFS data can be accessed from all availability zones in the same region while EBS is tied to a single availability zone.
EFS has the capability to mount the same persistent volume to multiple pods at the same time using the ReadWriteMany access mode.
EFS will not hit the AWS Instance Volume Limit as it is a software mount and will avoid the Impaired EBS issue.
EFS mount times are better than EBS.
EFS provides encryption in transit support using TLS and it’s enabled by default.

If you need to use EFS to provision volumes, be advised:

All Kubernetes persistent volumes will be stored in the same EFS instance. You can deploy multiple provisioners per cluster, each having its own storage-class and EFS instance.
EFS throughput need to be set up accordingly in order to not have performance issues. We only recommend Provisioned Throughput, and if you need high performance you will need EBS.
EFS backups are done with AWS Backup and it does not have the snapshot feature of EBS.
You cannot limit the amount of data stored in an EFS volume. The requested value in Kubernetes is ignored.
EFS mount targets are limited to 1 subnet per availability zone. Each NodePool will create a different subnet per AZ, plan accordingly.

Provision an EFS instance on AWS

Note: Currently only static provisioning is supported. This means an AWS EFS file system needs to be created manually on AWS first. After that it can be mounted inside a container as a volume using the driver.

Before installing the provisioner in Kubernetes we will need to create the EFS instance in the same AWS account:

Open the AWS management console for the AWS account holding your cluster resources.
From the Services menu, select EFS.
Create a new file system and select the VPC where your cluster is located. The VPC can be identified by your cluster ID.
Select the availability zone with the subnets your instances are located on and the security groups of your node pools. Subnets and security groups can be identified by you cluster ID.
Choose the throughput and performance mode. Setting a file system policy or access points is optional.
Create the instance and note the EFS instance ID.
In case of dynamic provisioning, you need to add the roles to the instance profile or use service account annotation in controller to enable access to AWS EFS.

Installing the EFS CSI driver

To install the EFS CSI driver in the workload cluster, you will need to follow these steps:

Access the web interface and select the cluster on which you want to install the EFS CSI driver.
Open the Apps tab.
Click the Install App button
Select the Giant Swarm Playground catalog.
Select the App named aws-efs-csi-driver.
Click the Configure & Install button. Make sure that the correct cluster is selected.
Create and submit a configuration values file in case you want to deploy the storage class and controller (by default disable to allow a smooth transitation to CSI driver)

Example configuration:

storageClasses:
- name: efs-sc
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  mountOptions:
  - tls
  parameters:
    provisioningMode: efs-ap
    fileSystemId: fs-92107666
    directoryPerms: "700"
    gidRangeStart: "1000"
    gidRangeEnd: "2000"
    
controller:
  create: true

Click the Install App button.

Deploy a sample application

In order to verify that the EFS CSI driver works as expected, we suggest to deploy a test workload based on the following manifests.

First we apply a persistent volume claim with a capacity of 5 GB storage.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: efs-claim
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: efs-sc
  resources:
    requests:
      storage: 5Gi

And finally, we create a pod which repeatedly writes the current date into a file located in our EFS storage mount:

apiVersion: v1
kind: Pod
metadata:
  name: efs-app
spec:
  containers:
  - name: linux
    image: amazonlinux:2
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo $(date -u) >> /efs-data/out.txt; sleep 5; done"]
    volumeMounts:
    - name: efs-storage
      mountPath: /efs-data
  volumes:
  - name: efs-storage
    persistentVolumeClaim:
      claimName: efs-claim

Warning: By default, new EFS file systems are owned by root:root. You might need to change file system permissions if your container is not running as root. To learn about exposing separate data stores with independent ownership and permissions, check the AWS guide on working with Amazon EFS Access Points.

Using persistent volumes on AWS with EFS

Provision an EFS instance on AWS

Installing the EFS CSI driver

Deploy a sample application

Further reading

Need help, got feedback?