Longhorn AWS S3 Backup


estimated read time: 5 minutes

Longhorn AWS S3 Backup Setup

Longhorn can use block storage provided by AWS S3. For an explanation of block storage, see IBM’s overview. Whilst replicas are excellent for protecting against the loss of a single node, if you’re as data-loss-conscious as I am, following the 3-2-1 Backup Rule is essential for any disaster recovery scenario.

Prerequisites

This article assumes you’ve already followed the steps in my article on setting up Longhorn on K3s.

Create an AWS IAM User

In the AWS Console, search for IAM and select it.

AWS Console IAM service with Users option selected

Click Add users in the top right corner.

AWS Console adding a new IAM user

Select Access key - Programmatic access, then click Next.

AWS Console add user to group

On the Add user to group screen, select Create group.

AWS Console create new user group menu

Select Create policy, then choose the JSON tab.

AWS Console custom policy screen with JSON tab selected

Enter the following policy. At this point, you’ll need to decide on a bucket name (see the Create Bucket section below):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "GrantLonghornBackupstoreAccess0",
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject", "s3:ListBucket", "s3:DeleteObject"],
      "Resource": ["arn:aws:s3:::<your-bucket-name>", "arn:aws:s3:::<your-bucket-name>/*"]
    }
  ]
}

Click Create, then download the user’s credentials CSV—you’ll need this later.

Create a KMS Key

In the AWS Console, search for KMS and select Key Management Service.

AWS Console creating a new KMS key

Select Create Key.

AWS Console KMS menu specifying an alias

Give the key a memorable alias, such as longhorn-s3-kms.

AWS Console KMS menu specifying administrative user

Select your AWS account administrator user.

AWS Console KMS menu specifying usage permissions

In Define key usage permissions, select the IAM user created in the previous step.

Click Next, review the request, then click Finish.

Create a Bucket

In the AWS Console, search for S3.

AWS Console S3 menu creating a bucket

Click Create bucket.

AWS Console S3 create bucket menu

Enter the bucket name you specified in the policy earlier.

AWS Console S3 create bucket public access settings

Ensure Block all public access is ticked.

Enable Encryption

AWS Console S3 encryption settings

Under Default encryption, tick Enable for server-side encryption.

Select AWS Key Management Service key (SSE-KMS) as the encryption key type.

Choose Choose from your AWS KMS keys and select the KMS key created earlier.

Intelligent Tiering (Optional)

I’ve included this section because I’m cost-conscious when it comes to cloud services. What starts as a few pounds a month can snowball quickly, especially with multi-gigabyte backups.

This isn’t a silver bullet, but it ensures misconfigured backup jobs are caught and archived data moves to the cheapest S3 tiers.

In the AWS Console, navigate to your S3 bucket and select the Properties tab.

AWS Console S3 properties

Scroll to Intelligent Tiering.

AWS Console S3 Intelligent Tiering configuration

Give the configuration a name and select This configuration applies to all objects in the bucket.

AWS Console S3 Intelligent Tiering archive rules

Under Archive rule actions:

Click Create.

Setting Up Kubernetes

Create a Kubernetes Secret for Longhorn

Before configuring Longhorn, we need to create a secret for AWS authentication.

Open the CSV file containing your IAM user credentials from earlier. Convert both values to Base64:

echo -n "<value>" | base64

Create both Base64 values for AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Create a file called longhorn-aws-secret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: aws-secret
  namespace: longhorn-system
data:
  AWS_ACCESS_KEY_ID: <IAM Access Key Base64 value>
  AWS_SECRET_ACCESS_KEY: <IAM Secret Access Key Base64 value>

Apply to your cluster:

kubectl apply -f longhorn-aws-secret.yaml

Configure Longhorn Settings

Access the Longhorn admin panel (see the Longhorn setup article).

From the web console, click Settings then General.

Longhorn dashboard settings button

Scroll to Backup Target and enter your S3 bucket path:

Longhorn backup settings

s3://<bucket-name>@<region>/

Note: Both the region and trailing / are required. Triple-check the bucket name and region to avoid debugging issues later.

In Backup Target Credentials Secret, enter aws-secret.

Click Save.

Running Your First Backup

From the Longhorn menu, click Volume.

Longhorn volume tab

Click on the PVC you want to back up.

Longhorn PVC details

Select Create Backup.

Create backup popup

Enter any custom tags. I typically use:

Depending on size, the backup may take a few hours. Once complete, check in the AWS Console.

Select your bucket and the Objects tab.

AWS S3 showing Longhorn backupstore folder

You should see the backupstore/ root directory.

Pi Cluster Series

Further Reading