Longhorn AWS S3 Backup Setup
Longhorn has the capability to use block storage provided by AWS S3, for an explanation of block storage have a look here. Although replicas are great to ensure you are covered in the case of losing one node, if you’re as data-loss-conscious as myself then sticking to the 3-2-1 Backup Rule to protect against the worse case scenario is essential part of any disaster recovery scenario.
Prerequisites
This article assumes you’ve already followed the steps described in my first article for setting up longhorn on k3s.
Create AWS IAM User
From AWS console, in the search type iam then select it.
Next, click on the add users button in the top right corner.
Click the Access key - Programmatic access button.
Click Next
On the Add user to group screen select Create group.
Next we need to select create policy
Select the Json tab.
enter the following in the policy text area, it is at this point you will have to decide what you are going to name your bucket. (see create bucket section later on.)
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "GrantLonghornBackupstoreAccess0",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:ListBucket",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::<your-bucket-name>",
"arn:aws:s3:::<your-bucket-name>/*"
]
}
]
}
Click Create.
Now download the users credentials csv. (you will need this later).
Create KMS Key
Back on the search bar in the AWS Console, search for kms and select Key Management Service
Select Create Key
Give the key a memorable alias, for example longhorn-s3-kms
Select the user you use to administrate your AWS account.
in the Define key usage permissions select the iam user you created in the previous step.
Click Next, Once you’ve reviewed the request click Finish
Create Bucket
Now go back to the search bar, this time type S3.
Click on Create bucket
Now enter the name of the bucket you decided earlier whilst writing the policy.
Ensure Block all public access is ticked.
Enable Encryption
Under default encryption and further under server-side encryption tick Enable.
Select the Encryption key type of AWS Key Management Service key (SSE-KMS)
Select Choose from your AWS KMS keys
in the Available AWS KMS keys dropdown box select the KMS key we created earlier.
Intelligent Tiering (Optional)
I’ve included this as I’m very cost aware when it comes to cloud services and their pay-as-you-go pricing model. What can start out being a couple of dollars a month can snowball quickly, especially if have multi gigabytes backing up. This is not a silver bullet but it just ensures any misconfigured backup jobs are caught-out and that such backups get put into the cheapest tiers offered by AWS for S3 archiving.
From the AWS Console Search for s3. Click on the name of the bucket you created
Select the Properties tab.
Scroll down to the Intelligent Tiering plane.
in Configuration name give a name to the policy.
Select This configuration applies to *all* objects in the bucket
Under Archive rule actions
Select the Archive Access tier and set the days to the minimum number of days which at the time of writing is 90.
Select the Deep Archive Access tier and set the days to the minimum number of days which at the time of writing is 180.
Finally, click Create.
Setup Kubernetes
Create Kubernetes Secret for Longhorn
Before we can configure Longhorn, we first need to make the secret available that will allow it to authenticate with AWS S3.
From the earlier step Create AWS IAM User the open the csv file with the new users credentials init.
For the secret to work in K8s we first much convert these values to base64.
Todo this on Linux/WSL/MacOS you can use the following command:
echo -n "<value>" | base64
We will need to create base64 values for both the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
Create a new file called longhorn-aws-secret.yaml and enter the following:
apiVersion: v1
data:
AWS_ACCESS_KEY_ID: <IAM Access Key Base64 value>
AWS_SECRET_ACCESS_KEY: <IAM Secret Access Key Base64 value>
kind: Secret
metadata:
name: aws-secret
namespace: longhorn-system
now we can apply this to your cluster with:
kubectl apply -f longhorn-aws-secret.yaml
Configure Longhorn Settings
You will need to access the Longhorn admin panel see setting up Longhorn above.
From the Longhorn web console, click Settings and then General
Now, scroll down until you find Backup Target.
in the box enter the path to your S3 bucket name
s3://<bucket-name>@<region>/
*Note* Both the region and the final */* are required to be correct.
It’s worth triple checking both the bucket name and region to avoid later debugging the issue.
Next, in Backup Target Credentials Secret enter the name of the secret we created earlier aws-secret.
Now scroll down and click Save.
Run Your First Backup
From the Longhorn Menu click Volume.
Click on the PVC you want to backup.
Select Create Backup
Enter any custom tags you want. I normally supply two:
APP <name of the app>
REASON <type of backup, DR, SCHEDULE etc>
Depending on the size, it could be a couple of hours before it is complete, however once it is done. You can check in AWS Console.
Select your bucket and select the Objects tab.
You should see the root directory of ‘backupstore/’