Server backup to S3 without having AWSAccessKey and AWSSecretKey on server

Backups are good! ( multiple servers dotted around with different backup solutions I wanted to consolidate on one consistent backup solution utilising S3 for the actual storage.

While researching using S3 for storage it is clear that the usual principle requires your AWSAccessKey and AWSSecretKey to be copied onto each server where uploading to S3 is required.

But as Amazon states...

IMPORTANT: Your Secret Access Key is a secret, and should be known only by you and AWS. You should never include your Secret Access Key in your requests to AWS. You should never e-mail your Secret Access Key to anyone. It is important to keep your Secret Access Key confidential to protect your account.

 Ideally I wanted a solution that didn't require the AWSAccessKey and AWSSecretKey to be copied onto every server requiring backup, increasing the likelihood of the AWS account being compromised if any server became compromised.

The solution to this comes from the Browser based uploads using POST.

This is designed to allow visitors to your web site to upload content directly to your S3 account with out going through your server and without you passing any credentials to the web browser.

It has two features that allow us to use this for server backup.

  1. A signature that is generated from the AWSAccessKey and AWSSecretKey
  2. An expiration that can be set to a long time in the future

So the solution is to created a signature that allows uploads to a bucket that doesn't expire until well into the future. Then use curl to POST to S3 using the signature.

Generating the correct policy document and curl parameters to make this happen took some time. So in case anyone would like to do this here's a bit of Ruby code to generate the curl command line...

require 'base64'
require 'openssl'
require 'digest/sha1'

aws_access_key_id = '*** AWS ACCESS KEY ***'
aws_secret_key = '*** AWS SECRET KEY'
content_type = 'application/octet-stream'
bucket = '** BUCKET ***'
acl = 'private'
key_prefix = '***FOLDER****/'

policy_document = '{
  "expiration": "2012-01-01T12:00:00.000Z",
  "conditions": [
    {"bucket": "' + bucket + '" },
    {"acl": "' + acl + '" },
    ["starts-with", "$key", ""],
    ["starts-with", "$Content-Type", ""],

policy = Base64.encode64(policy_document).gsub("\n","")

signature = Base64.encode64(
        aws_secret_key, policy)

print 'curl '
print "-F 'key=#{key_prefix}${filename}' "
print "-F AWSAccessKeyId=#{aws_access_key_id} "
print "-F acl=#{acl} "
print "-F policy=#{policy} "
print "-F signature=#{signature} "
print "-F Content-Type=#{content_type} "
print "-F file=@FILENAMEHERE "
print "-F Submit=OK "
print "http://#{bucket}"
print "\n"

Just fill in the variables with the required information and run the script.

The curl command line generated will then upload any file to S3. Just replace FILENAMEHERE in the command line with the required filename (leave the @ before the filename).

Also, rate limiting is possible with curl. For example --limit-rate 20K will keep curl to only using 20K/s of your bandwidth. Handy for stopping the backup using the full bandwidth of your connection.