django-cuddlybuddly-storage-s3

Updated Amazon S3 storage from django-storages. Adds more fixes than I can remember, a metadata cache system and some extra utilities for dealing with MEDIA_URL and HTTPS, CloudFront and for creating signed URLs.

Installation

1. Add cuddlybuddly.storage.s3 to your INSTALLED_APPS.
2. Set DEFAULT_FILE_STORAGE to cuddlybuddly.storage.s3.S3Storage (as a string, don't import it).
3. Set MEDIA_URL to your bucket URL , e.g. http://yourbucket.s3.amazonaws.com/.
4. Enter your AWS credentials in the settings below.

Settings

AWS_ACCESS_KEY_ID

Your Amazon Web Services access key, as a string.

AWS_SECRET_ACCESS_KEY

Your Amazon Web Services secret access key, as a string.

AWS_STORAGE_BUCKET_NAME

Your Amazon Web Services storage bucket name, as a string.

AWS_HEADERS

A list of regular expressions which if matched add the headers to the file being uploaded to S3. The patterns are matched from first to last:

# see http://developer.yahoo.com/performance/rules.html#expires
AWS_HEADERS = [
    ('^private/', {
        'x-amz-acl': 'private',
        'Expires': 'Thu, 15 Apr 2000 20:00:00 GMT',
        'Cache-Control': 'private, max-age=0'
    }),
    ('.*', {
        'x-amz-acl': 'public-read',
        'Expires': 'Sat, 30 Oct 2010 20:00:00 GMT',
        'Cache-Control': 'public, max-age=31556926'
    })
]

• x-amz-acl sets the ACL of the file on S3 and defaults to private.
• Expires is for old HTTP/1.0 caches and must be a perfectly formatted RFC 1123 date to work properly. django.utils.http.http_date can help you here.
• Cache-Control is HTTP/1.1 and takes precedence if supported. max-age is the number of seconds into the future the response should be cached for.

AWS_CALLING_FORMAT

Optional and defaults to SUBDOMAIN. The way you'd like to call the Amazon Web Services API, for instance if you need to use the old path method:

from cuddlybuddly.storage.s3 import CallingFormat
AWS_CALLING_FORMAT = CallingFormat.PATH

CUDDLYBUDDLY_STORAGE_S3_GZIP_CONTENT_TYPES

A list of content types that will be gzipped. Defaults to ('text/css', 'application/javascript', 'application/x-javascript').

CUDDLYBUDDLY_STORAGE_S3_SKIP_TESTS

Set to a true value to skip the tests as they can be pretty slow.

CUDDLYBUDDLY_STORAGE_S3_SYNC_EXCLUDE

A list of regular expressions of files and folders to ignore when using the synchronize commands. Defaults to ['\.svn$', '\.git$', '\.hg$', 'Thumbs\.db$', '\.DS_Store$'].

HTTPS

Because when you use S3 your MEDIA_URL must be absolute (i.e. it starts with http) it's more difficult to have URLs that match how the page was requested. The following things should help with that.

cuddlybuddly.storage.s3.middleware.ThreadLocals

This middleware will ensure that the URLs of files retrieved from the database will have the same protocol as how the page was requested.

cuddlybuddly.storage.s3.context_processors.media

This context processor returns MEDIA_URL with the protocol matching how the page was requested.

Cache

Included is a cache system to store file metadata to speed up accessing file metadata such as size and the last modified time. It is disabled by default.

FileSystemCache

The only included cache system is FileSystemCache that stores the cache on the local disk. To use it, add the following to your settings file:

CUDDLYBUDDLY_STORAGE_S3_CACHE = 'cuddlybuddly.storage.s3.cache.FileSystemCache'
CUDDLYBUDDLY_STORAGE_S3_FILE_CACHE_DIR  = '/location/to/store/cache'

Custom Cache

To create your own cache system, inherit from cuddlybuddly.storage.s3.cache.Cache and implement the following methods:

• exists
• modified_time
• save
• size
• remove

Utilities

create_signed_url(file, expires=60, secure=False)

Creates a signed URL to file that will expire in expires seconds. If secure is set to True an https link will be returned.

To import it:

from cuddlybuddly.storage.s3.utils import create_signed_url

CloudFrontURLs(default, patterns={}, https=None)

Use this with the above context processor to return varying MEDIA_URLS depending on the path to improve page loading times.

To use it add something like the following to your settings file:

from cuddlybuddly.storage.s3.utils import CloudFrontURLs
MEDIA_URL = CloudFrontURLs('http://cdn1.example.com/', patterns={
    '^images/': 'http://cdn2.example.com/',
    '^banners/': 'http://cdn3.example.com/',
    '^css/': 'http://cdn4.example.com/'
    }, https='https://example.s3.amazonaws.com/')

s3_media_url Template Tag

This is for use with CloudFrontURLs and will return the appropriate URL if a match is found.

Usage:

{% load s3_tags %}
{% s3_media_url 'css/common.css' %}

For HTTPS, the cuddlybuddly.storage.s3.middleware.ThreadLocals middleware must also be used.

Commands

cb_s3_sync_media

Synchronizes a directory with your S3 bucket. It will skip files that are already up to date or newer in the bucket but will not remove old files as that has the potential to go very wrong. The headers specified in AWS_HEADERS will be applied.

It has the following options:

• --cache, -c - Get the modified times of files from the cache (if available) instead of checking S3. This is faster but could be inaccurate.
• --dir, -d - The directory to synchronize with your bucket, defaults to MEDIA_ROOT.
• --exclude, -e - A comma separated list of regular expressions to ignore files or folders. Defaults to CUDDLYBUDDLY_STORAGE_S3_SYNC_EXCLUDE.
• --force, -f - Uploads all files even if the version in the bucket is up to date.
• --prefix, -p - A prefix to prepend to every file uploaded, i.e. a subfolder to place the files in.

cb_s3_sync_static

Exactly the same as cb_s3_sync_media except that dir defeaults to STATIC_ROOT.

A note on the tests

The tests in tests/s3test.py are pretty much straight from Amazon but have a tendency to fail if you run them too often / too quickly. When they do this they sometimes leave behind files or buckets in your account that you will need to go and delete to make the tests pass again.

The signed URL tests will also fail if your computer's clock is too far off from Amazon's servers.