Adding Direct S3 Uploads

This walkthrough shows how to add asynchronous uploads to a Roda & Sequel app, though the instructions are equally applicable to a Rails & Active Record app. The flow will go like this:

1. User selects file(s)
2. For each file a request is made to a presign endpoint to fetch AWS S3 upload parameters
3. Files are uploaded asynchronously to AWS S3
4. Uploaded file JSON data is written to a hidden field
5. Form is submitted instantaneously as it only has to submit the JSON data
6. JSON data is assigned to the Shrine attachment attribute (instead of the raw file)

AWS S3 setup

You'll need to create an AWS S3 bucket, which is where the uploads will be stored. See this walkthrough on how to do that.

Next you'll need to configure CORS for that bucket, so that it accepts uploads directly from the client. In the AWS S3 Console go to your bucket, click on the "Permissions" tab and then on "CORS configuration". There paste in the following:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <CORSRule>
    <AllowedOrigin>https://my-app.com</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
    <AllowedHeader>Authorization</AllowedHeader>
    <AllowedHeader>x-amz-date</AllowedHeader>
    <AllowedHeader>x-amz-content-sha256</AllowedHeader>
    <AllowedHeader>content-type</AllowedHeader>
  </CORSRule>
  <CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
  </CORSRule>
</CORSConfiguration>

Replace https://my-app.com with the URL to your app (in development you can set this to *). Once you've hit "Save", it may take some time for the new CORS settings to be applied.

Installation

Add Shrine and aws-sdk-s3 to the Gemfile:

# Gemfile

gem "shrine", "~> 2.11"
gem "aws-sdk-s3", "~> 1.2"

Configuration

Create an initializer that will be loaded when your app boots, and replace placeholders with the actual credentials of your S3 bucket.

# config/shrine.rb

require "shrine"
require "shrine/storage/s3"

s3_options = {
  access_key_id:     "<YOUR_ACCESS_KEY_ID>",
  secret_access_key: "<YOUR_SECRET_ACCESS_KEY>",
  bucket:            "<YOUR_BUCKET>",
  region:            "<YOUR_REGION>",
}

Shrine.storages = {
  cache: Shrine::Storage::S3.new(prefix: "cache", **s3_options),
  store: Shrine::Storage::S3.new(**s3_options),
}

Shrine.plugin :sequel # load integration for the Sequel ORM
Shrine.plugin :cached_attachment_data # for forms
Shrine.plugin :restore_cached_data # refresh metadata when attaching the cached file

Uploader

Create an uploader for the types of files you'll be uploading:

# uploaders/image_uploader.rb

class ImageUploader < Shrine
end

Now add an attachment attribute to your model:

# models/article.rb

class Article < Sequel::Model
  include ImageUploader::Attachment.new(:cover_photo)
end

You'll also need to add the <attachment>_data text or JSON column to that table:

Sequel.migration do
  change do
    add_column :articles, :cover_photo_data, :text
  end
end

View

In your model form you can now add form fields for the attachment attribute, and an image tag for the preview:

<div class="form-group">
  <input type="hidden" name="article[cover_photo]" value="<%= @article.cached_cover_photo_data %>" class="upload-hidden">
  <input type="file" name="article[cover_photo]" class="upload-file">
</div>
<img class="upload-preview">

The file field will be used for choosing files, and the hidden field for storing uploaded file data and retaining it across form redisplays in case of validation errors.

You should now be able to upload the images via the form, and display them in your views:

<img src="<%= @article.cover_photo_url %>" width=500>

Direct upload

We can now add asynchronous direct uploads to the mix. We'll be using a JavaScript file upload library called Uppy.

Direct file uploads to S3 from the browser work in the following way:

1. User selects the file
2. On client side we fetch request params and URL for the S3 upload from the app
3. Using this information we upload the file to S3

Presign endpoint

So, on the server side we'll need to add an endpoint which returns valid request params and URL for the S3 upload. Shrine's presign_endpoint plugin bakes this functionality in, all we need to do is load the plugin and mount the endpoint to the desired path:

# config/shrine.rb

Shrine.plugin :presign_endpoint, presign_options: -> (request) {
  # Uppy will send the "filename" and "type" query parameters
  filename = request.params["filename"]
  type     = request.params["type"]

  {
    content_disposition:    "inline; filename=\"#{filename}\"", # set download filename
    content_type:           type,                               # set content type (defaults to "application/octet-stream")
    content_length_range:   0..(10*1024*1024),                  # limit upload size to 10 MB
  }
}

# For Roda app
route do |r|
  r.on "s3/params" do
    r.run Shrine.presign_endpoint(:cache)
  end
  # ...
end


# For Rails app (config/routes.rb)
Rails.application.routes.draw do
  mount Shrine.presign_endpoint(:cache) => "/s3/params"
  # ...
end

We've mounted it to /s3/params because that's the same path at which Uppy Companion mounts its own endpoint, and it's the one that Uppy will call by default.

Uppy

Now we can setup Uppy to do the direct uploads. First we'll pull in the necessary JavaScript and CSS files:

<!DOCTYPE html>
<html>
  <head>
    <script src="https://unpkg.com/[email protected]/dist/polyfill.min.js"></script>
    <script src="https://unpkg.com/[email protected]/fetch.js"></script>
    <script src="https://transloadit.edgly.net/releases/uppy/v0.28.0/dist/uppy.min.js"></script>

    <link href="https://transloadit.edgly.net/releases/uppy/v0.28.0/dist/uppy.min.css" rel="stylesheet" />
  </head>

  <body>
    ...
  </body>
</html>

Now we can add the following JavaScript code which will perform direct uploads to S3 when the user selects the file, using Shrine's presign endpoint, assigning the results to the hidden attachment field to be submitted:

function fileUpload(fileInput) {
  var imagePreview = document.querySelector('.upload-preview')

  fileInput.style.display = 'none' // uppy will add its own file input

  var uppy = Uppy.Core({
      id: fileInput.id,
      autoProceed: true,
    })
    .use(Uppy.FileInput, {
      target: fileInput.parentNode,
    })
    .use(Uppy.Informer, {
      target: fileInput.parentNode,
    })
    .use(Uppy.ProgressBar, {
      target: imagePreview.parentNode,
    })

  uppy.use(Uppy.AwsS3, {
    serverUrl: '/', // will call Shrine's presign endpoint on `/s3/params`
  })

  uppy.on('upload-success', function (file, data) {
    // show image preview
    imagePreview.src = URL.createObjectURL(file.data)

    // construct uploaded file data in the format that Shrine expects
    var uploadedFileData = JSON.stringify({
      id: file.meta['key'].match(/^cache\/(.+)/)[1], // object key without prefix
      storage: 'cache',
      metadata: {
        size:      file.size,
        filename:  file.name,
        mime_type: file.type,
      }
    })

    // set hidden field value to the uploaded file data so that it's submitted with the form as the attachment
    var hiddenInput = fileInput.parentNode.querySelector('.upload-hidden')
    hiddenInput.value = uploadedFileData
  })

  return uppy
}

document.querySelectorAll('.upload-file').forEach(function (fileInput) {
  fileUpload(fileInput)
})

And that's it, now when a file is selected it will be asynchronously uploaded directly to your S3 bucket. During the upload a nice progress bar will be displayed, and when the upload finishes an image preview will be shown.

If you're dealing with larger files, you can make the uploads resumable by using the AwsS3Multipart Uppy plugin instead, with uppy-s3_multipart gem on the backend.

See Also

• Adding Direct App Uploads
• Authenticating Endpoints
• Using Checksums in Direct Uploads
• uppy-s3_multipart