Shrine Direct S3 Upload results in octet-stream mime-type

[Aurel-Allard]

After following this tutorial, we set up a Direct Upload using Shrine and Uppy on S3.

During our tests, a pdf file is well uploaded on S3 in the cache folder via a Drag&Drop box on our app. The problem is that when we assign it to our Doc model, two things happened:

• The Doc object created do not take the mime_type into account in its file_data column, and instead, labels it as application/octet-stream.
• Consequently (I think), the cached file on S3 do not get promoted.

Our setup seems to be ok, as it used to work just a month ago. Still, we cannot figure it out after many tries and ivestigations... Please note that our S3 CORS and ENV setup is ok, and that there do not seem to be any problems with authorizations/access.

shrine.rb

require "shrine/storage/s3"

s3_options = {
  access_key_id:     ENV['AWS_ACCESS_KEY_ID'],
  secret_access_key: ENV['AWS_SECRET_ACCESS_KEY'],
  region:            ENV['AWS_REGION'],
  bucket:            ENV['AWS_BUCKET'],
}

Shrine.storages = {
  cache: Shrine::Storage::S3.new(prefix: "cache", **s3_options), # For direct uploads
  store: Shrine::Storage::S3.new(public: true, prefix: "store", **s3_options)
}

Shrine.plugin :activerecord
Shrine.plugin :cached_attachment_data
Shrine.plugin :restore_cached_data
Shrine.plugin :determine_mime_type
Shrine.plugin :backgrounding
Shrine.plugin :remove_invalid
Shrine.plugin :presign_endpoint, presign_options: lambda { |request|
  filename = request.params['filename']
  type     = request.params['type']
{
    content_disposition: "inline; filename=\"#{filename}\"",
    content_type: type,
    content_length_range: 0..(15 * 1024 * 1024)
  }
}

file_uploader.rb

class FileUploader < Shrine
  require "content_disposition"

  plugin :validation
  plugin :validation_helpers
  plugin :derivatives

  Attacher.validate do
    validate_max_size 9.megabyte, message: 'Votre fichier ne doit pas dépasser 9Mo.'
  end

  plugin :default_storage, store: -> {
    org = record.documentable_type.constantize.find(record.documentable_id).organization
    if record.documentable_type == 'Organization'
      org = record.documentable_type.constantize
      "store_organizations/#{record.documentable_id}/documents"
    elsif record.documentable_type == 'Entity'
      "store_entities/#{record.documentable_id}/documents"
    elsif record.documentable_type == 'Project'
      "store_projects/#{record.documentable_id}/documents"
    end
  }

  plugin :dynamic_storage
  storage /store_(.+)/ do |match|
    bucket_name = "#{match[1]}"
    Shrine::Storage::S3.new(bucket: ENV["AWS_BUCKET"], prefix: bucket_name)
  end
end

upload_controller.js (Uppy part)

[...]

this.uppy
      .use(DragDrop, {
        target: '.file-dropzone .for-DragDrop',
        }
      })

// Load to S3 cache folder -> OK
      .use(AwsS3, {
        companionUrl: this.urlValue,
        allowedMetaFields: [
          'name'
        ]
      })

// Construct uploaded file data in the format that Shrine expects
      .on('upload-success', function (file, data) {
        var filename = file.name
        var uploadedFileData = JSON.stringify({
          id: file.meta['key'].match(/^cache\/(.+)/)[1], // object key without prefix
          storage: 'cache',
          metadata: {
            size: file.size,
            filename: file.name,
            mime_type: "application/pdf" // file.type
          }
        })

// docs_controller#direct_upload
        fetch(postUrl, {
          method: "POST",
          credentials: "same-origin",
          body: JSON.stringify({
            filename: filename,
            file: uploadedFileData,
          }),
          headers: {
            'X-CSRF-Token': document.querySelector('meta[name="csrf-token"]').content,
            'Content-Type': 'application/json'
          },
        })
      })

[...]

doc.rb

class Doc < ApplicationRecord
  include ActionView::Helpers::TagHelper
  include FileUploader::Attachment.new(:file)
end

docs_controller.rb

[...]

  def direct_upload
    # binding.pry
    @doc = Doc.new(
      name: params[:filename],
      file: params[:file],
    )
  end

[...]

The doc_controller binding.pry leads to those params[:file]: "{\"id\":\"dd5acef2cd5f7303309406308c849720.pdf\",\"storage\":\"cache\",\"metadata\":{\"size\":1180288,\"filename\":\"Demande + certificat.pdf\",\"mime_type\":\"application/pdf\"}}"

But once assigned to the Doc object we got octet-stream in file_data column:
file_data: "{\"id\":\"dd5acef2cd5f7303309406308c849720.pdf\",\"storage\":\"cache\",\"metadata\":{\"size\":1180288,\"filename\":\"Demande + certificat.pdf\",\"mime_type\":\"application/octet-stream\"}}"

We are completly lost after trying all Shrine plugins, etc, etc...

Many thanks for your help!

[janko]

The restore_cached_data plugin will re-extract the metadata when attaching the cached file, disregarding the metadata included in the file data. This is recommended because the user could have tampered with that metadata, so it cannot be trusted.

The determine_mime_type plugin will determine the MIME type from file content, using file command by default, and that will get saved to mime_type. In your case, my bet is that the file command failed to correctly detect the file type as PDF, so it returned the generic application/octet-stream (which is the same as saying "this is some kind of byte stream", it's what's returned when MIME type couldn't be determined).

I would recommend trying a different MIME type analyzer, see the docs for more details.

[Aurel-Allard]

Thank @janko.

We added this part to the uploader :

  plugin :metadata_attributes, filename: :name, size: :size, mime_type: :type
  plugin :determine_mime_type, analyzer: :marcel, analyzer_options: { filename_fallback: true }
  plugin :refresh_metadata

It works as we now have the right mime_type.
Still, the allocation from cache to regular folder does not work. Would you have any clue?

We've been struggling with this for days now.

[janko]

In what way promoting from temporary to permanent storage doesn't work? The MIME type gets reset back to application/octet-stream? Could you come up with a self-contained example that reproduces the problem you're experiencing? There is a template in CONTRIBUTING.md.

[Annedj]

Hi @janko 👋

I work with @Aurel-Allard, our initial issue with MIME type is now fine, as the file uploaded keeps the right type:

[1] pry(main)> Doc.last.file
=> #<FileUploader::UploadedFile storage=:cache id="0c767e034478015bc62c80ddf6dc8b01.pdf" metadata={"size"=>29894, "filename"=>"94b8f427-6efc-47a3-ac78-cfa675e10d2a", "mime_type"=>"application/pdf"}>

Our main issue now is that the document uploaded to S3 does not leave the cache folder to get promoted to the actual folder.
So Doc.last.file.storage.prefix stays cache. When adding a debugger inside both default_storage and dynamic_storage plugins, we do enter each of them and get the expected result, but the prefix does not seem to be used when the file is saved.

Our FileUploader and Shrine initializers at the moment:

# file_uploader.rb
class FileUploader < Shrine
  require "content_disposition"

  plugin :validation
  plugin :validation_helpers
  plugin :derivatives

  plugin :metadata_attributes, filename: :name, size: :size, mime_type: :type
  plugin :determine_mime_type, analyzer: :marcel, analyzer_options: { filename_fallback: true }
  plugin :refresh_metadata

  Attacher.validate do
    validate_max_size 9.megabyte, message: 'Votre fichier ne doit pas dépasser 9Mo.'
    # validate_extension %w[pdf]
    # validate_mime_type %w[application/pdf]
  end

  Attacher.derivatives do |original|
  end

  plugin :default_storage, store: -> {
    if record.documentable_type == 'Organization'
      "store_organizations/#{record.documentable_id}/documents"
    elsif record.documentable_type == 'Entity'
      "store_entities/#{record.documentable_id}/documents"
    elsif record.documentable_type == 'Project'
      "store_projects/#{record.documentable_id}/documents"
    end
  }

  plugin :dynamic_storage
  storage /store_(.+)/ do |match|
    Shrine::Storage::S3.new(bucket: ENV["AWS_BUCKET"], prefix: match[1])
  end
end

# shrine.rb
require "shrine/storage/s3"

s3_options = {
  access_key_id:     ENV['AWS_ACCESS_KEY_ID'],
  secret_access_key: ENV['AWS_SECRET_ACCESS_KEY'],
  region:            ENV['AWS_REGION'],
  bucket:            ENV['AWS_BUCKET'],
}

Shrine.storages = {
  cache: Shrine::Storage::S3.new(prefix: "cache", **s3_options), # For direct uploads
  store: Shrine::Storage::S3.new(public: true, prefix: "store", **s3_options)
}

Shrine.plugin :activerecord
Shrine.plugin :cached_attachment_data
Shrine.plugin :restore_cached_data
Shrine.plugin :determine_mime_type
Shrine.plugin :backgrounding
Shrine.plugin :remove_invalid
Shrine.plugin :presign_endpoint, presign_options: lambda { |request|
  filename = request.params['filename']
  type     = request.params['type']
  {
    content_disposition: "inline; filename=\"#{filename}\"",
    content_type: type,
    content_length_range: 0..(15 * 1024 * 1024)
  }
}

We're thinking that we might be missing a plugin or a method indicating we want our file to use the Storage specified in the dynamic_storage plugin, but we are struggling to find any similar examples of that config.

Thanks for your help!

[janko]

@Annedj Were you able to reproduce it in a self-contained script? I don't know why the cached file doesn't get promoted to permanent storage in your case, there is no plugin configuration that should prevent this 🤷🏻‍♂️