If you don’t use Shrine, you are doing the wrong way to upload files.

You create a page with a new, long form filled with fields. At the end of the form is a file upload field where users need to choose to upload 1.2GB of documents. After pressing submit and waiting for exhaustion for 30 minutes, what caught the user’s eye is: ” the title must not be longer than 20 characters “. Users revise the title and submit the form again, waiting 30 minutes in bitterness, without knowing that what waits for them behind is another validation error …

Implementing a file upload feature has a lot of complicated cases like that. And the example above is just one of many problems that Shrine – a new and small Ruby library – solves completely.

In this article, I want to share a little about how to design – the wise decisions that the Shrine creator has carefully selected when developing his library, and how this library helps to develop character. How simple and interesting the file upload feature is!

Storage and promotion

Shrine allows you to define lots of storage as you like. It can simply be saved to a filesystem, or saved on Amazon S3, Google Cloud Storage or even Flickr. If you have any other hosting service (eg fshare), you can easily write a plugin storage for it.

By default, Shrine has 2 types of storage available, and you must set “storage” in advance for it:

Why save files in 2 places?

One is the cache , a storage location for any files uploaded, may or may not have passed validation, and has not been saved to a model yet. Once the model has successfully saved into the database with the file, that file will be transferred to the storage store . The process of moving from storage cache to store is called promotion process.

Shrine fully supports the promotion process as a background job . And not just simply moving the file, you can also perform many “cool” things during the promotion process (extract metadata, create different versions of the file).

Database

Shrine supports other types of ORM not only ActiveRecord, but also many other ORMs like Sequels, ROMs, etc.But let’s talk about it, now let’s find out in advance how Shrine stores data about uploaded files to the facility. data like.

Problem

You are developing a system for photographers, allowing users to freely share photos. At first when designing the database, you thought that simply creating a table with the picture_path field (saving the path / link to that image) is enough.

After going into operation for a few days, you realize that the image uploaded by the user is too heavy (5MB or more), and you need to create a version that has resized the resolution and optimized the size to display. OK, create another column named picture_optimized_path and picture_thumbnail_path .

On another day, your website got bigger and had bandwidth problems, and you decided to create a .webp version for your photos, to get the best size. So you have to create migration again, add 2 more columns picture_optimized_webp_path and picture_thumbnail_webp_path .

Then one day you want to save the information about size, EXIF, the main color of the image. You must create 3 more columns. Gradually, the board pictures of you look not unlike a battlefield.

Solution of the Shrine

Shrine realized this problem very early: file-related data is extremely flexible in nature , and how to store data individually is not a viable solution. Instead, the author of Shrine chose to add a jSON field to the table, which means that in the case above, you only need a single JSON field named picture_data . JSON type is something that many popular relational databases like mariaDB or postgreSQL support, give good query performance, and are suitable for storing highly flexible data types.

The JSON data saved by the Shrine looks like this:

In this JSON field, Shrine stores everything related to the file: file versions, sizes, file types, other metadata information, etc.

Uploaders

The uploader is just pure Ruby classes, inside it defines how the file is validated, the metadata to be extracted, the file versions (derivatives) to create, etc.

The above uploader can then be included in any model:

You can create many uploaders: ImageUploader to validate and resize images, VideoUploader contains processing functions, video encode, … Through the uploader, we can reuse the upload, validate, and file handling logic of Uploader on many different models.

Derivatives

Shrine supports the ability to create different versions of the same file. Shrine selected authors call it the derivatives (derivatives).

Typically, you will want to create derivatives during the promotion process – that is, when the record has been saved to the database, and the file is moved from the cache to the store . Creating derivatives can be done in the background job , or execute on-the-fly .

Imagine a site like Netflix, with each episode, people have to create hundreds of different file versions , with all sizes, and corresponding to each size is the format format for the systems. Shrine helps you to manage so many file versions so easily. You can also group (nested) derivatives together for convenient management.

For example, when you want to create 3 versions: small, medium, large for an uploaded file, define it first in an Uploader :

Upload the file, call the command to create derivatives and save the model:

Later, you can get any derivative you desire:

Optimize the upload experience

Form upload retain

Thanks to always caching the data that the user has uploaded, Shrine can help keep the files that the user has uploaded, in case the model is not successfully created (validate error, …). For simple forms, Shrine has cached_attachment_data plugin cached_attachment_data makes it easy for you to do this.

Direct Upload

With the traditional upload method, the file is only uploaded when the user clicks the submit button. Today, with support from the frontend framework, the user experience will be a lot better if we upload the file as soon as the user selects it, and then simply “assign” the uploaded file reference to the form. when the user submits. This is called the Direct Upload technique.

Shrine supports direct uploads to both the file system and AWS S3 and many other types of storage. Setting up Direct Upload in Shrine is very simple. First, add a route for the direct upload:

Then, when you want to direct upload, you just need to send POST to / images / upload with the data form of the file you want to upload. That file will be uploaded to cache storage using ImageUploader. A JSON format representing the uploaded file will be sent back:

The above JSON string can be attached to the submission form as usual. When the form is submitted and the model is saved, the file is automatically attached to the model and begins to promote.

Resumable upload

When users often have to upload large files of 1-2GB or more, traditional file uploads are also very risky. When the connection to the server is unstable or the user loses the network, the upload process will have to start from the beginning. But Shrine is willing to accept and support uploading via the Tus protocol , a protocol that makes file upload process resumable.

Clear cache storage

The direct upload method causes the files to be regularly cached, regardless of whether or not the file is assigned to the model. So what if the storage cache gradually piles up redundant files and takes up disk space?

Shrine won’t help you, but simply scheduling via cron to regularly clean up cache storage is enough. The bias in the cron / schedule task may be as follows:

Conclusion

Above are just a few of the Shrine’s solutions to solve file upload problems for systems. While it’s only a library for Ruby and Ruby’s web frameworks, there’s a lot you can learn from Shrine when designing any system with upload-file storage as the focus.

If you use Ruby or the Ruby on Rails framework and are interested in learning more about Shrine, see the Shrine documentation at https://shrinerb.com/docs/getting-started .