Where the bytes of an attachment live.
Coelho stores the document and the attachment metadata; the file itself is
somebody else's problem, and this is the smallest contract that lets it be
solved without writing the same glue in every application. A storage is a
struct whose module implements these callbacks, so an application can point
at the one it ships with — Coelho.Storage.Disk — or write its own for
object storage without anything else changing.
storage = Coelho.Storage.Disk.new("priv/uploads")
:ok = Coelho.Storage.put(storage, key, {:file, upload_path})
{:ok, path} = Coelho.Storage.path(storage, key)Keys come from Coelho.Attachment.generate_key/1. They are opaque and URL
safe, and a storage must treat them as untrusted: Coelho.Storage.Disk
refuses a key that is not what it hands out, so a key cannot walk out of
the directory it belongs to.
Writing one for object storage
Coelho ships Coelho.Storage.Disk and nothing else, deliberately: an S3
adapter means an HTTP client and a signing library, and a package whose
document core has no dependencies at all should not acquire two so that
applications not using object storage can carry them. Here is the whole
thing, against ExAws, to copy into an
application and adjust:
defmodule MyApp.Uploads.S3 do
@behaviour Coelho.Storage
defstruct [:bucket]
def new(bucket), do: %__MODULE__{bucket: bucket}
@impl true
def put(%{bucket: bucket}, key, {:binary, binary}) do
bucket |> ExAws.S3.put_object(key, binary) |> request()
end
def put(%{bucket: bucket}, key, {:file, path}) do
path
|> ExAws.S3.Upload.stream_file()
|> ExAws.S3.upload(bucket, key)
|> request()
end
@impl true
def read(%{bucket: bucket}, key) do
case bucket |> ExAws.S3.get_object(key) |> ExAws.request() do
{:ok, %{body: body}} -> {:ok, body}
{:error, {:http_error, 404, _response}} -> {:error, :enoent}
{:error, reason} -> {:error, reason}
end
end
# No local path: the plug falls back to read/2, or to the redirect
# below, which is the one worth having.
@impl true
def path(_storage, _key), do: :error
@impl true
def delete(%{bucket: bucket}, key) do
bucket |> ExAws.S3.delete_object(key) |> request()
end
@impl true
def exists?(%{bucket: bucket}, key) do
match?({:ok, _response}, bucket |> ExAws.S3.head_object(key) |> ExAws.request())
end
@impl true
def redirect_url(%{bucket: bucket}, key, opts) do
filename = Keyword.get(opts, :filename)
ExAws.Config.new(:s3)
|> ExAws.S3.presigned_url(:get, bucket, key,
expires_in: Keyword.fetch!(opts, :expires_in),
query_params:
[{"response-content-type", Keyword.fetch!(opts, :content_type)}] ++
if(filename,
do: [{"response-content-disposition", ~s(inline; filename="#{filename}")}],
else: []
)
)
end
defp request(operation) do
case ExAws.request(operation) do
{:ok, _response} -> :ok
{:error, reason} -> {:error, reason}
end
end
endTwo things to know before it goes to production.
put/3 must stream a file, not read it. The {:file, path} clause
above uses ExAws.S3.Upload, which does a multipart upload; the obvious
shortcut, File.read!(path) into put_object, holds the whole upload in
memory per request.
ExAws over HTTP/2 fails above about a megabyte, with
{:error, :send_buffer_full} — the request outgrows the connection's send
buffer and nothing retries it. A naive adapter meets it the first time
someone attaches a real PDF, and it looks like a Coelho problem rather than
a transport one. Pin the client to HTTP/1.1, or raise the buffer:
config :ex_aws, :hackney_opts, protocols: [:http1]redirect_url/3 is the difference between a proxy and a redirect.
Without it, Coelho.Plug.Attachments falls back to read/2 and every byte
of every attachment travels through the application. With it, the plug
checks its signature and then gets out of the way — and the presigned URL
has to pin the :content_type it is given, because the headers the plug
would have set do not survive a redirect. That is not decoration: it is
what stops a file that lies about what it is from being rendered as
whatever the bucket decides.
Summary
Callbacks
Removes the bytes. Removing what is not there is not an error.
Whether the storage holds anything under this key.
A local path for the bytes, when there is one.
Stores the bytes under a key, replacing whatever was there.
Reads the bytes back.
Somewhere the reader can fetch the bytes directly, when there is such a place.
The bytes in pieces, for a storage that can read them that way.
Functions
Asks the storage for a URL to redirect to, or :error when it has none.
The bytes in pieces, or :error where the storage cannot answer that way.
Types
Callbacks
Removes the bytes. Removing what is not there is not an error.
Whether the storage holds anything under this key.
A local path for the bytes, when there is one.
Lets a plug send the file rather than read it into memory. A remote
storage answers :error, and the caller falls back to read/2.
Stores the bytes under a key, replacing whatever was there.
Reads the bytes back.
Somewhere the reader can fetch the bytes directly, when there is such a place.
Object storage can hand out a URL of its own — presigned, short lived —
and answering with one is what stops every byte travelling through the
application. Coelho.Plug.Attachments redirects to it after checking its
own signature, so the check still happens and the transfer does not.
opts carries:
:expires_in— the seconds left on the signature that got the reader this far. A URL outliving it would widen the window the signature was there to narrow.:content_type— what the application recorded for this file. An implementation is expected to pin it, through whatever its service offers —response-content-typeon a presigned S3 URL, and the like. The plug's own defence against a file that lies about what it is lives in headers a redirect does not carry, so an implementation that passes this over hands that defence back to whatever the bucket decides.:filename— for a service that can pin a download name too.
Optional: a storage that has no such URL — the local filesystem — simply does not implement it.
@callback stream(t(), key()) :: {:ok, Enumerable.t()} | :error
The bytes in pieces, for a storage that can read them that way.
read/2 answers with the whole object on the heap of whichever process
asked, so N concurrent downloads hold N copies of it. A storage that
implements this hands Coelho.Plug.Attachments an enumerable instead and
the response is chunked, which is what keeps a large object from being a
large allocation per reader.
Optional. Without it the plug falls back to read/2, and a storage with a
local path never reaches either — path/2 is served by send_file, which
does not read the bytes into the VM at all.
Answer :error for a key this storage has nothing for, or cannot tell
apart from one it refuses: the caller then asks read/2, whose
{:error, :invalid_key} is what says 403 rather than 404. A lazy
enumerable handed back for a key that is not there sends a 200 with an
empty body instead, because the status is gone by the time the first piece
is asked for.
Functions
Asks the storage for a URL to redirect to, or :error when it has none.
Answers :error for a storage that does not implement the callback, so
callers need not know which do.
@spec stream(t(), key()) :: {:ok, Enumerable.t()} | :error
The bytes in pieces, or :error where the storage cannot answer that way.
Answers :error rather than raising for a storage that does not implement
the optional callback, so a caller asks once and falls back.