@spec blank?(term(), Coelho.Schema.t()) :: boolean()

Whether a document would put anything on the page.

What an application asks before deciding to render a block at all — a portal panel, an announcement, a set of opening hours — where an empty document should mean the block is not there rather than a heading with nothing under it.

The obvious stand-in, text_length(document) == 0, is wrong, and wrong in the direction that loses content: a document holding one image, or one attachment, has no text and is very much not blank. So the question is put to the schema instead — a node it declares void: true renders an element of its own and counts, whatever text it has none of.

Coelho.blank?(page.intro_doc, MyApp.RichText.schema())

Blank means: no text anywhere, and no void node with anything to show. Empty paragraphs and empty lists are blank, however many of them there are, and so is a paragraph of nothing but hard breaks — an inline void node declaring no attributes is punctuation between words, which is what a pasted-then-emptied field usually leaves behind. An image or an attachment has a source to point at, and a horizontal rule draws a line; all three count.

This is a narrower question than the one hash/2 answers with nil, which is "was there anything to agree to" and needs no schema. The two differ only on a document whose whole content is an attribute-less void node — a horizontal rule and nothing else is blank to hash/2 and not blank here, because it does put a line on the page.

@spec canonical(term()) :: binary()

A byte-for-byte stable serialisation of a document.

Two documents describing the same rich text serialise identically, which is what hash/2 needs and what a plain JSON encoding cannot promise: map key order is not part of a map, and jsonb reorders keys of its own accord.

Three things make it stable, and all three are already true of a document validate/2 returned:

• object keys are emitted in sorted order
• marks are in the schema's declaration order, not the order the editor added them
• attributes left at their schema default are absent, not written out

Which is why this must be given a validated document. Serialising what came back from the database instead — where a jsonb round trip has reordered the keys and an older writer may have spelled the defaults out — answers a different question, and answers it differently on two rows that hold the same text.

@spec hash(term(), :sha256 | :sha512 | :sha384 | :sha224 | :sha) :: String.t() | nil

The hex digest of canonical/1, or nil for a document with nothing in it.

What makes a proof of acceptance hold: store the digest of the terms the reader agreed to, and a later document that hashes the same is the same document, whatever the editor or the database did to the key order in between.

iex> document = %{"type" => "doc", "content" => [
...>   %{"type" => "paragraph", "content" => [%{"type" => "text", "text" => "hi"}]}
...> ]}
iex> {:ok, document} = Coelho.validate(document)
iex> Coelho.Document.hash(document)
"00dc4439f0dcbb463ab186b5b8f81b68e50d70a7b1e3538b86a13e532a17a65d"

A document is empty when it holds no text and no node carrying attributes, and nil is what every such document answers — an empty paragraph, a top node with no children, a document whose only content is a horizontal rule, three hard breaks in a row, an empty bullet with an empty item. They are different documents and they draw different pages; as digests they are one thing, which is "nothing was agreed to".

So nil == nil is not "the same document": an application comparing a stored digest with a fresh one has to decide what an absent digest means before it compares. Any document with a word in it has a digest of its own, and no empty one can collide with it.

Hash a validated document, for the reason canonical/1 gives.

@spec sanitize(term(), Coelho.Schema.t(), keyword()) :: map()

Turns any term into a document the schema accepts, without failing.

validate/2 is the boundary at the keyboard: it says no, and says where. This is the boundary at the screen. Stored documents are not re-validated when they are read — Coelho.Ecto.Type deliberately trusts the column — so a row written under a looser schema, by a direct SQL write, or by a version of the application that has since tightened its vocabulary, would otherwise reach a public page unchecked.

Nothing is reported and nothing is raised: what falls outside the schema is removed, and what is left is a document validate/2 accepts. A hostile document becomes a poor document, never an unexpected rendering.

What removal means, from the gentlest repair to the harshest:

• a key the schema does not know is dropped
• an attribute failing its validator is dropped, so the schema default applies — a heading claiming level: 99 renders as a level 1 heading
• a mark that is unknown, not allowed here, or whose own attributes fail is dropped, and the text it covered stays — a link with a javascript: href becomes plain text
• a node whose type is unknown, or whose content cannot satisfy its content expression, is dropped whole, along with the text inside it
• a document over the schema's bounds is cut to fit them: text past :max_text_length is truncated, nodes past :max_nodes are cut off, and what is nested deeper than :max_depth is dropped — the rest of the document stays either way
• a document that cannot be repaired at all becomes Coelho.empty/1

A document stamped with another schema version is repaired against this schema and restamped with its version, rather than refused the way validate/2 refuses it. Rendering a document written under an older vocabulary badly beats rendering it as nothing; migrating it properly is Coelho.migrate/2.

It is idempotent: a document that already validates comes back normalised and unchanged, and sanitising twice is sanitising once.

Coelho.Document.sanitize(row.body, MyApp.RichText.schema())
|> Coelho.Render.to_html(MyApp.RichText.schema())

Bounds this call does not want

:limits overrides the schema's for this call alone. A bound is a bound on writing — the browser posts into a hidden field no maxlength constrains, which is what :max_text_length is there to refuse — and reading is a different question: what is already stored is stored, and cutting it to fit on the way to the page loses text nobody asked to lose.

Coelho.Document.sanitize(row.body, schema,
  limits: [max_text_length: :infinity, max_attr_length: :infinity]
)

So an application that wants the structure cleaned and the length left alone says so here, instead of keeping a second schema per field to sanitise against. Judging the length itself is then its own to do, with the whole document in hand to do it on.

:max_attr_length belongs in that list for the same reason and bites differently: an attribute over the bound is dropped, and a required attribute dropped takes its node with it — a stored image whose src is longer than the bound disappears from the page rather than being shortened. A row written before the bound existed, or under a schema that set it higher, is exactly the case to lift it for.

@spec text_length(term()) :: non_neg_integer()

The number of characters a writer typed.

This is the concatenation of the text nodes, nothing else: no bullet, no blank line between paragraphs, no filename standing in for an attachment. to_text/2 materialises all of those because full text search wants them, and a length counted on its result rejects a document the editor still shows as under the limit — with nothing on screen to explain the gap.

The browser half counts the same way, so the editor's counter and the server's check agree on the number.

@spec to_text(map(), Coelho.Schema.t()) :: String.t()

Extracts the plain text of a document, for full text search.

Bullets are materialised, blocks are separated, and a node with a :to_text in its spec contributes whatever that says — an attachment its caption or its filename, a hard break a newline. What comes out reads like the document, which is what a search index wants and what text_length/1 deliberately does not count.

Indexing it

A jsonb document is not searchable as it stands: an index over it can answer "does this key exist", never "does this say tomato". The text has to become a column of its own, written when the document is:

# migration
alter table(:posts) do
  add :body_text, :text
end

create index(:posts, ["body_text gin_trgm_ops"], using: :gin)

# changeset
def changeset(post, attrs) do
  post
  |> cast(attrs, [:body])
  |> put_body_text()
end

defp put_body_text(changeset) do
  case fetch_change(changeset, :body) do
    {:ok, document} -> put_change(changeset, :body_text, Coelho.to_text(document))
    :error -> changeset
  end
end

Derived at write time and not read time, because the alternative is extracting the text of every row on every search. A generated column would do as well where the database can call out to nothing — PostgreSQL cannot run this from SQL, so the application writes it.

Two things follow. The column is a derivative, so it is never the source: a migration that changes what to_text/2 produces means rewriting it, the same way any denormalisation does. And it holds no markup at all, which is what makes to_tsvector and trigram search behave — indexing rendered HTML matches on strong and href.

@spec validate(term(), Coelho.Schema.t()) ::
  {:ok, map()} | {:error, [Coelho.Document.Error.t()]}

Validates and normalises a document against a schema.

Returns the normalised document, or every error found. Paths in the errors read from the root, as in content[0].attrs.href.