# `Coelho.Document`
[🔗](https://github.com/nseaSeb/coelho/blob/main/lib/coelho/document.ex#L1)

Validation, normalisation and plain text extraction of documents.

A document is a plain map tree with string keys, exactly as it comes out
of `Jason.decode/1` or out of ProseMirror's `toJSON()`:

    %{
      "type" => "doc",
      "content" => [
        %{
          "type" => "paragraph",
          "content" => [
            %{"type" => "text", "text" => "hello", "marks" => [%{"type" => "bold"}]}
          ]
        }
      ]
    }

No struct wraps it, so what is validated is what is stored, and a jsonb
round trip is the identity.

## Two boundaries, not one

`validate/2` is the boundary at the keyboard, and it is strict on purpose:
an unknown node type, an unknown mark, an unknown attribute or an attribute
failing its validator all reject the document, and say where. Nothing
outside the schema reaches the database, so rendering never has to escape
its way out of untrusted markup. This is what storing the document buys
over storing HTML and filtering tags on the way in.

`sanitize/2` is the boundary at the screen. Stored documents are *not*
re-validated when they are read, so a row written under a looser schema, or
by a direct SQL write, is not covered by the paragraph above. Put it
through `sanitize/2` before rendering it anywhere a reader will see.

## Validation is also normalisation

What comes back from `validate/2` is canonical: the same rich text always
produces the same document, byte for byte, which is what makes `hash/2`
worth storing.

  * marks are sorted into the schema's declaration order, not the order the
    editor happened to add them in
  * an attribute left at its schema default is not written out, so two
    editors that disagree on whether to send `align: "left"` still store
    the same thing
  * adjacent text nodes carrying the same marks are merged

A renderer therefore reads attributes with the schema default in hand —
`Coelho.Render.attr/3` is the one place that knows the shape.

## Untrusted input

`validate/2` is the boundary a hostile document hits first, so it is
written to survive one: node type names are resolved against the schema
rather than converted to atoms, the schema's `:limits` are checked before
anything is allocated, and error paths are accumulated in reverse so that
validating a deep document stays linear in its size.

## What is not a document

`nil`, `""` and any other non-map are rejected with a single error,
`expected an object`, on the empty path. `%{}` is rejected with
`missing "type"`. None of them raise, and none of them are quietly treated
as the empty document — `Coelho.empty/1` is how you ask for that. Casting
a form field is the one place where an empty string means "no document",
and `Coelho.Ecto.Type` and `Coelho.Ash.Type` handle it there, before
validation.

# `blank?`

```elixir
@spec blank?(term(), Coelho.Schema.t()) :: boolean()
```

Whether a document would put anything on the page.

What an application asks before deciding to render a block at all — a
portal panel, an announcement, a set of opening hours — where an empty
document should mean the block is not there rather than a heading with
nothing under it.

The obvious stand-in, `text_length(document) == 0`, is wrong, and wrong in
the direction that loses content: a document holding one image, or one
attachment, has no text and is very much not blank. So the question is put
to the schema instead — a node it declares `void: true` renders an element
of its own and counts, whatever text it has none of.

    Coelho.blank?(page.intro_doc, MyApp.RichText.schema())

Blank means: no text anywhere, and no void node with anything to show.
Empty paragraphs and empty lists are blank, however many of them there are,
and so is a paragraph of nothing but hard breaks — an inline void node
declaring no attributes is punctuation between words, which is what a
pasted-then-emptied field usually leaves behind. An image or an attachment
has a source to point at, and a horizontal rule draws a line; all three
count.

This is a narrower question than the one `hash/2` answers with `nil`, which
is "was there anything to agree to" and needs no schema. The two differ
only on a document whose whole content is an attribute-less void node — a
horizontal rule and nothing else is blank to `hash/2` and not blank here,
because it does put a line on the page.

# `canonical`

```elixir
@spec canonical(term()) :: binary()
```

A byte-for-byte stable serialisation of a document.

Two documents describing the same rich text serialise identically, which
is what `hash/2` needs and what a plain JSON encoding cannot promise: map
key order is not part of a map, and `jsonb` reorders keys of its own
accord.

Three things make it stable, and all three are already true of a document
`validate/2` returned:

  * object keys are emitted in sorted order
  * marks are in the schema's declaration order, not the order the editor
    added them
  * attributes left at their schema default are absent, not written out

Which is why this must be given a **validated** document. Serialising what
came back from the database instead — where a `jsonb` round trip has
reordered the keys and an older writer may have spelled the defaults out —
answers a different question, and answers it differently on two rows that
hold the same text.

# `hash`

```elixir
@spec hash(term(), :sha256 | :sha512 | :sha384 | :sha224 | :sha) :: String.t() | nil
```

The hex digest of `canonical/1`, or `nil` for a document with nothing in it.

What makes a proof of acceptance hold: store the digest of the terms the
reader agreed to, and a later document that hashes the same is the same
document, whatever the editor or the database did to the key order in
between.

    iex> document = %{"type" => "doc", "content" => [
    ...>   %{"type" => "paragraph", "content" => [%{"type" => "text", "text" => "hi"}]}
    ...> ]}
    iex> {:ok, document} = Coelho.validate(document)
    iex> Coelho.Document.hash(document)
    "00dc4439f0dcbb463ab186b5b8f81b68e50d70a7b1e3538b86a13e532a17a65d"

A document is *empty* when it holds no text and no node carrying
attributes, and `nil` is what every such document answers — an empty
paragraph, a top node with no children, a document whose only content is a
horizontal rule, three hard breaks in a row, an empty bullet with an empty
item. They are different documents and they draw different pages; as
digests they are one thing, which is "nothing was agreed to".

So `nil == nil` is not "the same document": an application comparing a
stored digest with a fresh one has to decide what an absent digest means
before it compares. Any document with a word in it has a digest of its
own, and no empty one can collide with it.

Hash a validated document, for the reason `canonical/1` gives.

# `sanitize`

```elixir
@spec sanitize(term(), Coelho.Schema.t(), keyword()) :: map()
```

Turns any term into a document the schema accepts, without failing.

`validate/2` is the boundary at the keyboard: it says no, and says where.
This is the boundary at the screen. Stored documents are not re-validated
when they are read — `Coelho.Ecto.Type` deliberately trusts the column —
so a row written under a looser schema, by a direct SQL write, or by a
version of the application that has since tightened its vocabulary, would
otherwise reach a public page unchecked.

Nothing is reported and nothing is raised: what falls outside the schema is
removed, and what is left is a document `validate/2` accepts. A hostile
document becomes a poor document, never an unexpected rendering.

What removal means, from the gentlest repair to the harshest:

  * a key the schema does not know is dropped
  * an attribute failing its validator is dropped, so the schema default
    applies — a heading claiming `level: 99` renders as a level 1 heading
  * a mark that is unknown, not allowed here, or whose own attributes fail
    is dropped, and the text it covered stays — a link with a
    `javascript:` href becomes plain text
  * a node whose type is unknown, or whose content cannot satisfy its
    content expression, is dropped whole, along with the text inside it
  * a document over the schema's bounds is **cut to fit** them: text past
    `:max_text_length` is truncated, nodes past `:max_nodes` are cut off,
    and what is nested deeper than `:max_depth` is dropped — the rest of
    the document stays either way
  * a document that cannot be repaired at all becomes `Coelho.empty/1`

A document stamped with another schema version is repaired against this
schema and restamped with its version, rather than refused the way
`validate/2` refuses it. Rendering a document written under an older
vocabulary badly beats rendering it as nothing; migrating it properly is
`Coelho.migrate/2`.

It is idempotent: a document that already validates comes back normalised
and unchanged, and sanitising twice is sanitising once.

    Coelho.Document.sanitize(row.body, MyApp.RichText.schema())
    |> Coelho.Render.to_html(MyApp.RichText.schema())

## Bounds this call does not want

`:limits` overrides the schema's for this call alone. A bound is a bound
on *writing* — the browser posts into a hidden field no `maxlength`
constrains, which is what `:max_text_length` is there to refuse — and
reading is a different question: what is already stored is stored, and
cutting it to fit on the way to the page loses text nobody asked to lose.

    Coelho.Document.sanitize(row.body, schema,
      limits: [max_text_length: :infinity, max_attr_length: :infinity]
    )

So an application that wants the structure cleaned and the length left
alone says so here, instead of keeping a second schema per field to
sanitise against. Judging the length itself is then its own to do, with
the whole document in hand to do it on.

`:max_attr_length` belongs in that list for the same reason and bites
differently: an attribute over the bound is *dropped*, and a **required**
attribute dropped takes its node with it — a stored image whose `src` is
longer than the bound disappears from the page rather than being shortened.
A row written before the bound existed, or under a schema that set it
higher, is exactly the case to lift it for.

# `text_length`

```elixir
@spec text_length(term()) :: non_neg_integer()
```

The number of characters a writer typed.

This is the concatenation of the text nodes, nothing else: no bullet, no
blank line between paragraphs, no filename standing in for an attachment.
`to_text/2` materialises all of those because full text search wants them,
and a length counted on its result rejects a document the editor still
shows as under the limit — with nothing on screen to explain the gap.

The browser half counts the same way, so the editor's counter and the
server's check agree on the number.

# `to_text`

```elixir
@spec to_text(map(), Coelho.Schema.t()) :: String.t()
```

Extracts the plain text of a document, for full text search.

Bullets are materialised, blocks are separated, and a node with a
`:to_text` in its spec contributes whatever that says — an attachment its
caption or its filename, a hard break a newline. What comes out reads like
the document, which is what a search index wants and what
`text_length/1` deliberately does not count.

## Indexing it

A `jsonb` document is not searchable as it stands: an index over it can
answer "does this key exist", never "does this say *tomato*". The text has
to become a column of its own, written when the document is:

    # migration
    alter table(:posts) do
      add :body_text, :text
    end

    create index(:posts, ["body_text gin_trgm_ops"], using: :gin)

    # changeset
    def changeset(post, attrs) do
      post
      |> cast(attrs, [:body])
      |> put_body_text()
    end

    defp put_body_text(changeset) do
      case fetch_change(changeset, :body) do
        {:ok, document} -> put_change(changeset, :body_text, Coelho.to_text(document))
        :error -> changeset
      end
    end

Derived at write time and not read time, because the alternative is
extracting the text of every row on every search. A generated column would
do as well where the database can call out to nothing — PostgreSQL cannot
run this from SQL, so the application writes it.

Two things follow. The column is a *derivative*, so it is never the source:
a migration that changes what `to_text/2` produces means rewriting it, the
same way any denormalisation does. And it holds no markup at all, which is
what makes `to_tsvector` and trigram search behave — indexing rendered HTML
matches on `strong` and `href`.

# `validate`

```elixir
@spec validate(term(), Coelho.Schema.t()) ::
  {:ok, map()} | {:error, [Coelho.Document.Error.t()]}
```

Validates and normalises a document against a schema.

Returns the normalised document, or every error found. Paths in the
errors read from the root, as in `content[0].attrs.href`.

---

*Consult [api-reference.md](api-reference.md) for complete listing*