# `Grammar.VM`
[🔗](https://github.com/joetjen/ichor/blob/v0.7.0/lib/grammar/vm.ex#L1)

The interpreted runtime backend: compiles an `Aether.Grammar` to
bytecode and runs it against real input. No compile-time codegen step
-- give it a grammar (from `Aether.Parser.parse/2` +
`Grammar.Analysis.run/1`) whenever you have one, including one your
own program only learns about at runtime (a user-supplied grammar, a
plugin, a REPL's `:load`). See the tutorial's "Loading a grammar at
runtime" section for a full worked example, and "Which path is right
for you?" for how this compares to `use Ichor`/`Mix.Tasks.Ichor.Gen`.

Two genuinely separate compiled programs are involved, matching
Aether's own "every grammar compiles to a Lexer feeding a Parser, never
a scannerless single pass" design: `Grammar.VM.CharCompiler` +
`Grammar.VM.Tokenizer` turn the input string into a token stream (maximal
munch over every declared token); `Grammar.VM.RuleCompiler` +
`Grammar.VM.TokenInterpreter` then run the rules over *that* stream,
never over raw characters -- building a raw capture tree as they go.

`parse/3` is a bare recognizer (does `grammar.root` match `input`, full
stop) with no `Ichor.Actions` involved. `run/4` is the actions-aware
entry point: it matches, then hands the raw capture tree to
`Ichor.Actions.evaluate/6` for a given Actions module and initial
context, producing the grammar's actual result (a sandboxed program's
value, a config map, ...) instead of just a yes/no. `run_sequence/4`
is for a source that's a *sequence* of top-level matches against
`grammar.root` rather than one single match spanning the whole input
(loading a standard-library file one top-level form at a time, each one
threading context into the next, is the motivating case) -- most
grammars only ever need `run/4`.

`context` reaches the *match* phase too now (`parse/3`'s third
argument, `run/4`'s `initial_context`, `run_sequence/4`'s
per-form-accumulated context for *rule*-level matching), read-only --
the one thing that ever consults it there is a `Grammar.IR.Custom`
`@native(...)` node, via `c:Ichor.CustomRule.match/4`, or a
`Grammar.IR.CustomLexeme` one via `c:Ichor.CustomLexeme.scan/3`. Nothing
else in `Grammar.VM.TokenInterpreter`/`Grammar.VM.Tokenizer` reads it; only
`Ichor.Actions.evaluate/6` ever produces a *new* one.

One real gap for `CustomLexeme` specifically: `run_sequence/4`
tokenizes the *entire* input once, up front, with only
`initial_context` -- unlike rule-level matching (re-run per top-level
form, seeing that form's own accumulated context), a token that reads
`context` can't see anything a *later* form's own evaluation produced.
Fine for heredocs/string-interpolation (context there is either unused
or fixed for the whole file); a hazard only for something
`\catcode`-like that needs re-tokenization as context evolves
mid-sequence -- not something `run_sequence/4` supports today.

# `parse`

```elixir
@spec parse(Aether.Grammar.t(), String.t(), term()) ::
  {:ok, non_neg_integer()} | {:error, Ichor.Error.t()}
```

Compiles `grammar` and matches it against `input`, requiring the root
rule to consume the entire (tokenized) input. Returns the number of
tokens consumed on success -- a bare recognizer result, no
`Ichor.Actions` involved.

# `parse_lossless`

```elixir
@spec parse_lossless(Aether.Grammar.t(), String.t(), term()) ::
  {:ok, Ichor.Syntax.Node.t()} | {:error, Ichor.Error.t() | [Ichor.Error.t()]}
```

Like `parse/3`, but returns the lossless tree of `input`
(`Ichor.Syntax`): every byte of it, trivia and punctuation included,
under every rule that matched. Fails exactly as `parse/3` fails.

# `run`

```elixir
@spec run(Aether.Grammar.t(), String.t(), module(), Ichor.Actions.context()) ::
  {:ok, term()} | {:error, Ichor.Error.t() | [Ichor.Error.t()]}
```

Like `parse/2`, but runs the match through `actions_module` (a
`Ichor.Actions` implementation) starting from `initial_context`,
returning the grammar's actual evaluated result.

# `run_lossless`

```elixir
@spec run_lossless(Aether.Grammar.t(), String.t(), module(), Ichor.Actions.context()) ::
  {:ok, term(), Ichor.Syntax.Node.t()}
  | {:error, Ichor.Error.t() | [Ichor.Error.t()]}
```

Like `run/4`, returning the lossless tree of `input` beside the value:
`{:ok, value, tree}`, `value` being exactly what `run/4` returns.

# `run_sequence`

```elixir
@spec run_sequence(Aether.Grammar.t(), String.t(), module(), Ichor.Actions.context()) ::
  {:ok, [term()], Ichor.Actions.context()}
  | {:error, Ichor.Error.t() | [Ichor.Error.t()]}
```

Evaluates `input` as a sequence of top-level matches against
`grammar.root`, one after another (skipping only the grammar's own
`@skip` trivia between them, the same as it would between any two
ordinary tokens), threading context from each into the next. Returns
every top-level value, in order, plus the final context.

# `token_descriptions`

```elixir
@spec token_descriptions(Aether.Grammar.t()) :: %{required(atom()) => String.t()}
```

How an error names each anonymous token: by its literal text, quoted,
where it is a literal. Every other token is named by its own name.

Public because `Grammar.Native` writes the same names into a generated
parser, so the two backends report a failure in the same words.

# `tokenize`

```elixir
@spec tokenize(Aether.Grammar.t(), String.t(), Ichor.Actions.context()) ::
  {:ok, [Grammar.VM.Token.t()]} | {:error, Ichor.Error.t()}
```

`input`'s token stream under `grammar` -- interpreted, and for any
engine, since every engine lexes the same way.

What a generated module's own `tokenize/2` answers: every token in
order, trivia included, `@keywords`/`@refine` already applied, or the
lexer error where no token matches. The stream a parse runs over,
before the trivia around it is trimmed.

---

*Consult [api-reference.md](api-reference.md) for complete listing*