# `Grammar.VM.TokenInterpreter`
[🔗](https://github.com/joetjen/ichor/blob/v0.3.0/lib/grammar/vm/token_interpreter.ex#L1)

Runs a `Grammar.VM.RuleCompiler`-produced program against a lexed token
stream -- the machine `Grammar.VM` uses for the parser stage.

Alongside recognition, it builds the raw capture tree `Ichor.Actions`
needs: each rule invocation gets its own "frame" (a
`%{pending: [...], captures: []}` accumulator), pushed on `:call` and
popped on `:return`, populated by the `:cap_start`/`:cap_end` pairs
`Grammar.VM.RuleCompiler` brackets every capture (implicit or explicit)
with. `captures` is an ordered list, not a map -- list order is the
one thing Elixir actually guarantees, which is what lets
`Ichor.Actions.eval_all/2` evaluate sibling captures in true source
order instead of trusting a plain map's own (cross-OTP-version-
unstable) iteration order; `merge_capture/3` mirrors
`Grammar.Native.Runtime.Parser.merge_one/3` (from `ichor_runtime`)
exactly, for parity with the native backend.

Structurally the same backtracking discipline as
`Grammar.VM.CharInterpreter` (including snapshotting the call stack in
every choice point, for the same reason -- see that module's docs),
plus two more things to snapshot/restore: the capture-in-progress
`frame` (a failed alternative's partial captures must never leak into
a sibling alternative that goes on to succeed) and the `@indent`/
`@samecol` reference-column stack.

`Star`'s loop-back uses `:test_progress`, not plain `:commit` -- a
runtime guard against a body that matches without consuming anything
(see `Grammar.VM.Compiler`'s docs on why this can't be left to
`Grammar.Analysis`'s static empty-repetition check alone).

`context` is threaded through every instruction purely so `{:custom,
...}` (a `Grammar.IR.Custom` `@native(...)` node) can hand it to
`c:Ichor.CustomRule.match/4` -- every other instruction ignores it. It's
read-only here: matching never produces a *new* context, only
`Ichor.Actions.evaluate/5` does that, between top-level forms.

# `raw_capture`

```elixir
@type raw_capture() :: Ichor.Capture.node_t()
```

# `rule_matcher`

```elixir
@type rule_matcher() :: (tuple(), non_neg_integer() ->
                     {:ok, non_neg_integer(), raw_capture()} | :fail)
```

# `run`

```elixir
@spec run(tuple(), non_neg_integer(), tuple(), term()) ::
  {:ok, non_neg_integer(), Ichor.Capture.raw_captures()} | :fail
```

Attempts to match `entry` against `stream` (a tuple of `Grammar.VM.Token`,
for O(1) indexed access) starting at position 0. On success, returns the
first unconsumed stream index and the matched rule's own raw captures
list -- the caller decides whether the index means "matched everything".

# `run_from`

```elixir
@spec run_from(tuple(), non_neg_integer(), tuple(), non_neg_integer(), term()) ::
  {:ok, non_neg_integer(), Ichor.Capture.raw_captures()} | :fail
```

Like `run/4`, but starts at `start_pos` instead of 0 -- for matching one of *several* top-level occurrences in the same stream (`Grammar.VM.run_sequence/4`).

---

*Consult [api-reference.md](api-reference.md) for complete listing*