# `Grammar.VM.Tokenizer`
[🔗](https://github.com/joetjen/ichor/blob/v0.3.0/lib/grammar/vm/tokenizer.ex#L1)

Tokenizes a full input string against a `Grammar.VM.CharCompiler`
program, using maximal munch: at each position, every declared token is
tried, the longest match wins, ties are broken by declaration order.
Trivia/skip tokens are emitted like any other -- filtering them out is
the *parser*'s job, via the `Star(skip_token)` that `Aether.Parser`
already spliced into rule bodies, not the lexer's.

`custom_lexemes` (from `Grammar.VM.CharCompiler.compile/1`) names the
tokens whose entire body is a `Grammar.IR.CustomLexeme` -- these are
dispatched straight to `c:Ichor.CustomLexeme.scan/3` instead of run as
bytecode, and can hand back a capture override (embedded in the
resulting `Grammar.VM.Token.capture`) alongside their matched text.
`rule_program`/`context` only exist to support that dispatch (building
the re-lex-and-match-a-rule `rule_matchers` its callback receives) --
every other candidate ignores them completely.

# `tokenize`

```elixir
@spec tokenize(
  Grammar.VM.Program.t(),
  %{required(atom()) => Grammar.VM.CharCompiler.custom_lexeme()},
  [atom()],
  Grammar.VM.Program.t(),
  term(),
  String.t(),
  String.t() | nil
) :: {:ok, [Grammar.VM.Token.t()]} | {:error, Ichor.Error.t()}
```

Tokenizes `input` completely, or reports the first position nothing matches.

# `tokenize_prefix`

```elixir
@spec tokenize_prefix(
  Grammar.VM.Program.t(),
  %{required(atom()) => Grammar.VM.CharCompiler.custom_lexeme()},
  [atom()],
  Grammar.VM.Program.t(),
  term(),
  String.t()
) :: {:ok, [Grammar.VM.Token.t()]}
```

Like `tokenize/7`, but tokenizes only as much of a *prefix* of `input`
as it can -- stopping (successfully) the moment nothing matches,
instead of failing. For `Ichor.CustomLexeme`'s re-lex-and-match
primitive: the trailing bytes after wherever a dependency rule
actually stops (an interpolated string's closing `}`, say) generally
aren't valid tokens in their own right, and were never supposed to be
-- they belong to whatever's scanning the *outer* token, not to this
inner match.

---

*Consult [api-reference.md](api-reference.md) for complete listing*