# `Grammar.Native.TokenizerCompiler`
[🔗](https://github.com/joetjen/ichor/blob/v0.7.3/lib/grammar/native/tokenizer_compiler.ex#L1)

Generates the Tokenizer -> Lexer half of a compiled grammar's `tokenize/2`
(char-level maximal munch via `Grammar.Native.CharCompiler`, then
`@keywords`/`@refine` reclassification via the shared, backend-agnostic
`Grammar.Lexer`) -- extracted once out of `Grammar.Native.generate/2` so
`Grammar.Native.LR`/`Grammar.Native.GLR` can reuse it unchanged: every
engine consumes the exact same token stream, only the *parser* half
differs (direct PEG combinator calls, vs. compiled LR/GLR state
dispatch).

The caller supplies its own `alias Grammar.Native.Runtime.Tokenizer`
(and `Grammar.VM.Token`, for the `@spec`) -- this module only builds
the quoted defs, it doesn't own the generated module's own alias list.

# `generate`

```elixir
@spec generate(Aether.Grammar.t()) :: {[Macro.t()], Macro.t()}
```

Returns `{support_defs, tokenize_def}`: `support_defs` are private
helpers (`lex_candidates/2` plus every compiled char-matcher
function) to splice in alongside the caller's own parser defs;
`tokenize_def` is the public `def tokenize/2` itself.

# `has_customlexeme_deps?`

```elixir
@spec has_customlexeme_deps?(Aether.Grammar.t()) :: boolean()
```

Does `grammar` have any token-position `@native(...)`
(`Grammar.IR.CustomLexeme`) with a non-empty dependency list? Those
dependencies need a "re-lex and match a rule" primitive that only
exists against `Grammar.Native.RuleCompiler`'s own per-rule generated
functions (ordinary PEG codegen, one function per rule) --
`Grammar.Native.LR`/`.GLR` compile rules into per-automaton-state
functions instead, which have no such per-rule entry point. When this
is true, the caller (`Grammar.Native.LR`/`.GLR`) splices in
`Grammar.Native.RuleCompiler.compile/1`'s ordinary PEG rule functions
too -- unused by its own `parse`/`run`, present only so a dependency
reference resolves -- exactly mirroring how the *interpreted*
`Grammar.LR`/`Grammar.GLR` already compile a full
`Grammar.VM.RuleCompiler` program purely for this same purpose, even
though their own top-level parse never runs it.

---

*Consult [api-reference.md](api-reference.md) for complete listing*