Lists every token a grammar declares, in the exact order the Lexer's maximal-munch tie-break uses. Useful whenever you need to know why two tokens compete for the same input, without running the rest of the pipeline — see the cheatsheet for the one-line version.
mix ichor.tokens PATH_TO_GRAMMAR
PATH_TO_GRAMMAR must be .aether source — unlike
mix ichor.gen, this task doesn't go through
Ichor.GrammarImport, so ABNF/BNF/EBNF/PEG grammars aren't accepted
here.
Why maximal-munch order matters
Aether's Lexer always prefers the longest match at a given position;
token_order only breaks ties when two tokens match the same length —
e.g. a keyword like SELECT against the general IDENT token. Getting
that tie-break order wrong silently reclassifies input (an identifier
named selection matching as SELECT plus leftover ion, say). mix ichor.tokens prints the order the Lexer actually committed to, so you
can check it against what you expect before it turns into a confusing
runtime bug.
What it shows
Three kinds of tokens, in a single token_order-sorted table:
declared— tokens your grammar names explicitly (NUMBER := ...).anonymous— tokens auto-promoted from an inline string literal used directly in a rule (e.g."+"insideexpr := term (op:("+" | "-") term)*), so the Lexer still tokenizes it consistently even though the grammar never gave it a name.predefined— the five tokens every grammar gets whether it uses them or not:DIGIT,ALPHA,ALNUM,SPACE,HEX. Always present in the table, whether left at their built-in pattern or overridden in the grammar source.
Only needs the Aether front-end to run — no analysis pass, no VM/native backend — so it works on any grammar that parses, even one that would later fail the left-recursion/reference-check analysis pass. That makes it useful for inspecting a grammar mid-edit, before it's necessarily valid enough to compile.
Example
Given a calculator grammar:
@grammar "calculator"
@root expr
NUMBER := DIGIT+ ("." DIGIT+)?
expr := term (op:("+" | "-") term)*
term := factor (op:("*" | "/") factor)*
factor := NUMBER | "(" expr ")"$ mix ichor.tokens calculator.aether
# NAME KIND PATTERN
1 NUMBER declared DIGIT+ ("." DIGIT+)?
2 ANON_1 anonymous "+"
3 ANON_2 anonymous "-"
4 ANON_3 anonymous "*"
5 ANON_4 anonymous "/"
6 ANON_5 anonymous "("
7 ANON_6 anonymous ")"
8 DIGIT predefined [0-9]
9 ALPHA predefined [a-zA-Z]
10 ALNUM predefined [0-9] | [a-zA-Z]
11 SPACE predefined [ \u{9}\u{D}\u{A}]
12 HEX predefined [a-fA-F0-9]
NUMBER is declared, so it sorts first; the six bare string literals
("+", "-", "*", "/", "(", ")") each got auto-promoted to
their own anonymous token (ANON_1...ANON_6, numbered in the order
they were first seen); and all five predefined tokens close out the
table at their default patterns, even though this grammar never
references ALPHA or HEX directly — ALNUM's own default is a
choice between DIGIT and ALPHA, which is why it still appears
composed from them in the PATTERN column.
Example: an overridden predefined token
Overriding a predefined token in the grammar source changes its
PATTERN column but not its predefined kind or its position in
token_order:
@grammar "identifiers"
@root program
ALPHA := [a-zA-Z_]
program := ALPHA+$ mix ichor.tokens identifiers.aether
# NAME KIND PATTERN
1 ALPHA predefined [a-zA-Z_]
2 DIGIT predefined [0-9]
3 ALNUM predefined [0-9] | [a-zA-Z_]
4 SPACE predefined [ \u{9}\u{D}\u{A}]
5 HEX predefined [a-fA-F0-9]
ALNUM's pattern picks up the override too, since its own default is
defined in terms of DIGIT/ALPHA rather than a fixed pattern of its
own.
See also
mix ichor.gen— compile a grammar once it's ready, instead of just inspecting it.- Cheatsheet — quick reference.
Grammar.Tokens— the module this task is a thin CLI wrapper around, if you need the same introspection from inside Elixir code.