# `Grammar.Tokens`
[🔗](https://github.com/joetjen/ichor/blob/v0.3.0/lib/grammar/tokens.ex#L1)

Token introspection for a compiled `Aether.Grammar` -- the logic behind
`mix ichor.tokens`. Depends only on the Aether front-end's output
(`token_order`/`tokens`/`anon_tokens`), not the analysis pass or either
backend, so it works on any grammar that parses, even one the analysis
pass would otherwise reject.

# `entry`

```elixir
@type entry() :: %{name: atom(), kind: kind(), pattern: String.t()}
```

# `kind`

```elixir
@type kind() :: :declared | :anonymous | :predefined
```

# `describe`

```elixir
@spec describe(Grammar.IR.expr()) :: String.t()
```

Renders a `Grammar.IR` expression as a compact PEG-like pattern string,
for display only -- not a serialization format, and not fed back into
the parser anywhere.

    iex> Grammar.Tokens.describe(Grammar.IR.literal("SELECT"))
    "\"SELECT\""

    iex> Grammar.Tokens.describe(Grammar.IR.char_class([{?0, ?9}, {?a, ?f}]))
    "[0-9a-f]"

    iex> digit = Grammar.IR.char_class([{?0, ?9}])
    iex> Grammar.Tokens.describe(Grammar.IR.seq([Grammar.IR.plus(digit), Grammar.IR.opt(Grammar.IR.literal("."))]))
    "[0-9]+ \".\"?"

    iex> Grammar.Tokens.describe(Grammar.IR.choice([Grammar.IR.literal("+"), Grammar.IR.literal("-")]))
    "\"+\" | \"-\""

# `list`

```elixir
@spec list(Aether.Grammar.t()) :: [entry()]
```

Every token in `grammar`, in `token_order` (the same order the Lexer's
maximal-munch tie-break uses) -- user-declared tokens, tokens
auto-promoted from an inline rule literal (`kind: :anonymous`), and the
five always-present predefined tokens (`kind: :predefined`), whether
overridden or left at their default.

---

*Consult [api-reference.md](api-reference.md) for complete listing*