Tokenizing Templates

A template is mostly literal text with embedded markers. Lexer recognizes:

• TEXT: literal text.
• VAR_OPEN {{, VAR_CLOSE }}.
• TAG_OPEN {%, TAG_CLOSE %}.
• COMMENT_OPEN {#, COMMENT_CLOSE #}.

Inside {{ ... }} and {% ... %}: regular expression-style scanner emits sub-tokens (IDENT, OP, STRING, NUMBER, etc.).

Edge cases:

• Escaping: {{ '{{' }} outputs literal {{. Or {% raw %}...{% endraw %} block.
• Whitespace control: {%- if -%} strips surrounding whitespace.
• Nested templates: not directly nested; parser handles via include.

Hello {{ name }}, today is {% if is_weekend %}weekend{% else %}weekday{% endif %}.
{# This is a comment, not rendered #}

Tokenizer emits: TEXT("Hello ") VAR("name") TEXT(", today is ") TAG("if is_weekend") TEXT("weekend") TAG("else") TEXT("weekday") TAG("endif") TEXT(".\n") (comment skipped).

Inside VAR / TAG, second-pass mini-lexer parses identifiers, operators, etc.

Whitespace control:

• {%- ... %}: strip whitespace BEFORE the tag.
• {% ... -%}: strip whitespace AFTER.
• Useful to keep templates readable without producing extra blank lines.