Unicode Character Classes

Real text isn't ASCII. Real regex engines need to know about Unicode.

What does \w match?

• ASCII regex: [A-Za-z0-9_] — 63 chars
• Unicode regex: every Unicode letter (Devanagari, Cyrillic, CJK), every Unicode digit (Devanagari १२३, Arabic ٠١٢, full-width ０１２), plus _

Python str regex is Unicode by default:

To force ASCII-only, pass re.ASCII:

In JavaScript: use the u flag for Unicode-aware regex. In Go: use (?U) flag or \p{Letter}. In Rust regex: Unicode is default.

Unicode property classes

The gold standard is \p{Property}:

\p{L}      any letter
\p{Lu}     uppercase letter
\p{N}      any number (digit, letter-like number, fraction)
\p{Nd}     decimal digit
\p{Greek}  Greek alphabet
\p{Han}    Chinese-Japanese-Korean Han characters
\P{L}      negation: NOT a letter

Python's re does NOT support \p{} — use the regex library for that. re2, Go's regexp, and Rust's regex all support it.

Code-point ranges

Character classes work with code points:

The ranges use UTF-32 code points, NOT UTF-8 bytes.

Encoding gotchas

UTF-8 is variable-width: ASCII chars are 1 byte, others are 2-4. If your regex operates on bytes (RE2 does, internally), the engine handles UTF-8 decoding for char classes but reports BYTE offsets in matches.

Python's re operates on Unicode code points if you pass str, but on bytes if you pass bytes. Don't mix.

Why this is hard

The Unicode tables for \w/\d/\p{} are HUGE — Unicode 15 defines over 150,000 code points across hundreds of properties. Real engines build these as compact bitmaps or interval lists at compile time. RE2's Unicode tables alone are ~1 MB of generated C++.

For THIS lesson: leverage Python's Unicode-aware re. The exercise teaches the semantic difference between ASCII and Unicode regex — the implementation details belong in a dedicated Unicode tables generator.