Skip to content
String Escape Sequences
step 1/5

Reading — step 1 of 5

Read

~3 min readTokenizing JSON

String Escape Sequences

A JSON string is a sequence of Unicode characters — not bytes — wrapped in double quotes. RFC 8259 gives the exact rule for what may appear between the quotes:

char      = unescaped /
            "\" ( %x22 / %x5C / %x2F / "b" / "f" / "n" / "r" / "t" / "u" 4HEXDIG )
unescaped = %x20-21 / %x23-5B / %x5D-10FFFF

Read the unescaped ranges carefully: exactly three things can never appear raw — the quote " (0x22), the backslash \ (0x5C), and every control character U+0000 through U+001F. Everything else, including é, 汉, and emoji, may appear literally. The nine legal escapes:

EscapeMeaning
\"quotation mark
\\backslash
\/forward slash (legal but never required; emit /)
\bbackspace U+0008
\fform feed U+000C
\nline feed U+000A
\rcarriage return U+000D
\ttab U+0009
\uXXXXa UTF-16 code unit, 4 hex digits (hex digits case-insensitive)

Anything else after a backslash — \a, \v, \x41, \' — is invalid JSON, even though C or JavaScript accept some of them. Reject; never pass it through.

\uXXXX is a UTF-16 code unit, not always a character

Four hex digits only reach U+FFFF. Code points above that (emoji, many CJK extensions) are written as two escapes forming a UTF-16 surrogate pair: a high surrogate in D800–DBFF followed by a low surrogate in DC00–DFFF. The decode formula:

python

Worked example for 😀 (U+1F600), written \uD83D\uDE00: hi − 0xD800 is 0x3D, lo − 0xDC00 is 0x200, so 0x10000 + (0x3D << 10) + 0x200 = 0x1F600. Your decoder must detect a high surrogate, demand the next \uXXXX, and combine them into one chr(code).

The idiom and the traps

Dispatch through a dict for the simple escapes, with an explicit branch for u:

python
  • Trap 1 — decoding surrogates separately. chr(0xD83D) and chr(0xDE00) are two unpaired surrogate characters, not 😀. Python will happily build that broken string and then fail to encode it. You must combine the pair before calling chr.
  • Trap 2 — a backslash at the very end. If the body ends right after \, indexing body[i + 1] explodes. Malformed input must become a clean error, not an IndexError traceback.
  • Trap 3 — trusting the closing quote. In "abc\" the final quote is escaped, so the string is actually unterminated. A left-to-right scan catches this naturally; a first-char/last-char check does not. (The graded inputs are gentler, but a real lexer scans.)

Your exercise

Decode quoted strings, printing the decoded value — or exactly ERR, bare, with no reason attached (unlike the tokenizer lesson's error format). "bad\xescape" must print ERR. The grader-caught mistake is the surrogate pair: hidden input "\uD83D\uDE00" must print the single character 😀 — decoding each escape independently prints two lone surrogates and fails. Also verify: "\u00e9clair" prints éclair, "\/" prints /, the empty string "" prints an empty line, and unterminated "unterminated prints ERR.

Discussion

Ask a question, share an insight, or help someone who’s stuck.

Sign in to post a comment or reply.

Loading…