Working with JSON

After this lesson you can move data across the JSON boundary on purpose: you know what survives the trip, can write reproducible text, reject malformed input, and teach json your own types.

A JSON object is a dict and an array is a list, which you already navigate. This lesson is about the edges: the text going in and out, and what is lost on the way.

Round trips are lossy

json.dumps returns text and json.loads parses text; dump and load do the same with a file object such as sys.stdin or io.StringIO.

The tuple came back as a list and the key 3 as the string "3": JSON keys are always strings and arrays are always lists. True and None are Python spellings, so str(value) is never JSON text; json.dumps(value) is. Sets, dates and bytes have no JSON form: TypeError: Object of type set is not JSON serializable. float('nan') dumps as NaN, which is not valid JSON.

Controlling the output

The default separators are ", " and ": ", with spaces; separators=(",", ":") removes them. With sort_keys=True the same data always yields the same bytes, which suits hashing and diffing. Non-ASCII characters are written as \uXXXX unless you pass ensure_ascii=False.

Malformed input

json.JSONDecodeError is a subclass of ValueError with msg, lineno, colno and pos attributes:

Single quotes, trailing commas, trailing text and empty input are all errors. Wording differs between Python versions, so print a fixed word, never str(e), when something compares your output.

Parsed is not valid

A successful parse proves the text was JSON, not that it has the shape you expect. Check it, and remember isinstance(True, int) is True:

Without the bool test, {"qty": true} would pass as a count. Also check the top level: json.loads("[1]") is a list, so doc["qty"] raises TypeError, not KeyError.

Your own types

default= is called only for values json cannot encode and returns something it can. object_hook= is the reverse: it receives every decoded object, innermost first, and returns a replacement.

Walking nested data

Parsed JSON is dicts, lists and scalars, so one branch per kind visits everything:

JSON Lines

Logs are often one document per line. Parse each line alone inside a generator, so one bad line costs one record:

Common mistakes

• Printing parsed values with print or str. {"ok": true} prints as {'ok': True}: Python spelling, single quotes. Use json.dumps.
• Leaving the default separators on. {"a": 1} and {"a":1} are the same data but different text, so exact comparison fails.
• Ordering the wrong thing. As text, $.a[10] sorts before $.a[2], and the line $.a-b=2 sorts before $.a=1 because - is less than =. Sort the paths alone.
• ensure_ascii=False "to be tidy". It turns \u00e9 in the output into a raw character.

Your exercise

Read all of stdin as one JSON document and report it two ways: as canonical text and as a list of leaf paths. The mistake to watch for is treating an empty {} or [] as having no leaves.