Streaming CSV

For large files (gigabyte CSVs are common in data engineering), DON'T load the whole thing into memory.

Same for writing: stream rows out without buffering everything.

For massive parallel processing:

• Split into chunks at line boundaries (CSV is line-friendly)
• Map per chunk — one worker per chunk
• Merge results

Tools:

• pandas.read_csv(chunksize=10000) — chunked iterator
• Apache Spark, Dask — distributed CSV
• csvkit, xsv — streaming command-line tools

Don't sort or aggregate within the parse step — defer to specialized tools.

xsv (in Rust) is one of the fastest CSV tools and handles 10GB+ files easily.