Storage Files & Layout

Modern column stores organize files as:

table/
  column_id.col
  column_name.col
  column_age.col
  ...
  metadata.json

Each .col file:

• Header (column type, encoding, statistics).
• Data blocks (~256KB-1MB per block).
• Footer (block index).

A row-group / stripe (Parquet term) groups rows together:

stripe_0/
  column_id [rows 0..1000000]
  column_name [rows 0..1000000]
  ...
stripe_1/
  ...

Why stripes?

• Row reconstruction: to fetch row 5, you read offset 5 of each column file in the same stripe.
• Pruning: stripe-level statistics (min/max per column) let queries skip whole stripes.
• Parallelism: stripes processed independently across cores/machines.

Parquet structure:

File:
  RowGroup 0
    Column 0: PageHeader, DataPages
    Column 1: PageHeader, DataPages
  RowGroup 1
    ...
  Footer: RowGroupMetadata, ColumnMetadata, Schema

ORC (Hive) is similar.

Statistics: per-column-per-stripe min/max/null_count. A query like WHERE age > 50 skips any stripe with max(age) <= 50.

Bloom filters: per column, can skip stripes for point lookups too.

Sort by primary key:

• Write sorted within stripes (great for range queries).
• Cross-stripe ordering only meaningful if globally sorted.

In-memory columnar:

• Apache Arrow: cross-language in-memory format.
• Allows zero-copy data sharing between Python pandas, Spark, R, etc.