Production Column Stores

Real-world systems and their choices:

ClickHouse (Yandex):

• MergeTree storage engine: like LSM but columnar.
• Vectorized + compiled.
• Low latency, very fast aggregations.
• Used by Cloudflare, Uber, eBay.

DuckDB:

• Embedded analytics database (SQLite-style).
• Column-vector + JIT.
• Single-process; great for laptops + notebooks.

Apache Parquet:

• File format, not engine. Read by Spark, BigQuery, Athena, Snowflake, Trino.
• Industry standard for data lakes.

Apache Iceberg / Delta Lake / Hudi:

• Table formats over Parquet files.
• Add: ACID transactions, schema evolution, time travel, optimistic concurrency.
• Used in modern data warehouses on object storage (S3).

Snowflake:

• Proprietary columnar engine on cloud storage.
• Decouples compute + storage.
• Auto-optimization (clustering, materialized views).

BigQuery:

• Google. Capacitor format (proprietary columnar).
• Massively parallel: 1000s of cores per query.
• Pay-per-query model.

Redshift:

• Amazon. PostgreSQL-style SQL on columnar engine.
• Older but huge install base.
• New version (RA3) decouples storage too.

Vertica (old-school commercial):

• Pioneered many techniques (projections, encodings).
• Less popular now but influential.

Trends:

• Lakehouse: data lake (object storage) + warehouse (SQL + ACID + perf). Iceberg/Delta + Spark/Trino.
• Streaming ingest: writes go to a "staging" tier; periodic compaction to columnar.
• Vectorized JIT: query compiled to native code per query.
• GPU acceleration: BlazingSQL, HEAVY.AI for super-fast aggregations.

Building one is a 10-year commercial effort. Use Parquet + DuckDB / ClickHouse / BigQuery / Snowflake.