Tag

Parquet

2 stories

DuckDB 2.0 adds a server mode and async I/O, and files that 1.x cannot open (duckdb.org)

The release adds a Quack server protocol, background reads for Parquet and CSV, VARIANT with automatic shredding, and triggers. A new PEG parser is now the default, ICU is gone, and 2.0 is supported until at least October 2027.

One user reports shrinking 7 TB of Azure NDJSON logs to 2.5% of their size in Parquet in under two hours on a medium VM.

Not just a headline — the discussion unpacked

The top HN threads: facts from the discussions, both sides, insider voices. Email, Telegram, or RSS.

A file format ships a WASM decoder inside every file, and HN argues archival sense (github.com)

F3 is a columnar data format — a Parquet rival — that carries the WASM binaries needed to decode it inside every file. One engineer who builds data connectors called the self-describing trick genius.

The pushback came from people who archive at scale: keeping petabytes for ten years, you don't want to depend on a WASM interpreter being fast in 2051, you want a dead-simple byte spec like Parquet's. And the README, several noted, never says which Parquet shortcomings F3 fixes.