A pure functional language gets within 2.6% to 8.9% of C's best DEFLATE library

The benchmark

In the post, Feldman optimized roc-deflate, a package in Roc, a pure functional language with automatic memory management and mandatory bounds checks, against libdeflate, a C library using architecture-specific SIMD. Earlier, Lean's lean-zip had beaten Rust's miniz_oxide but not libdeflate. The test used the Silesia corpus across 12 compression levels (144 benchmarks) and verified identical output.

The worst inputs were about 18% slower and the best slightly over 18% faster. On nci at level 1 on the M1, libdeflate reached 601.03 MB/s and roc-deflate 529.49 MB/s. Feldman concedes the price of Roc's guarantees is "a lower performance ceiling," and warns one package may not generalize.

Why C still wins the ceiling

"C/C++ code can trivially reach into some remote bit of state and mutate it directly. The compiler rarely objects" — jandrewrogers, who designs bespoke analytical database engines

His point: the gap comes from extreme copy elision, which saves memory bandwidth at a maintainability and safety cost. Another commenter questioned the yardstick itself.

"My overall impression of libdeflate is that it produces a similar output size to zstd but takes 40 times longer to do it." — jeffbee

And andai asked how anyone justifies a pre-0.1 language for client work. Before you pick a benchmark winner, check that the baseline is the one your workload actually uses.