ClickHouse deleted a Postgres rewrite from its benchmark after finding per-query code in the binary

What the maintainer found inside the binary

ClickBench is ClickHouse's public database benchmark. On September 22, its maintainer Alexey Milovidov merged a PR removing the pgrust entry and gave two reasons.

First, the v0.4-preview binary embeds hand-written kernels for the benchmark queries, in files named q00.rs through q36.rs, plus plan matchers with refusal messages named after ClickBench. The v0.3 source describes "hand-lowered physical plans for all 43 ClickBench queries."

Second, hot runs reuse earlier answers. Query 20 took 10.8 s cold, then 11 ms and 5 ms on reruns. When the author perturbed queries, such as SELECT MIN(URL) FROM hits, 20 of 67 near-benchmark probes were refused with errors. The 43 benchmark answers themselves were correct.

"The actual accusations are false"

The thread's centre was one comment from the pgrust co-creator:

"For the most part, the actual accusations are false. This submission was an earlier version of pgrust 0.4 called pgrust 0.4-preview." — Michael Malis, co-creator of pgrust

Malis pointed to a newer 0.4-beta that was open-sourced the day before. Commenters did not accept the framing. Another commenter called it a non-sequitur: "It doesn't matter what the earlier version is called. It is what you submitted, and it is what the accusations are against." Others asked Malis to explain the file paths and the environment variable. A Google software engineer wrote that they loved "the confidence and complete lack of acknowledgement of the initial fuck up."

Closed development, then a drop

Commenters also asked why a beta ships with source to follow later. The answer: the team does large refactors and keep an internal version ahead of the public one. A commenter noted that this is the old cathedral model from Eric Raymond's essay, and cited SQLite as a current example.

The Goodhart problem

The sharpest framing came from another commenter: "Benchmarks should be used for measuring general purpose query engines, not be an end in themselves." If an engine can only answer the queries it was tuned on, its score says nothing about the database a buyer would actually run. A second ClickBench PR for the new version already exists, so the real test is whether the maintainers accept a submission whose source anyone can now audit.