He generated 1,008 AI images of animals riding vehicles to test if labs train on a meme benchmark

The benchmark that became a meme, and the theory it spawned

For several years, Simon Willison — creator of Datasette, a well-known voice in the LLM-tooling community — has run the same informal test on every major model release: ask it to draw an SVG of a pelican riding a bicycle, and see how recognizable, and how absurd, the result is. It's become a genuine, if unscientific, industry-watched signal — good enough at spotting model quality quirks that it's referenced constantly whenever a new frontier model ships.

That popularity raised an obvious suspicion: if enough people are watching pelican-on-bicycle as a quality signal, would an AI lab ever quietly optimize training data toward nailing it specifically — "pelicanmaxxing," a play on "benchmaxxing," training toward a specific known eval rather than general capability? Dylan Castillo set out to test the theory properly rather than argue about it.

A properly controlled experiment

The design: an 8-animal by 6-vehicle grid — 48 total prompt combinations, with pelican-on-bicycle as just one cell among many — run across seven frontier models via OpenRouter, three samples per prompt, for 1,008 total generated SVGs. Each image was scored on animal recognizability, vehicle recognizability, and overall coherence using GPT-5.6 as an automated rater on 1–5 scales, with feature extraction to catch structural patterns.

The results undercut the theory cleanly. Pelicans ranked 6th of 8 animals in drawing quality overall; bicycles ranked 7th of 6 vehicle types (near the bottom of vehicles models render well). The specific pelican-bicycle combination landed 42nd of 48 — nowhere near the top of the pack, which is what you'd expect to see if a lab were quietly training toward that exact prompt. A regression analysis looking for a lab-specific boost on pelicans, or on the pelican-bicycle pairing specifically, found no statistically significant effect (p-values all above 0.05) for any of the seven models tested. One structural quirk did survive scrutiny — all 21 pelican-bicycle SVGs generated across every model faced right — but Castillo found the same rightward bias showing up across other animal-vehicle combinations too, suggesting it's a general tendency in how these models default to laying out a scene, not evidence of prompt-specific training.

"Little evidence" — and one HN pushback that reframes it

Castillo's conclusion: there's little evidence any lab is specifically pelicanmaxxing. GLM-5.2 showed the single largest pelican-bicycle score boost among the seven models tested, but even that result didn't clear statistical significance. If anything is being optimized, the more plausible story, per the article, is a broader "SVGmaxxing" — general improvement at SVG generation as a capability, not a narrow benchmark-specific trick, something the 48-combination design can't distinguish from genuine capability gains.

One Hacker News reply reframed the whole premise usefully: "you have to ask yourself at this point what 'maxxing' really means, since 'get better at drawing SVGs' is a useful skill." If a lab trains harder on vector graphics generally because it's a genuinely useful capability — UI mockups, diagrams, icons — that's not benchmark gaming in any meaningful sense, even if pelicans happen to benefit as a side effect. The rigor of Castillo's design is what makes that distinction possible to draw at all — without the controlled 48-combination grid, "pelicans got better" and "pelicanmaxxing is happening" would be indistinguishable.