A DLL the loader swears is pinned, but its code memory is gone

Windows reported combase.dll as loaded with a pinned reference count, yet the bytes holding CoTaskMemFree had been deallocated. Calling it crashed the process, and shell32 took the blame for a corpse someone else had freed.

In brief

Raymond Chen pulled the 100 most recent crash dumps for one third-party program into a pivot table. They looked like four separate bugs: 11 stack overflows in shell32, 23 access violations in an unknown component, more overflows in ctxapclient64. One cause produced all of them.

An engineer who reproduces Windows bugs under virtualization had a sharper question than the diagnosis: what support tier do you need before the legendary Chen ever looks at your coredumps?

The crash that was four crashes wearing a trench coat

Cover of Raymond Chen's The Old New Thing
Cover of Raymond Chen's The Old New Thing

Raymond Chen's investigation opens on a stack-overflow crash that the dumps pinned on shell32.dll: a recursive loop of RtlLookupFunctionEntry and KiUserExceptionDispatch repeating hundreds of times until the stack ran out. Following the original exception led to combase!CoTaskMemFree and the line that turns a routine bug into a mystery: "The memory that contains the CoTaskMemFree function has been freed."

The loader still listed combase.dll as loaded with a load count of 0xFFFFFFFF — pinned, never to be unloaded. The page table disagreed. Someone had VirtualFreed the memory out from under a module the loader believed was permanent, so the normal FreeLibrary bookkeeping never ran. During process shutdown, shell32 happened to be the first to call into the dead module, which is how it ended up holding the murder weapon.

To size the damage, Chen pulled the 100 most recent crashes for that program into a pivot table. The signatures scattered: 11 stack overflows in shell32, 23 access violations in an unknown component, more overflows in ctxapclient64. He calls the pattern bucket spray — one root cause throwing crashes into many different buckets. A total of 46% of the crashes traced back to the same rogue force-unload. Part one ends without a named culprit: "The good news for the shell32 team is that they are off the hook; they are the victim. The bad news is that we don't know who the culprit is."

The thread argues about who gets Chen, not what he found

The people in the thread who'd shipped C++ recognized the shape of it before the technical payoff.

"The good news for the shell32 team is that they are off the hook; they are the victim. The bad news is that we don't know who the culprit is. The story of software development through the ages." — a former C++ programmer and project manager

One reader did the arithmetic on what bucket spray means for triage: a single defect inflating into four unrelated-looking crash reports.

"This is a case of bucket spray, where a single underlying cause generates a large number of different types of crashes." — a commenter who totalled the distribution himself

The loudest current, though, wasn't about the bug. It was envy of the access. An engineer who files Windows bugs from virtualized setups asked what most readers were thinking:

"What MSFT support policy do you need to have the legendary Raymond Chen take a look at it? … getting them to pay attention at all is an up-hill battle." — a developer who reports Windows-under-virtualization bugs

One detail reassured the skeptics rather than the opposite: Chen got those 100 dumps because the third-party crash-reporting pipeline actually delivered usable telemetry to Microsoft, as a reader who'd assumed it was theater noted. The unresolved part isn't the mechanism — it's that even after proving shell32 innocent, the dumps don't say which module called VirtualFree on combase. Part one identifies the corpse and clears the wrong suspect; the actual killer is left for a follow-up post, which one reader had already surfaced in the thread.

Get breakdowns like this every day

The top Hacker News stories, unpacked — by email, Telegram, or RSS.

Next story · 32 of 34 Read more
🚀Tech

RAM per GB now costs what it did in 2010 — a 16-year price regression

A Stanford chart plots the cheapest $/GB of memory from 1957 to 2026 on a log scale. The spike at the right end erases over a decade of progress: by per-GB price, DDR5 today sits roughly where DDR3 was in 2010.

★ 194 · 💬 74 · dam.stanford.edu · 3 min