90% of US working hours fall outside what AI benchmarks measure, Mercor finds

Counting what the benchmarks cover

Mercor's analysis takes the US Department of Labor's ONET taxonomy (923 occupations, 22 job families, 18,797 tasks) and asks which tasks have a benchmark. It selects 29 public, agentic, non-saturated benchmarks, uses an LLM judge to map each benchmark prompt to ONET tasks, and counts a task as covered if at least five prompts map to it. Weighting by the time-share estimates of Hatgis-Kessell et al. produces the headline: "90% of American working hours are spent on tasks with little or no coverage from existing benchmarks."

The raw counts: 208 of 923 occupations have at least one covered task, six of 22 job families have none, and 924 of 18,797 tasks are covered. AI-exposed tasks are about six times likelier to be covered than unexposed ones, and the most important tasks within an occupation are six to ten times likelier to be covered than the least important. Even after controlling for AI exposure, occupations with larger wage bills are overrepresented. Lawyers' tasks are well covered while legal secretaries' and court clerks' tasks are barely covered.

In a pilot on four occupations using Mercor's own APEX benchmarks, weighting by task barely moved scores: for financial analysts, a simple average of 73.3% became 70.2% weighted. Fewer than a third of relevant tasks had enough APEX coverage to be included. The authors flag the 29-benchmark list as incomplete and subjective.

The thread: bias, or just physics?

The top comment read the wage-bill finding as intent, concluding the labs "want AI to take the good jobs". A reply cut it down:

"No, we are currently unable to do most tasks that require physical manipulation, independent of what anyone wants." — a reply to the "good jobs" reading

Another commenter put the cause in training data rather than ambition:

"AI is coming for jobs involving programming and math (which tend to pay higher) because it's possible to generate infinite training data for those fields via using a compiler/theorem prover to check correctness." — a commenter on why code and math get benchmarked first

That leaves manual work. One commenter who switched from data scientist to electrician wrote that the day starts uncomfortable "but I feel way better at the end of the workday than after an office job." Others were less relaxed: a reply listing a YC company that picks strawberries argued that hardware merely takes longer than software.

What the coverage gap implies

A benchmark-free task is not a safe task; it is an unmeasured one. For anyone deciding what to retrain for, the report's more useful reading is that current evaluations say little in either direction about most jobs. The authors' own list of open problems, including low statistical power and physical tasks, is where the next set of benchmarks has to go.