Mercor estimates that 90% of US working hours involve tasks with little or no coverage in its provisional benchmark map. Google's engineering test exposes a separate problem: passing individual constraints is not the same as delivering work that clears every hard requirement.
An AI model can lead a public ranking and still be a poor choice for a particular company's workflow. There are two distinct questions. Was work resembling the buyer's own tasks included in the test? And did the resulting artifact satisfy every requirement that determines whether the work can be accepted?
Mercor's occupational map addresses the first measurement gap; Google's Physical Engineering's Last Exam examines the second. Neither study measures the return on deploying a model in a business.
The distinction matters before any buyer turns a score into a procurement decision. A benchmark can measure performance accurately on the tasks it contains without representing the tasks a company needs done. A model can also satisfy many component requirements without passing a complete deliverable check. The studies are complementary warnings, not measurements of the same population.
What Mercor’s prompt coverage map measures
Mercor mapped 29 benchmarks to the US Department of Labor's O*NET classification of 923 occupations and 18,797 tasks. It reports that 924 tasks met its threshold of at least five benchmark prompts. That is 4.9% of the task list by CTOL's arithmetic. Only 208 occupations, or 22.5%, had even one task that reached the threshold. Six of the 22 job families had no occupation that qualified under that rule.
These percentages count mapped tasks and occupations, not working hours, job automation or model accuracy. One qualifying task is enough for an occupation to enter the count, and benchmark prompts can map to multiple O*NET tasks. The 4.9% and 22.5% calculations therefore do not describe independent, comprehensive coverage of either all tasks within those occupations or the hours spent on them.
Mercor separately estimates that 90% of US working hours involve tasks with little or no representation in the benchmarks it examined. That estimate concerns working hours, not the complement of the 4.9% task figure. Mercor calls its map provisional: its benchmark list is neither complete nor perfectly representative, an LLM judge assigns prompts to tasks, and human experts validate only a subset of APEX tasks.
This is a warning about measurement coverage, not a finding that AI cannot perform 90% of work.
Google's engineering test measures complete deliverables
Google's Physical Engineering's Last Exam tests long horizon mechanical, electrical, civil and structural engineering work. Google reports 85 tasks, project folders averaging 61 files, and evaluations involving 14 frontier models and seven agent harnesses. The tested setting is demanding, but specific. It is not a representative sample of all enterprise work.
The highest performing model on its native harness achieved a mean score of 48.3%. Google also reports that the top configuration satisfied 91% of individual constraints on co authored tasks, while 41.3% of deliverables passed every hard safety and feasibility check. A mean benchmark score, a share of individual constraints and a share of whole deliverables are different units. They cannot be combined or subtracted to estimate an enterprise failure rate.
The relevant distinction is the acceptance rule. A deliverable must meet every applicable hard condition to pass the complete check, even if most individual conditions are satisfied. Google's figures show that separation within this engineering test. They do not establish acceptance rates for office software, customer service, legal review or other enterprise processes. Those workflows would need their own tests and their own definitions of an acceptable artifact.
The economic test is cost per accepted output
For a business buyer, an accepted result matters more than a score in isolation. The proposed decision measure is cost per accepted output, including model use, human review, rework and recovery when an attempt fails. A lower initial model cost is not necessarily a lower total cost if reviewers must repair more deliverables. That is a commercial hypothesis to test, not an outcome established by either study.
A useful private evaluation would sample actual company workflows and record full artifact acceptance, failures against mandatory checks, review time, rework and recovery. It would separately assess whether public benchmark rankings predict those results. The supplied Mercor and Google figures contain no enterprise cost or reviewer time measurements, so they cannot identify a financially superior model or quantify expected savings.
Scenarios for enterprise evaluation
The most plausible near term path is that public scores remain useful for shortlisting and internal acceptance tests determine deployment choices. The limited coverage in Mercor's map and the distinction between component checks and complete deliverables in Google's test support treating these as separate decisions. A published comparison of ranks, acceptance and costs on the same enterprise tasks would show whether the extra testing changes model selection.
A second plausible path is that public evaluations become better aligned with actual work. It would require more complete benchmark inventories, broader and human validated task mapping, and repeatable tests of finished artifacts. A later Mercor style map could document whether more occupations and working hours have qualifying prompt coverage. Even then, expanded representation would not by itself demonstrate model competence or attractive deployment economics.
The least supported outcome is that a general leaderboard becomes sufficient as a purchasing rule. Evidence for that case would require independent enterprise evaluations showing that public rankings reliably predict whole artifact acceptance and total cost across representative workflows. Neither study provides that comparison.
CTOL's testable judgment, held with moderate confidence, is that public rankings alone will not consistently identify the lowest cost accepted output across varied enterprise workflows. Independent studies finding that those rankings remain reliable after review and recovery costs are included would challenge that judgment. This is an inference from the measurement limits, not a reported buyer outcome.
Evidence to watch
An independently replicated mapping with a disclosed benchmark inventory, human checked task assignments and transparent working hours weights could test the robustness of Mercor's 90% estimate. A separate evaluation could report, for each model on identical enterprise tasks, first pass artifact acceptance, failed hard requirements, reviewer time, recovery effort and cost per accepted output. These are different tests and should not be compressed into one headline number.
Until such evidence exists, a public model score is evidence about the benchmark that produced it. It is not proof that a buyer's work was represented, that its finished outputs will pass or that deployment will pay for itself. The supplied sources establish no date for the proposed independent comparisons.