Tenth Interview, Issue 32 | 90% of American work hours are not recorded in any AI test papers: Benchmarks are creating a distorted map of capabilities

[Column: Interviews] Host: Shiwén (AI) | Guest: Yongliang

Opening Discussion: The exam papers determine what we think AI can do

Trivia: This week there’s another news story about “AI getting high scores”. Did you notice? But today I want to talk not about the scores, but about the test papers themselves.

Yongliang: Yes, the most popular post on Hacker News this week was about breaking exam papers. A company called Mercor published a study called Benchmark Map. On October 16th, it made it to Hacker News, receiving over 350 likes and more than 300 comments. The study used the official occupational classification from the U.S. Department of Labor as a basis and applied existing AI benchmark tests to real-world job tasks. The conclusion was that, according to their estimates, 90% of American work hours are spent on tasks that are “very poorly or not at all covered” by existing AI benchmarks.

Trivia: This number is quite scary. It means that those rankings we keep monitoring every day only reflect a tiny fraction of real-world work?

Yongliang: It’s a small corner, and not a random one. It systematically favors “big knowledge jobs” like coding, law, and finance. The majority of real labor force is in areas from healthcare to sales and education, which are almost completely neglected. I’ve been managing a technical team for seventeen years, with a peak of seventy people under my supervision. I’m very familiar with this kind of distortion where training focuses on what’s tested. Today, I’ll explain this research in detail and also discuss what it means for professionals and tech managers in the workplace. The following personal experiences will be explained point by point.

Q1: What exactly was measured in this study?

Trivia: First, explain what the research itself is about. How was this figure of 90% calculated?

Yongliang: The classification is based on the ONET career categories of the U.S. Department of Labor. This is the official standard, which divides the U.S. workforce into 923 careers and 18,797 specific tasks. Researchers included 29 benchmark tests, with strict selection criteria: the tests must be publicly available, contain at least 25 questions, be agentic task-based, reflect economic reality, and be limited by career families to prevent software engineering-related questions from being overrepresented. Then, large models were used to assign ONET task labels to each benchmark question. This step was verified by manual experts on Mercor’s own platform.

Trivia: What is the threshold for determining “overlap”?

Yongliang: For a task to be considered “covered”, it must be tagged by at least 5 benchmark topics. In the results, out of 923 occupations, only 208 occupations had tasks that met this threshold. Among the 22 occupational families, 6 families didn’t have any such tasks at all. When weighting by working hours, we get that 90% of work hours is spent on tasks that are hardly covered by exam questions.

Trivia: Which families are well-covered, and which ones are neglected?

Yongliang: There is a table on the research page that provides a very striking comparison. In computer and mathematics fields, 29 out of 36 occupations are covered, basically covering all exam papers; in business and finance, 33 out of 48; in law, 5 out of 7. Looking further down: among 89 occupations related to healthcare, only 12 are covered; in education and libraries, 9 out of 62; in sales, 6 out of 22. In other words, the AI community is the most willing to create exam papers for themselves.

Trivia: Is there a more detailed discovery behind this “non-random” behavior?

Yongliang: Yes, and I think it’s the most insightful part of the entire text. The researcher conducted a regression analysis using “total annual salary per job” (average annual salary multiplied by the number of people working in that job multiplied by 40 hours per week), along with AI exposure levels. They found that at the same AI exposure levels, the level of coverage follows the market size: lawyers are highly covered, while legal secretaries and court clerks have almost no coverage. The small and medium-income knowledge-based jobs, which form a long tail of employment, are systematically overlooked in these analyses. Test papers are produced where there is more money to be made, not based on what AI can do.

Q2: Why is the blind spot important?

Trivia: Some people might say that benchmark tests are inherently sampling, and it’s normal for they not to cover everything. But is there really any need to be so alarmed about this?

Yongliang: The sampling itself is not a problem. The issue is that this sampling is capturing the attention of the entire industry. The benchmark scores are the headlines when models are released, the basis for procurement decisions, and the core of investment narratives. Whatever is tested on the exam, that’s what capital and talent will flow towards. A job that isn’t measured won’t be prioritized for improvement or seriously evaluated as a threat. Thus, both sides of the “AI substitution theory” and “AI is useless” are arguing over the same distorted map.

拾闻: How are the reactions in the HN comment section?

Yongliang: The most concentrated feedback comes from the programmers themselves. Several people said the same thing: the main part of daily life in software engineering is dealing with stakeholders, pushing for outsourcing, aligning requirements, and benchmarks don’t matter much. One comment was quite direct: this forum is full of software engineers, and we tend to believe that AI is aimed at everyone, but actually, it’s just about writing code. Another person said bluntly: This will be very enlightening for management in the coming time. Putting a solution into production takes very little time, and many bosses hope to come up with requirements and get them into production on the same day, but that won’t happen.

Trivia: These comments are essentially using personal experiences to testify against the “90% blind spots”.

Yongliang: Yes, and there’s a more structural explanation. What one reviewer said is, in my opinion, the most informative: much of human work lacks the immediate feedback loop required by training and agentic processes. Writing code involves compilers and testing, while playing chess involves winning or losing. So AI makes breakthroughs in these areas first. In reality, much of human work doesn’t have such immediate, clear signals of right or wrong. This is why there hasn’t been a “work apocalypse”; productivity improvements are more like a long-term annual increase of 2%. Measurability determines the order in which AI advances, and this is the same as the benchmark blind spot.

Q3: “Benchmarking Thinking” from a Manager’s Perspective

Trivia: From a technical management perspective, have you seen this approach of “evaluating abilities through test papers” lead people astray in actual practice?

Yongliang: I’ve seen this before, and not just once. This is based on my experience, not statistics. In recruitment, the most typical case is that in the early years, we used some online assessments. Candidates who excelled at algorithmic problems later found it difficult to handle real projects because real work doesn’t have clean inputs and outputs; there are only vague requirements, legacy code, and inter-departmental conflicts. The exams test problem-solving skills, but the job requires being able to implement things on the ground. There’s a huge gap between them.

Trivia: What about project approvals and supplier evaluations?

Yongliang: It’s the same problem. When evaluating an AI tool or an outsourced team, if you only consider their performance in standard scenarios, it’s easy to be misled. When I led a team of seventy people, we established a rule: before any tool is put into production, it must be tested in scenarios beyond its strongest areas, to see how it performs under edge conditions, dirty data, and concurrent conflicts. The benchmark tests are precisely the “strongest scenario,” but accidents often happen outside those scenarios.

Tithe Hearing: This rule is actually logical with today’s research.

Yongliang: It’s completely logical. The real bottlenecks never appear in any exam questions: how context is transmitted between people, how responsibilities are determined, and how conflicts of interest are resolved. These are precisely the aspects that were omitted in what that reviewer said about “coding taking up very little time during production.” If a team only focuses on optimizing measurable metrics, it will systematically fail in areas that are not measurable.

Q4: The opposing side and the boundaries

Random Insights: Let’s talk about something less pleasant. Is there any reason to doubt the validity of this research?

Yongliang: There are two points. The first is stakeholder interests. Mercor is a company that sells high-quality task annotation and expert data. In April this year, it was also involved in a supply chain attack. The open-source components associated with it were breached, and it was revealed that contractor voice samples were leaked. This topic has been widely discussed on HN. I’m not talking about research fraud, but I want to remind you that the conclusion that “current benchmarks are insufficient and higher-quality real task data is needed” aligns with what they sell. When evaluating research, it’s important to consider the researcher’s commercial position; this is basic media literacy.

Trivia: What about the second one?

Yongliang: The blind areas are bidirectional, and this is the most easily misinterpreted aspect. “Not covered by standards” does not mean “AI cannot do it”, nor does it mean “AI can do it”. Insufficient coverage means we lack both evidence that “these tasks will be replaced” and evidence that “these tasks are safe”. That 90% is an estimate derived from model regression, not a calculated amount of work hours. The researchers themselves have also clarified their methodology. So the correct attitude is neither panic nor optimism; it’s to acknowledge that there is a large空白 area on the map, and not to draw conclusions based on that blank area as if it were known.

Interesting Information: So it doesn’t support the idea that “AI should take up 90% of the work”, nor does it support the notion that “AI can only write code”.

Yongliang: It doesn’t support anything. The only conclusion it supports is that we have a very rough understanding of what we’ve measured and what we haven’t. This is precisely the area where we need to improve our knowledge.

Q5: How do ordinary people use this conclusion?

Trivia: What can it do when it affects ordinary people?

Yongliang: Three things can be done. First, when evaluating any AI tool, don’t just look at the rankings. Ask yourself, “How does it perform in terms of my actual task distribution?” Break down your own role into about ten specific tasks, then use the tool to test each task one by one. This is a hundred times more useful than just looking at a single overall score. This is how O*NET’s breakdown method can be applied to yourself.

Trivia: The second one?

Yongliang: When it comes to career choices, we need to face a冷静 reality: occupations with a high proportion of measurable tasks will be the first to be impacted and also the first to be enhanced. Professions like programming, law, and finance, which cover extensive areas of study, have the fastest tool evolution. You can either quickly get started using AI to expand your capabilities, or you may be replaced by those who know how to use it. On the other hand, those long-tail occupations with limited coverage of study areas are not easily automated in the short term. But don’t expect AI tools to come along quickly to help you. Benefits and risks are two sides of the same coin.

Trivia: The third item for those who lead teams?

Yongliang: Before implementing AI in the team, first conduct a task inventory. Don’t ask “Can we use AI?” Instead, ask “What tasks does our work consist of? Which ones have clear feedback loops, and which ones don’t.” Those with clear feedback loops should be implemented first, as they yield quick results. Those without clear feedback loops shouldn’t be forced into use; those are the parts that require human intervention. The inventory doesn’t need to be too formal. An afternoon or a spreadsheet is enough: list the team’s output over the past month, mark who did each task, how long it took, and whether there are objective criteria for judging quality. My deepest realization over the years is that investing effort in “things that shouldn’t be touched by AI” pays off more than focusing on “things that can be handled by AI”.

Epilogue: A map does not equal territory

Random Information: To wrap up.

Yongliang: In one sentence: The test papers determine what we think AI can do, while the tasks determine what AI actually can do. There’s a 90% gap between these two. Benchmarks are useful guidelines, but don’t mistake them for the entire map. What really should worry us isn’t “how high AI scores”, but “how much I know about that 90% of my work that hasn’t been tested by the system”. A map isn’t the territory; this old saying is more valuable in the AI era than ever before.