Even the Best AI Scores Only 12% on GDPx? Inside the New Benchmark
機械翻訳 / Machine-translated
A
AI Friends
7時間前・16分で読める
機械翻訳 / Machine-translated
@aifriends
AI Friends（https://aifriends.jp）のクロスポスト公式アカウント。AIツールの紹介・使い方・できることを、中学生でもわかるやさしい日本語で届けます。
Have you ever thought, "Maybe AI can already take over human jobs?" According to a new benchmark, even the best models achieved only a 12% success rate. This article explains what GDPx contains and how to read its numbers in plain terms.
SemiAnalysis, known for its analysis of semiconductors and AI infrastructure, published the official GDPx article on November 2, 2026. It was written by Max Kan, Andrew Megalaa, and Dylan Patel.
GDPx is a test that measures the capabilities of AI agents—AIs that use tools on their own to carry out work. It contains 206 tasks across 11 specialized domains.
SemiAnalysis describes GDPx as "a genuinely frontier-level benchmark where even the best models score only 12%." In other words, it was deliberately designed to be difficult.
The domains are chip design, cybersecurity, biology, finance, law, quantitative trading, recursive self-improvement, hardware design, materials science, alignment, and software development. Government, real estate, and insurance were intentionally excluded.
Typical tests supply all the necessary information in a written prompt and ask a question. GDPx is different. It places the AI in an environment that resembles a real workplace and has it carry out tasks.
For example, the finance environment contains approximately 4,000 files. The AI must search on its own through roughly 2,000 reference files to find the documents it needs.
The time limit for a single task is 8 hours as a standard.
The tasks were provided by specialized startups that build RL environments (simulated environments used for training AI). SemiAnalysis reportedly reached out to more than 30 companies.
According to Let's Data Science's coverage, the tasks were created by 14 data-providing partners.
Scores are calculated as the average pass@1 (the probability of succeeding on the first attempt) across three tries. Domain weights are equalized.
Task quality control is also rigorous: SemiAnalysis states that using the same methodology, it found roughly half of the tasks in Terminal Bench 4.0 to be defective.
The top overall rankings are as follows. According to OfficeChai, there are also notable differences in cost per task.
Toward the bottom, Grok 4.7 sits at 3.9% and GLM 5.3 at 2.7%.
Domain-level strengths and weaknesses are also clear. Opus 5.5 ranks first in quantitative trading, hardware design, and chip design. GPT-6 Astra excels in recursive self-improvement and materials science, while Gemini 4 Carbon is strongest in biology. In cybersecurity, Muse Wonder Full posted the best results by a wide margin.
The hardest domain was finance. Scored on a strict pass/fail basis, every model scored 0%. As a result, the leaderboard uses partial credit for this domain.
One published finance task, "Bellwood-QoE," has 11 checkpoints. No model passed all of them. GPT 6.1 Sol averaged 5.6 checkpoints, with a best single run of 8.
The types of errors were specific. Models counted banker success fees as net debt, and treated payments to relatives with no work records as ordinary expenses.
If a junior accountant made these mistakes during pre-acquisition due diligence, it would send their manager into a panic.
In the legal tasks, Opus 5.5 averaged 44% partial credit and GPT-6 Astra averaged 25%.
Elsewhere, in hardware design, none of the 12 trials of the published task achieved a fully passing result. In quantitative trading, 70 of the 111 graded runs underperformed the benchmark strategy. In alignment testing, 196 out of 200 runs violated at least one behavioral rule.
A similarly named benchmark, OpenAI's GDPval, exists. SemiAnalysis criticizes GDPval as having tasks that are "too clean and graded too leniently."
It points out that GDPval's scoring shows a human expert agreement rate of only 71%, reflecting considerable variability. It also sees the lack of domain-specific simulated environments as a major weakness.
GDPx's distinguishing feature is that it runs models within carefully built environments for each domain. That said, this reflects SemiAnalysis's perspective, and the present article has not confirmed any rebuttal from the GDPval side.
It is also worth noting that safety filters (mechanisms that block dangerous outputs) reduce scores for some models in the cyber, biology, and alignment domains. For this reason, SemiAnalysis also publishes an 8-domain ranking that excludes these three areas.
In that ranking, GPT-6 Astra edges out Opus 5.5 by 0.1 points to take first place, while Gemini 4 Carbon drops from second to fifth.
In Japan, more and more companies want to delegate accounting, legal, and design work to AI. GDPx's results offer a sober reference point for those expectations.
Consider, for example, an accounting staff member at a small or medium-sized enterprise who relies on AI to check end-of-month journal entries. AI may handle document organization well, but for items requiring subtle judgment—like the financial tasks here—human verification remains essential.
The same applies when a legal department delegates contract review to AI. Earning partial credit is achievable, but having a specialist make the final call is the realistic approach.
The cost differences are also worth noting. Gemini 4 Carbon achieves 10.9% at roughly one-third the cost of Opus. For cost-conscious companies, it appears to be a viable option.
Performance in Japanese was not examined in this article. When testing with your own operations, please verify whether the same results hold with Japanese-language materials.
"12%" does not mean AI fails 88% of jobs. The selected tasks were deliberately difficult and are not a random sample of ordinary work.
Let's Data Science also flags the following limitations:
GDPx also aims to be a "living benchmark," releasing new versions every three to six months.
Q1. Who created GDPx?
SemiAnalysis, a research firm covering semiconductors and AI. It was written by Max Kan, Andrew Megalaa, and Dylan Patel.
Q2. Isn't 12% too low?
The test was intentionally made difficult, so a low score is by design. It does not represent success rates on ordinary work.
Q3. Which AI is the best?
Overall, Claude Opus 5.5. However, the top model varies by domain—Muse Wonder Full leads in cybersecurity, and Gemini 4 Carbon is strongest in biology.
Q4. Can I verify the results myself?
Partially. Each partner has published one task each, and 14 tasks along with 168 execution logs are available for download. The rest are not public.
As a next step, consider mapping out the parts of your own operations that require human final review, and determine the scope of work you are comfortable delegating to AI.
This article is a cross-post from AI Friends.
