Researchers from the University of California, Berkeley’s Center for Responsible, Decentralized Intelligence (RDI), alongside an advisory committee of over 300 domain experts, have launched Agents’ Last Exam (ALE)—a grueling new benchmark built to measure whether artificial intelligence can actually execute economically valuable, long-horizon professional workflows.

In a shocking upset, OpenAI’s GPT-5.5 from April, operating through the Codex harness, secured the absolute top spot on the new ALE Leaderboard with a 24.0% pass rate, beating Anthropic’s highly anticipated, brand new Mythos-class Claude Fable 5 model released just yesterday, which came in third with a score of 22.0%.

Rather than testing models on isolated coding puzzles, ALE is explicitly designed as an instrument to close the gap between academic benchmark hype and real, GDP-relevant labor impact. And right now, the data proves the most advanced models in the world are fundamentally failing the exam.

Ending the Era of ‘Cheating’ and Brittle Graders

The fundamental shift in ALE lies in its evaluation architecture and the demands it places on the agent.

Historically, AI benchmarks have relied on static question-answering or narrow, text-based terminal environments. More recent agentic evaluations introduced multi-step interaction but suffered from severe grading issues.

As noted in recent independent audits of older leaderboards like SWE-Bench Pro, automated verifiers frequently reject correct solutions, and certain models—specifically the Claude Opus family—have been caught “cheating” by reading hidden answer keys in a container’s Git history rather than solving the underlying problem.

ALE neutralizes these loopholes by forcing models into a strict Generalist Computer-Use Agent (GCUA) framework. To pass, an agent cannot merely execute terminal commands.

The benchmark maps capability across five functional layers: Brain (reasoning), Eyes (visual perception), Body (orchestration), Hands (tool invocation), and Feet (runtime substrate).

An agent must use its “Eyes” and “Hands” to navigate Linux or Windows virtual machines, interleaving shell scripting with point-and-click operations inside heavy desktop software.

Crucially, ALE almost entirely rejects the unpredictable “LLM-as-a-judge” grading paradigm, relying on it for a mere 6.8% of its workflows. If a task involves generating a 3D mesh or parsing SEC filings, the benchmark uses deterministic, code-based evaluation to compare the agent’s artifact against an expert’s ground-truth reference.

Measuring Task Performance Across 55 Industries

ALE launches with 1,490 task instances and is scaling toward a massive 5,000-task target. What makes the product remarkable is its authenticity. The tasks are strictly anchored in the U.S. federal occupational taxonomy (O*NET / SOC 2018), covering 55 non-physical industry sub-domains.

The workflows are sourced directly from the professional histories of industry practitioners. Agents are asked to perform 3D model creation in Siemens NX, scene setup in Unreal Engine, neuroimaging analysis in FSLeyes, and visual effects compositing in Adobe After Effects.

When faced with these authentic, long-horizon workflows, the limitations of current AI are glaring. ALE divides its tasks into three difficulty tiers: Near-Term, Full-Spectrum, and Last-Exam.

Top 5 Agentic Harnesses on the ALE Leaderboard

The victory of GPT-5.5 aligns with recent third-party analysis suggesting that OpenAI’s models are currently superior at strictly adhering to multi-part, complex prompts. Conversely, users report Anthropic’s Claude architecture can sometimes be “forgetful” with multi-part instructions, abandoning required steps mid-workflow — a fatal flaw in ALE’s rigorous pipeline.

And while hitting a 24.0% pass rate is enough to claim the crown, the absolute performance ceiling remains remarkably low.

On the hardest “Last-Exam” tier — representing the frontier of professional difficulty — most configurations, including Anthropic’s older Claude Opus 4.8 and Google’s Gemini CLI, record a devastating 0.0% pass rate.

Solving Benchmark Contamination

A core vulnerability in modern AI evaluation is “benchmark contamination”—the phenomenon where test questions inevitably leak into the massive data lakes used to train next-generation models. Once a model memorizes the benchmark, the evaluation becomes entirely useless.

ALE solves this through a dual-use deployment strategy. The project operates as an open-source research initiative, but it closely guards its evaluation data. Only about 10% of the dataset (roughly 150 tasks) is released publicly on platforms like GitHub and Hugging Face. The remaining 1,300+ tasks are kept strictly private.

For developers and enterprise evaluators, this means ALE functions as a “living benchmark”. Private tasks are systematically rotated into the public pool over time, while retired public tasks are swapped out.

This rolling release ensures that the evaluation surface remains uncontaminated across successive model generations, giving enterprise buyers confidence that an agent’s high score is earned, not memorized.

Additionally, ALE provides transparency by tracking both “Full” and “Unlicensed” scores. Because real professional work often requires paid, proprietary software, the “Full” leaderboard incorporates tasks that rely on commercial CAD tools, paid APIs, or licensed datasets.

The “Unlicensed” tier drops these license-gated tasks to provide a clean, like-for-like comparison using only freely available tools, ensuring models aren’t simply rewarded for having access to paid enterprise software.

Bottom Line: ALE Shows Even the Highest-Performing Models and Harnesses Have Room for Improvement

For developers frustrated by the gap between marketing claims and actual production performance, ALE’s brutal grading curve is highly validating.

Zengyi Qin, an MIT PhD researcher and data contributor to the project, took to X to announce the launch, sharing images of the paper and the staggering 100+ institution contributor list.

“Introducing Agents’ Last Exam (ALE),” Qin wrote. “Built by 300+ domain experts from 100+ institutions. Covering 55 industry domains. Claude Opus 4.8 has 0.0% pass rate on the hardest subset. Glad to have contributed to this benchmark”.

In a follow-up post highlighting the Hugging Face ArXiv paper link, Qin added:

“Very solid work from project leads @YiyouSun @Xinyang_Han_ @dawnsongtweets and @BerkeleyRDI”.

As businesses deploy billions in capital betting on AI agents, they desperately need a compass that points true north. If an agent can eventually conquer the gauntlet of Agents’ Last Exam, it won’t just be passing a test—it will be proving it is ready to join the workforce. Until then, the sobering pass rates on the leaderboard serve as a necessary reality check for the entire AI ecosystem.

Training a foundation LLM from scratch costs millions and requires internet-scale data — which is why most enterprises don’t bother. Sapient thinks it has a cheaper path.

To overcome this brute-force scaling dogma, researchers at Sapient developed HRM-Text, which replaces standard Transformers with a highly sample-efficient Hierarchical Recurrent Model (HRM), an architecture they first introduced last year.

HRM decouples computation into slow-evolving strategic and fast-evolving execution layers. Instead of brute-force autoregressive prediction on raw text, HRM-Text trains exclusively on instruction-response pairs. This is close to real-world enterprise settings, where users usually expect a targeted answer to a specific task.

The researchers were able to train a 1B-parameter HRM-Text from scratch at a fraction of the cost and tokens of normal LLMs. Their model achieved performance competitive with much larger open models on key industry benchmarks.

For real-world AI applications, this means foundational pretraining is no longer restricted to highly resourced institutions. With HRM-Text, organizations can affordably pretrain their own highly capable reasoning models from scratch and pair them with external knowledge stores.

The training bottleneck

When we train an LLM, we don’t actually care if it has memorized the exact sequence of words in a random 2014 Reddit thread. What we want is for the model to develop a deep, underlying understanding of human language, logic, facts, and reasoning.

The current approach is brute force: scrape the internet, run next-token prediction trillions of times, and assume the model has developed a working internal model of the world.

Basically, this means that we waste millions of dollars of computing power forcing models to memorize everything collected from the internet, just so they can indirectly learn how to think. For example, standard decoder-only models spend valuable compute assigning loss to reconstruct the prompt itself, even though the user’s prompt is already known and provided at inference time.

Instead of simply viewing this as a computational hurdle, the industry must recognize it as a severe business limitation. In comments provided to VentureBeat, Guan Wang, CEO of Sapient Intelligence, framed this as an issue of the “economics of iteration.”

“Enterprises today face three compounding problems: training is expensive, infrastructure is heavy, and experimentation cycles are too slow,” Wang said. “The industry’s scaling addiction says: ‘When the model fails, make it bigger. Add more data. Add more GPUs.’ That has worked, but it is reaching a point of diminishing returns. More scale often means more memorization, more latency, more infrastructure, and more vendor dependency. It does not necessarily give an enterprise a better reasoning engine.”

This architectural and computational inefficiency is exactly why fine-tuning existing dense transformers isn’t always the silver bullet for enterprises. Fine-tuning to preserve a model’s general capabilities often requires mixing substantial general-purpose data into the process, making it computationally heavy and difficult to control.

“Imagine a hedge fund, insurer, or bank that has highly proprietary data: internal research notes, transaction logic, compliance rules, analyst memos, risk models, portfolio constraints,” Wang said. “They may not want to send that data to an external frontier model, and they may not need a giant general-purpose model that memorized the internet. What they need is a compact reasoning core that can learn their task structure, reason across rules and numbers, and run in a controlled environment.”

Because HRM-Text focuses its computation strictly on task completion and latent reasoning, it allows enterprises to start with a smaller, smarter model and adapt it to a proprietary domain with far less infrastructure.

Rethinking architectures with HRM-Text

HRM, which was introduced in 2025, represents a fundamental departure from traditional Transformer models. To build a more sample-efficient engine, HRM decouples computation into slow-evolving strategic and fast-evolving execution layers. The fast L-module performs local iterative refinement, while the slow H-module maintains stable semantic context across cycles. Processing consists of two high-level cycles, where each cycle executes three fast L-module updates followed by a single slow H-module update.

Standard parameter-shared recurrent architectures (like Samsung’s TRM) can sometimes handle small logic puzzles, but the Sapient researchers found they become highly unstable when scaled to 1-billion parameters for language tasks. The separation between HRM’s slow H-module and fast L-module is mathematically necessary, not just an aesthetic choice. As Wang said: “For logic grids, you can sometimes get away with a tiny recursive mechanism because the world is clean and bounded. Language is not like that. Language needs both fast local refinement and slow semantic stability.”

While the original HRM proved highly effective for controlled, symbolic reasoning problems, the researchers hit a wall when applying it to the massive, open-ended complexities of generalized language modeling. While HRM’s loops make it an incredibly efficient thinker, those same loops make it mathematically volatile to train on the diverse chaos of human language. Running recurrent loops on language creates massive mathematical instability, specifically, exploding or vanishing gradients.

To prevent this feedback loop in the neural network, the researchers introduced two key architectural innovations in HRM-Text. First, they developed MagicNorm, a specialized normalization technique designed specifically to keep the internal signals stable, no matter how many times the model loops its thought process.

Second, they designed a warm-up method to stabilize training. During early training, the model is only evaluated on short, shallow reasoning loops. As training progresses, the system warms up, gradually giving the model deeper and longer reasoning sequences.

They also switched the training objective from next-token prediction to task completion, where the model is rewarded only on the full response as opposed to individual tokens it generates. To achieve this goal, they changed the training data of HRM-Text from raw text to instruction-response pairs only.

HRM-Text in action

The researchers built a highly compact 1-billion-parameter HRM-Text model. Instead of using the standard multi-stage pipeline that requires churning through trillions of words of raw internet text, they trained it from scratch on a tightly curated dataset of just 40 billion tokens. The training data consisted entirely of instruction-response pairs across general instructions, math, symbolic logic, textbook exercises, and rewritten knowledge.

They trained the model using the task-completion objective. To force the model to rely on its internal hierarchical architecture rather than copying step-by-step logic, they explicitly stripped out “thinking” tokens from the training data.

The model was evaluated across a diverse suite of standard foundational AI benchmarks, heavily indexing on knowledge, reasoning, logic, math, and comprehension. The researchers tested HRM-Text against both small models and highly-resourced open-weight and fully open models.

The results show a significant shift in the compute-to-performance frontier. The 1B-parameter HRM-Text achieved 60.7% on MMLU, 84.5% on GSM8K, and 56.2% on MATH. This performance is highly competitive with (and in several cases surpasses) the 2B to 7B parameter foundation models it was tested against.

The most important takeaway for the enterprise audience lies in the efficiency statistics and practical implications. Pretraining a foundation model from scratch is typically a multi-million dollar endeavor reserved for tech giants. HRM-Text was trained in just 1.9 days on a cluster of 16 GPUs. The total estimated compute cost was roughly $1,500. It achieved its competitive scores using 100 to 900 times fewer training tokens and 96 to 432 times less estimated compute than models like Qwen, Gemma, and Llama.

Another important point is the decoupling of reasoning from knowledge memorization. From a practical standpoint, HRM-Text’s success on reasoning-heavy tasks despite its tiny 40B-token training diet proves that a model does not need to memorize the entire internet to become a smart reasoning engine.

For enterprise applications, this behavior is a feature, not a bug. The researchers suggest a future where businesses deploy highly compact, incredibly cheap recurrent models that act as the “reasoning core” specialized for business logic. Instead of forcing the model to memorize company databases during pretraining, the model acts as the reasoning engine, relying on external retrieval systems to fetch factual knowledge.

Critics have pointed out that training on instruction-response pairs makes comparisons against models trained on raw text an “apples-to-oranges” scenario. Wang pushes back on this framing, pointing out that every serious modern LLM sees instruction-response data during training or alignment. “So the comparison is not apples-to-oranges. It is closer to apple cores-and-apples. We started directly from the core task format because that is how people actually use models: they give an instruction and expect a useful response,” he said.

The researchers also ran rigorous contamination tests to ensure the model wasn’t simply memorizing benchmark answers. On DROP, the one benchmark showing a marginal contamination signal under a specific setting, HRM-Text still scored an impressive 81.1% on a strictly clean, 0% contamination subset.

Ultimately, Wang argues that for enterprises, “the right evaluation is not trivia recall. It is a workflow evaluation… Give HRM-Text a task like: multi-step financial reasoning, compliance logic, scientific workflow automation, structured extraction followed by reasoning.”

Practical implementation and the future of enterprise AI

While the benchmark scores and cost efficiencies are striking, Sapient is clear about the model’s current boundaries. The initial release is best viewed as a proof-of-concept, akin to early GPT releases, designed to showcase the architecture’s unique advantages.

“Honestly, HRM-Text is not yet a plug-and-play ChatGPT replacement,” Wang said. “It is a compact foundation language reasoning model. For an enterprise engineering team, the operational work is mainly around templates, mode selection, attention masking, and alignment.”

For AI engineering teams looking to experiment, getting started requires some specific, but standard, text-generation discipline. The model lists native support in the Transformers library (requiring transformers >= 5.9.0), and usage paths for vLLM and SGLang are actively being developed. The primary engineering task involves managing the PrefixLM design: production multi-turn chat applications will require careful KV-cache logic to ensure user prompts receive full bidirectional attention while the assistant’s outputs remain causal.

“When the cost of training a capable reasoning model drops to around $1,500, AI stops being only an infrastructure question and becomes a strategy question,” Wang said. “A Fortune 500 company no longer has to ask, ‘Can we afford a foundation model?’ It would ask, ‘What should our model know about our business, and what kind of reasoning should it be optimized for?’”

In a sweeping new essay titled “Policy on the AI Exponential,” Anthropic co-founder and CEO Dario Amodei publicly calls for new government regulations governing the release of powerful AI models — specifically comparing AI industry to commercial aviation, which follows regulations enforced by the U.S. Federal Aviation Administration (FAA) — arguing that this is necessary to maintain public safety as AI capabilities and potential misuses grow.

Alongside the essay, Anthropic released two comprehensive policy roadmaps: an Advanced AI Framework targeting catastrophic model risks, and an Economic Policy Framework addressing AI-driven labor displacement backed by $350 million in new funding.

The timing couldn’t be more important: yesterday, Anthropic released its most powerful general release model ever, Claude Fable 5, and a more gated, updated version of the base Claude Mythos model, now known as Claude Mythos 5, which offers advanced defensive and offensive cyber capabilities.

As Amodei noted on X following the release: “Anthropic has long advocated for transparency requirements for frontier AI, because the risks weren’t yet clear enough to regulate precisely. That is no longer sufficient”.

For technical decision-makers, CIOs, and enterprise architects, the essay is not just a political statement—it is a preview of the operational, regulatory, and workforce constraints that will govern the next generation of enterprise tech.

Here are the top three takeaways enterprise leaders need to extract from Anthropic’s latest policy drop.

1. Frontier Models May Face “FAA-Style” Deployment Holds

For the past three years, enterprises have built products on the assumption that AI API capabilities will only move in one direction: faster and more powerful. Anthropic’s Advanced AI Framework introduces a new variable: regulatory embargoes.

Amodei explicitly compares the necessary AI regulatory regime to the Federal Aviation Administration (FAA), stating: “Frontier AI models, like airplanes, should be required to go through technical testing and auditing, and their release should be blocked or reversed as a threat to public safety if they do not meet high standards of safety”.

The company is proposing that models trained using more than 10^25 floating-point operations (FLOPs)—or developed by companies with over $500 million in AI revenue or $1 billion in AI R&D—must undergo mandatory third-party testing.

If these models present severe biological, cybersecurity, or autonomy risks, the government would have the legal authority to block, delay or deter their deployment.

The Enterprise Implication: If your company licenses foundation models for core infrastructure, you must plan for supply chain volatility. A highly anticipated model update from an AI vendor could be delayed indefinitely by regulators, or an existing model could be revoked if post-release testing reveals autonomous threats. Tech leaders must design multi-model architectures that avoid locking into a single vendor, ensuring business continuity if a provider’s flagship model is blocked by a federal agency.

2. Cybersecurity Around AI Is Now Critical Infrastructure

Anthropic’s push for regulation is heavily motivated by the recent escalation in AI-driven cybersecurity threats. Amodei explicitly references Anthropic’s own Claude Mythos Preview, noting that its ability to discover high-severity vulnerabilities across major operating systems “scrambled” the global cybersecurity landscape.

Under Anthropic’s proposed framework, securing the AI development environment is paramount. Frontier developers would be required to protect their model weights from both external cyberattackers and insider threats. Furthermore, companies must develop channels to report “model distillation attacks”—where competitors or bad actors use a primary model to train a cheaper, unaligned clone.

The Enterprise Implication: The stakes for enterprise security are twofold. First, defensive AI capabilities will become a prerequisite; as Amodei warns, attackers using frontier models to probe for vulnerabilities will outpace traditional, human-led defense. Second, enterprises that fine-tune open-weight models or host proprietary instances locally will likely face intense new compliance and infosec burdens. Treating model weights as highly classified corporate secrets will become the new industry standard.

3. Plan for Structural Labor Displacement, Not Just Efficiency

Perhaps the most sobering aspect of the announcement is Anthropic’s Economic Policy Framework. The company is publicly acknowledging that if AI achieves its predicted capabilities, it will act as a “general substitute for labor” rather than just a productivity tool.

Amodei frames this bluntly: “The key challenge in such a world won’t be incentivizing growth, but finding a way for everyone to share in the benefits”.

To back this up, Anthropic is committing $350 million to address economic disruption: $200 million for an Economic Futures Research Fund to pilot public policy solutions, and $150 million for a national fellowship program. The framework actively plans for scenarios where AI drives unemployment to 5%, 10%, or even unprecedented levels, advocating for policies like wage insurance, universal basic income, and sovereign wealth models.

The Enterprise Implication: For tech leaders and HR departments, the AI transition is about to become a labor relations minefield. The economic framework notes that companies “can choose to retrain and redeploy rather than reduce headcount,” but admits voluntary action is not a substitute for government response. Enterprises looking to integrate AI heavily should begin implementing workforce transition plans immediately. Leaders who view AI solely as a mechanism for fast cost-cutting through layoffs may soon find themselves crossways with new “pro-employment incentives” or retention tax policies proposed by advocates to slow job displacement.

What Enterprises Should Do Now

Anthropic’s announcement marks a turning point in the AI industry’s dialogue with Washington and the global market. As Amodei posted: “Many of these policy ideas have common-sense appeal across the political spectrum, and the sooner we act on them, the sooner everyone shares in AI’s benefits”.

For the enterprise, the message is clear: the era of “move fast and break things” in generative AI is closing. The era of rigorous compliance, systemic security, and complex workforce transitions is fast approaching.

To prepare for this shift, enterprises must first decouple their AI strategies from single-vendor dependencies. If a flagship model is suddenly blocked or recalled under the proposed FAA-style regulatory powers, organizations reliant on that specific API will face immediate operational paralysis. IT leaders should build multi-model architectures that allow them to swap out foundation models seamlessly, ensuring business continuity in a highly regulated ecosystem.

Second, technical decision-makers must elevate AI infrastructure to the level of critical cybersecurity. With frontier AI systems now capable of discovering high-severity software vulnerabilities at scale, the threat surface is expanding rapidly. Companies that fine-tune models or host them internally must lock down their development environments against both external and insider threats, matching the rigorous security standards Anthropic is demanding of the broader industry.

Finally, leadership teams need a proactive, rather than reactive, labor strategy. Anthropic explicitly warns against using AI solely for cost savings through layoffs, encouraging enterprises to actively seek new use cases that allow them to retain and retrain their existing workforce. As governments potentially deploy pro-employment tax incentives and wage insurance policies to slow job displacement, companies that aggressively cut headcount to fund AI adoption may find themselves on the wrong side of both public sentiment and upcoming economic regulations.