Beyond the Prompt: How AI Language Models Struggle and Succeed in 20-Year Strategic Simulations

Executive Overview

In the rapidly evolving landscape of artificial intelligence, foundational language models (LLMs) have successfully proven their competency in executing bounded, short-term tasks. From debugging code snippets to drafting concise emails and summarizing dense research papers, today’s AI agents operate with commendable reliability. However, a glaring question has loomed over the artificial intelligence research community: Can these systems sustain effective, strategic decision-making over long horizons, where actions possess cumulative, cascading consequences and the environment dynamically responds to their choices?

To answer this pressing question, a team of researchers introduced FM-Bench (Football Management Benchmark), a rigorous evaluation framework designed to test the limits of artificial intelligence in a complex, multi-decade management simulation. Released in August 2026, FM-Bench subjects frontier LLM agents to the ultimate test of endurance, strategic foresight, and resource allocation: running a professional football club across a simulated 20-in-game-year span.

The results of this benchmark challenge much of the conventional wisdom surrounding modern AI development. Scale, price, and commercial vendor affiliation do not reliably predict performance. Furthermore, human intuition and competence, when pitted against these systems in a first-play scenario, struggle to keep pace, with human testers landing near the bottom of the model leaderboard. Most intriguingly, the benchmark reveals that what separates high-performing models from failures is not raw computational power or token expenditure, but nuanced managerial behavior—such as cash flow management, foresight in contract renewals, and adaptive horizon planning.

As the artificial intelligence industry shifts its gaze from static question-answering to autonomous, long-horizon agentic workflows, FM-Bench provides a critical window into the cognitive and behavioral ceilings of contemporary frontier models.

Detailed Chronology & Structural Architecture

To comprehend the significance of FM-Bench, one must examine the intricate architecture of the simulation environment developed by lead researcher Chongyang Gao and the broader Analogy AI team. Operating a football club is not a trivial undertaking; it mirrors the complexities of corporate governance, macroeconomic forecasting, and personnel management under extreme uncertainty.

The Mechanics of FM-Bench

In FM-Bench, an LLM agent is handed the reins of a football club and tasked with navigating it through 20 grueling in-game years. To accomplish this, the agent must interact with the simulation via 26 distinct tools, executing roughly 340 to 400 decision stops over the course of the simulation.

The responsibilities mirror those of a real-world sporting director or chief executive officer:

• Squad Building and Financial Parity: The agent drafts an initial squad working under the exact same budgetary constraints as every other rival club in the simulation.
• The Transfer Market: Models must scout, negotiate, and execute player trades, navigating a complex market ecosystem.
• Contract Negotiations: Managing player egos, wage structures, and contract expirations before deadlines loom.
• Capital Allocation: Deciding when and where to invest in physical infrastructure, training facilities, and youth academies—investments that feature long payback periods.
• Tactical Execution: Setting matchday lineups and formations week in and week out.
• Boardroom Politics: Answering directly to a virtual board of directors that possesses the authority to sack the manager if performance dips below acceptable thresholds.

Crucially, FM-Bench eliminates human bias from the evaluation phase. There are no subjective LLM judges or human raters evaluating the "creativity" of a transfer strategy. Instead, a deterministic engine accumulates every single decision, match result, financial ledger, and developmental milestone over the 20-year span into one final, objective score.

The Dual Tracks: Solo and Arena

The research team deployed two distinct testing tracks to evaluate 15 frontier language models:

1. The Solo Track: Each of the 15 models independently plays out a 20-year career against a frozen, scripted world. This track isolates the model’s intrinsic decision-making capabilities without interference from competing AI agents.
2. The Arena Track: In a groundbreaking move for AI evaluation at this scale, the Arena places all 15 models—alongside a scripted baseline anchor—into one shared, dynamic 20-year world. Here, the models actively compete against one another in the same transfer market, vying for the same talent, negotiating against each other’s bids, and fighting for league supremacy.

Across three separate random seeds, the test yielded striking participation metrics: all 15 models successfully completed every single 20-year horizon. By contrast, blind scripted baselines routinely collapsed, failing to survive their simulation horizons due to financial ruin or boardroom dismissals.

Supporting Context & Metrics: Behavioral Insights

When the smoke cleared and the deterministic engine calculated the final tallies, the leaderboard offered several paradigm-shifting revelations.

The Performance Hierarchy and the Human Factor

At the apex of the Solo board (in terms of mean score) and dominating the Arena track was claude-fable-5. However, the path to victory was far from a monopoly. The title of league champion rotated dynamically among ten different models across various simulation runs, indicating that multiple architectures possess viable winning strategies.

More surprisingly, the correlation between standard AI industry metrics and FM-Bench success proved negligible:

• Scale does not guarantee victory: The largest parameter models did not automatically dominate.
• Price is irrelevant: Cost-per-token had no predictive power over managerial competence.
• Vendor brand loyalty fails: Superiority was scattered across different labs rather than concentrated under a single developer.

Even more humbling for biological intelligence, when human testers attempted the simulation for their first play, the best human performance landed squarely at the very bottom of the model leaderboard, highlighting the sheer cognitive load required to track hundreds of interdependent variables over a simulated two-decade career.

Behavioral Determinants of Success

The researchers moved beyond simple scoring to analyze how the models achieved their outcomes. They measured six core behavioral capabilities underpinning the final score, uncovering stark contrasts between high- and low-performing systems:

1. Cash Flow and Liquidity: Top-tier models avoided leaving capital idle, keeping funds actively invested while maintaining sufficient liquidity to avert cash-crunch crises.
2. Horizon Awareness: As models approached the 20-year finish line, higher-scoring systems intelligently curtailed slow-payoff investments (such as long-term youth academy overhauls) because the return horizon had expired, pivoting instead to short-term asset preservation.
3. Proactive Administration: Successful managers opened contract renewals well before deadlines approached, mitigating the risk of losing star players to free agency. Conversely, weaker models repeatedly ran into administrative crunches.
4. Token Spend Irrelevance: Interestingly, the total volume of tokens spent by a model during reasoning and tool-use had zero correlation with its final score. Verbose reasoning did not equate to better strategy.

Persistent Blind Spots in Machine Cognition

Despite their triumphs, the models exhibited deep, systemic cognitive limitations that expose the boundaries of current LLM agent architectures:

• Inability to Learn Market Pricing: Across hundreds of rejected transfer bids where models systematically overbid or underbid, no model successfully learned the market’s hidden price equilibria. They routinely repeated valuation errors without adapting their internal heuristics to market clearing rates.
• The Memory Management Paradox: When tasked with maintaining self-managed memory across decades, models failed uniformly in one of two opposite modes:
  • The Infinite Archive: The memory log grew monotonically, eventually clogging the context window with obsolete trivia from year two while drowning out critical year-nineteen context.
  • The Tabula Rasa: The model wiped its strategic plan clean at the start of every single season, continuously reinventing the wheel and abandoning multi-year tactical roadmaps.

Future Outlook

The release of FM-Bench and its accompanying open-source codebase (github.com/Analogy-AI/fm-bench) marks a foundational turning point in how the AI community evaluates agentic systems. As artificial intelligence transitions from isolated chat interfaces to autonomous digital workers capable of steering enterprises, managing supply chains, and executing long-term financial strategies, static benchmarks will no longer suffice.

FM-Bench demonstrates that true agentic reliability is not merely about generating syntactically correct code or executing a single tool call; it is about maintaining state integrity, temporal awareness, and adaptive equilibrium in a chaotic environment where actions have delayed consequences.

For AI developers, the findings point toward clear engineering imperatives. Future model iterations must look beyond scaling laws and token budgets, focusing instead on robust architectural solutions for long-horizon memory management, meta-learning from failed market interactions, and temporal reasoning. Until these cognitive bottlenecks are resolved, AI agents may conquer the realm of bounded tasks, but navigating the unpredictable, multi-year tides of a complex world will remain an ongoing test of machine resilience.