Executive Overview
Back in the summer of 2017, a team of Google AI researchers published a landmark paper titled "Attention Is All You Need." Within its pages, they introduced an entirely novel neural network architecture known as the transformer. Designed specifically to excel at processing long, sequential strings of data, the transformer quickly proved to be exceptionally adept at parsing, translating, and generating human language.
Nearly a decade later, transformers serve as the beating heart inside every major large language model (LLM) on the market. "The entire AI industry is built on transformers," says Justin Dangel, cofounder and CEO of the AI startup Subquadratic. "They are one of the most important innovations in the history of computer science, and they’ve changed the world."
Yet, despite their ubiquity, transformers are beginning to show their age. Many of the most celebrated recent breakthroughs in generative AI—such as the emergence of advanced "reasoning" models and the push toward handling massive context windows—are not clean, elegant evolutions of that original 2017 core. Instead, they are increasingly elaborate engineering workarounds designed to patch over the architecture’s fundamental flaws.
A growing faction of computer scientists, researchers, and venture-backed engineers are now asking a pivotal question: What comes next?
While large language models are here to stay, the foundational blueprint of how they are built is entirely up for grabs. Dubbing this forthcoming paradigm "LLMs+" in their annual technology outlook, industry analysts recognize that a new crop of ambitious startups is stepping up to challenge the status quo. While many of these ventures will inevitably stumble, they have everything to gain and far less to lose than the monolithic incumbents currently leading the pack.
Detailed Chronology: The Evolution and Bottlenecks of AI Architecture
To understand why the tech sector is actively seeking alternatives to the transformer, it is necessary to trace how we arrived at the current architectural bottleneck.
- June 2017: Google publishes "Attention Is All You Need," introducing the transformer architecture and its cornerstone innovation: dense attention.
- 2018–2022: The transformer rapidly displaces older recurrent neural networks (RNNs) and convolutional structures. It becomes the bedrock for early generative pre-trained transformers, revolutionizing natural language processing and setting off the modern generative AI boom.
- 2023–2025: As model parameter counts skyrocket, the computational costs associated with dense attention become staggering. Companies discover that scaling transformers requires exponentially more electrical power and capital expenditure. Reasoning models are introduced, relying on resource-intensive "scratch pads" (chains of thought) that further exacerbate memory and compute strains.
- 2024–2026: A wave of specialized startups emerges to address these hardware and efficiency walls. Innovators begin experimenting with sparse attention mechanisms, power retention, liquid neural networks, diffusion-based text generation, and state-space models.
Supporting Context & Metrics: The Cost of Dense Attention
The fundamental strength of the transformer—and simultaneously its greatest curse—lies in a mechanism called dense attention. This process encodes the semantic meaning of a block of text into a vast series of numbers. To achieve this, the model compares every single word, or token, in a text against every other token via an intensive matrix of multiplications.
While dense attention captures contextual meaning with remarkable precision, it introduces a brutal mathematical scaling problem. As text inputs grow longer, the required computations explode quadratically. For instance, a document containing 10,000 words may require a standard transformer to execute upwards of 50 million individual multiplication operations.
This quadratic scaling is the primary culprit behind the staggering power consumption of modern LLMs. The financial and environmental tolls are mounting rapidly:
- Capital Expenditure: OpenAI alone projected a staggering $50 billion investment in computing power, according to remarks made by company president Greg Brockman.
- Global Energy Strain: According to projections from the International Energy Agency (IEA), global electricity consumption by data centers is on track to double by 2030, driven largely by the surging compute demands of generative AI workloads.
Furthermore, transformers struggle natively with the complex tasks modern applications demand. Because they process text sequentially, token by token, expanding their "context window" (the amount of data they can hold in active memory at once) is intensely difficult and resource-heavy. Yet, if AI agents are to execute sophisticated workflows, they must be able to ingest massive troves of data simultaneously—ranging from entire codebases and corporate libraries to the live outputs of other autonomous agents.
When reasoning models are deployed, they function by writing iterative notes to themselves—a "chain of thought"—and subsequently reading them back. This recursive process piles even more data onto an already overburdened context window. As models grow larger and more capable, the transformer has officially transformed from an engine of progress into a severe performance bottleneck.
Four Radical Solutions to the Transformer Problem
In response to these limitations, engineers are advancing four distinct architectural alternatives that promise to make LLMs faster, exponentially more energy-efficient, and fundamentally smarter.
01: Rethinking Attention (Sparse Attention & Power Retention)
The most direct way to alleviate the computational drag of transformers is to reform attention mechanics.
- Sparse Attention: Instead of calculating relationships across every possible token pairing, sparse attention restricts calculations to targeted subsets of words. While early sparse attention models struggled to match the semantic accuracy of dense attention, Miami-based startup Subquadratic claims to have achieved a breakthrough. Their model, SubQ, dynamically determines on the fly which tokens are relevant and which can be ignored, reportedly rivaling mainstream LLMs in search and coding tasks.
- Power Retention: Operating out of San Francisco, Manifest AI has bypassed attention entirely, developing a mechanism dubbed "power retention." Rather than forcing an LLM to retain every single token in its context window, power retention maintains a dynamic, rolling summary. As new data streams in, lower-relevance information is pruned. Manifest AI has successfully retrofitted open-source models like the StarCoder coding model (renamed PowerCoder) and released Brumby, a model said to rival Alibaba’s popular Qwen architecture.
02: Making Models Smaller and More Flexible (Liquid Neural Networks)
Rather than entirely discarding transformers, MIT spinout Liquid AI bridges the gap. Founded by Ramin Hasani, the Cambridge, Massachusetts-based firm pairs transformers with liquid neural networks to build Liquid Foundation Models (LFMs).
Drawing inspiration from biological nervous systems (specifically worm brains), liquid neural networks allow models to continuously adapt their behavior to incoming data post-training—a feat impossible with static transformers. Liquid AI’s hybrid architectures typically consist of a 20% transformer and 80% liquid neural network blend, optimized via an automated designer AI. These models are compact enough to operate on low-power edge hardware, such as a $50 Raspberry Pi or the embedded chips inside Mercedes-Benz vehicles, while matching the performance of models four times their size.
03: Generating Text All at Once (Diffusion LLMs)
Standard LLMs produce text sequentially, mirroring human speech. For computers, however, token-by-token generation is inherently inefficient.
Inception, a Palo Alto-based startup, is applying diffusion models—the math traditionally used to generate high-definition images and videos from random static—to text generation. Led by Stanford researcher Stefano Ermon, Inception trains its models to ingest a randomized string of text tokens and iteratively refine them into coherent sentences all at once.
While adapting diffusion from continuous pixel spaces to discrete language tokens required entirely new mathematics, Inception’s latest model, Mercury 2, reportedly matches the performance of older OpenAI GPT-4 architectures while operating up to 10 times faster. Google is similarly exploring this frontier with its prototype Diffusion Gemma project.
04: Moving Beyond Words (State-Space Models & Non-Linguistic Reasoning)
Perhaps the most radical departure from traditional LLM design comes from Palo Alto startup Pathway. Co-founded by Zuzanna Stamirowska, Pathway argues that forcing artificial intelligence to route all cognition through human language is a fundamental architectural error.
To prove their point, Pathway developed an experimental model called Dragon Hatchling. When pitted against a rigorous benchmark of over 250,000 highly complex Sudoku puzzles, Dragon Hatchling successfully solved more than 97% of them—outperforming elite mainstream LLMs that failed to crack a single one.
Pathway achieves this by replacing the transformer’s attention mechanism with state-space models. Instead of encoding data word by word, state spaces compress information into abstract mathematical representations. This allows the model to perform non-linguistic reasoning tasks—akin to how a human grandmaster visualizes chess moves or a mathematician experiences a sudden "eureka" moment without consciously reciting words.
Official Statements and Industry Insights
The race to redefine AI architecture has sparked intense debate across the research community regarding the future trajectory of machine intelligence.
-
On the limits of current scaling:
"The entire AI industry is built on transformers. They are one of the most important innovations in the history of computer science, and they’ve changed the world. But as LLMs get bigger and better, transformers have become a bottleneck. The technology’s key strength is now a limitation." — Justin Dangel, Co-founder and CEO, Subquadratic
-
On biological efficiency and hybrid architectures:
"Your brain is an AGI system, you know, and it operates with 20 watts of power. How is it possible? We can get a lot more innovative." — Ramin Hasani, Co-founder and CEO, Liquid AI
-
On the economics of next-gen intelligence:
"Ultimately, the currency is going to be intelligence per dollar. We’re bullish about this [diffusion] approach because it’s the one that is going to scale up." — Stefano Ermon, Co-founder and CEO, Inception, and Researcher at Stanford University
-
On breaking free from linguistic constraints:
"Transformers are an engineering convenience that we fell on. It started a religion, but it’s silly to think that a breakthrough won’t happen again. The hope for AI is not to solve Sudoku; it’s to cure cancer. There’s not a book for that." — Zuzanna Stamirowska, Co-founder and CEO, Pathway
Future Outlook
The AI industry stands at a fascinating crossroads. While transformer-based LLMs command trillions of parameters and dominate enterprise budgets today, their insatiable appetite for computational power and electrical infrastructure is testing the limits of physical reality.
The next era of artificial intelligence—the era of LLMs+—will likely not be defined by a single, monolithic architecture. Instead, the future belongs to a diverse ecosystem of hybrid systems: sparse attention models capable of parsing massive documents efficiently, liquid neural networks operating locally on edge devices, diffusion models spitting out rapid generations of text, and state-space architectures capable of abstract mathematical reasoning.
As startups chip away at the foundational pillars of the 2017 transformer breakthrough, one reality remains clear: the race to build faster, cheaper, and genuinely smarter AI has only just begun.