TechFlow Logo
Login/ Sign up
ETH Gas
Gwei
Fear
gas
The Eve of the Transformer: Why Did Eight Google Researchers Dare to Completely Abandon Recurrent Neural Networks?

The Eve of the Transformer: Why Did Eight Google Researchers Dare to Completely Abandon Recurrent Neural Networks?

2026.07.20
Share
Throw away the crutch, keep only attention.

Author: Robonaissance

Translated by: TechFlow

TechFlow Summary: Before 2017, everyone believed that recurrent neural networks were essential for machine translation, with attention mechanisms serving merely as auxiliary tools. Eight Google researchers bet on a radical idea: discard the crutch and rely solely on attention. The theoretical foundation for this high-stakes wager is actually a single sentence written by William James in his 1890 work The Principles of Psychology.

In 1985, Jeffrey Moran and Robert Desimone inserted a microelectrode into the visual cortex of a macaque and waited.

This monkey was trained to perform a simple yet strange task. Two stimuli would appear within the receptive field of a single neuron, one at the location cued for attention, and the other at the location cued for ignoring. The cell could detect both. The question was whether the cell cared.

It cared deeply. When the monkey attended to one of the two stimuli, the neuronal response in area V4 resembled almost as if the other stimulus did not exist at all. Responses to the unattended object, in the authors' words, were significantly suppressed. Cells in earlier stages of the striate cortex showed no such behavior. Attention is not a metaphor for what an animal is thinking. It is a measurable event at the level of a single cell, and its mechanism is subtractive. The brain does not amplify important things. It suppresses everything else.

Ninety-five years ago, without microelectrodes or monkeys, William James had articulated the exact same point in the same sequence. In Chapter XI of The Principles of Psychology, he wrote that everyone knows what attention is, then still provided a definition: the mind takes possession of one out of several simultaneously possible objects. Then came the most important sentence. James wrote that attention implies withdrawing from some things in order to deal effectively with others.

Withdrawal. Not amplification. The oldest definition in the field and the first single-cell recording agree on one thing: attention is what you subtract.

Keep this idea in mind. In roughly two thousand words, it will return wearing different symbols.

The Price Everyone Agreed to Pay

By 2016, machine translation architecture had crystallized. You took a sentence, ran it word by word through a recurrent neural network, accumulated a hidden state carrying everything seen so far, and then used that state to generate the translation word by word at the other end. The dominant variants were LSTM and gated recurrent units. They worked. Google had already deployed them into production.

This architecture had a property that was constantly discussed in the field but no longer treated as a problem. A recurrent model computes the hidden state at position t as a function of the hidden state at position t minus one. This is not an implementation detail. It is the definition. Position five must wait for position four to exist before it can be computed; position four must wait for position three.

The Transformer paper itself stated the consequences with unusual bluntness in its introduction—a section that typically relies on courtesy. The authors wrote that this inherent sequential nature precludes parallelization within training samples, and this problem becomes critical at longer sequence lengths because memory limits how many samples you can batch to compensate.

Consider what this means when the hardware in the room is a GPU. A GPU is a machine that does thousands of things simultaneously. A recurrent network is a machine that does things one after another. Running the latter on the former is like hiring an orchestra but handing them a piece written for a solo violin. Everyone is there. Most are waiting.

The field knew. There were patches. The paper cited factorization tricks and conditional computation, both of which improved efficiency, with one also improving quality. Then it delivered the sentence that set the stage for everything that followed: nevertheless, the fundamental constraint of sequential computation remains.

This is what it looks like when a community prices a cost into its worldview. No one believed recurrence was free. Everyone believed it was necessary.

Attention Arrives as a Helper

Attention entered machine translation in 2014, and it arrived as a patch.

The problem it fixed was specific and worth stating precisely, because the form of this first patch determined how the field would think about attention for the next three years. In the encoder-decoder models of the time, the encoder read the entire source sentence and compressed it into a single fixed-length vector. The decoder then generated the translation solely from that vector. Every word of a forty-word sentence had to survive inside a numerical array whose size never changed, regardless of sentence length.

At the time, Dzmitry Bahdanau at Jacobs University Bremen, working with Kyunghyun Cho and Yoshua Bengio in Montreal, named this the bottleneck. Their proposal was to stop forcing compression. Let the encoder retain a representation for every input position, and let the decoder compute a set of weights across all these representations at each output step and take a weighted sum. The model would learn where to look. Their title clearly stated the resulting capability: Neural Machine Translation by Jointly Learning to Align and Translate. Alignment—something statistical translation systems required handcrafted machinery to approximate—dropped out of the network as a byproduct of learning.

This mechanism worked and spread rapidly. A year later, Minh-Thang Luong, Hieu Pham, and Christopher Manning published a set of improvements that made it cheaper and more versatile. By 2016, attention had become standard equipment in virtually every competitive translation system.

And in every single one of those systems, it sat on top of a recurrent network.

This is the precise detail that pivoted the entire story, a detail the Transformer paper flagged in a single sentence in its introduction. The authors noted that attention mechanisms have become an indispensable part of sequence modeling, enabling dependency modeling regardless of distance. However, in all cases except a few, these mechanisms are combined with recurrent networks.

Reread it, and notice what it leaves unsaid. It doesn't say attention was underutilized. It says that for three years, the field possessed a mechanism that directly linked any two positions in a single step, regardless of how far apart they were, yet leashed it to an architecture that was fundamentally incapable of doing just that. Attention was the passenger. Recurrence was the car.

No one asked whether the car could bear the weight.

The One Who Asked

Few, anyway.

Jakob Uszkoreit never intended to study language. He stumbled into it. His father, Hans Uszkoreit, was a computational linguist who spent fifteen months in an East German prison as a teenager for protesting the Soviet invasion of Czechoslovakia. After release, he fled west, studied computer science and linguistics in Berlin, and eventually found his way to an artificial intelligence laboratory in Menlo Park, where his son was born. The family returned to Germany. Jakob attended university there, interned at Google’s Mountain View office, and landed in the translation group. He described it as finally entering the family business. He never completed his PhD.

In 2012, he joined a Google team building a system that could answer questions directly on the search page. Apple had just released Siri, and Google leadership deemed it an emergency. Looking back, Uszkoreit considers the panic unfounded. But it funneled resources behind a team dedicated to machines capable of conversational dialogue, and that is where he hit a wall.

Recurrent networks, even with long short-term memory, could not stitch long passages together. The classic demonstration is a sentence where facts stated early determine the meaning of phrases stated late, and as the model traverses left to right, it must still carry those early facts upon reaching the later phrases. LSTMs made longer spans than ordinary recurrence possible. They did not make it work at the scale Uszkoreit wanted. His self-assessment of the toolkit available to him at the time was that his applied methods were, in his words, "basically band-aids."

Around 2014, he began developing an alternative he called self-attention: letting a model translate a word by directly referencing any other part of the passage, weighting each part according to how well it clarifies the word at hand. He harbored two doubts about it. The first was that it might simply work better. The second was less obvious but proved more critical. Because self-attention observes many inputs simultaneously rather than sequentially, its geometry matched the parallel processing chips being mass-produced during the machine learning boom perfectly. The mechanism and the hardware were built for each other. No one had ever designed it this way.

The reaction was lukewarm. Abandoning recurrence meant discarding architectures the entire field had spent years refining, and according to Uszkoreit, people raised eyebrows at the suggestion. Skeptics included his father, who, in his own description, didn't fully see eye-to-eye with him at the dinner table.

He convinced several colleagues to run experiments regardless. The results were hopeful enough to publish on a small scale in 2016, using only short text spans. The paper, authored by Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit, titled "A Decomposable Attention Model," was presented at EMNLP 2016. It performed natural language inference using attention and zero recurrence.

His collaborators then moved on. The technology was good enough to deploy, so they deployed it, rolling it out across Google Search and eventually advertising. By any standard, it was a success. Uszkoreit was the only one who saw in the small experiment a signal for a large one.

This is why, when "Attention Is All You Need" appeared a year later, acknowledging in its introduction that attention is almost always combined with recurrent networks, the citation attached to that except clause was precisely that 2016 paper. The exception to the rule, sitting at entry twenty-seven in the bibliography, was the work of the person about to break it.

A Cafeteria, a Corridor, and an Espresso Machine

The Transformer was assembled from eavesdropped conversations. This is not a flourish; this is how the participants describe it.

In 2016, over lunch in the Google cafeteria, Uszkoreit heard Illia Polosukhin complain about the team building direct answers for the search page, which had to return results in milliseconds but hadn't succeeded. Uszkoreit's suggestion was self-attention. Polosukhin occasionally collaborated with Ashish Vaswani, who had arrived at Google Brain from a machine translation PhD at USC and was looking for a big problem. Vaswani's floor was right next to Polosukhin's. He overheard the idea and jumped in.

The three of them wrote a design document. The name for the title was decided upfront, based on the theory that the mechanism transforms information passing through it, and also because Uszkoreit owned two Transformers toys as a kid. The document ended with a drawing of the six of them shooting lasers at each other in the mountains. The opening sentence told readers the authors were great.

Niki Parmar joined from Google Search, where she had been building model variants alongside Uszkoreit. Llion Jones heard about self-attention from colleague Mat Kelcey and joined; Kelcey later received the project brief, told Jones he doubted it would work, and now describes it as the most incorrect prediction of his life. Łukasz Kaiser arrived from a separate language modeling effort at Google Brain, bringing his intern Aidan Gomez, an undergraduate who talked his way into a position reserved for doctoral students and wouldn't discover this mismatch for months. Kaiser and Gomez decided to fold their project into the larger one.

Polosukhin left Google in early 2017 to start his own company, which is why his affiliation does not appear in the paper's byline.

Then the team hit a wall. They built a self-attention translation model, measured it against standard benchmarks, and found it roughly flatlined against the best LSTM systems of the time. Flatlined. Not leading. A radical architecture that discarded the field's accumulated priors and returned parity was not the result. It was an expensive way to stay put.

The wall fell because one person walked through a door.

Noam Shazeer had been at Google since 2000 and carried a reputation tracing back to the company's early ad systems. He had spent five years on deep learning, became interested in language models, and felt they were nowhere near the fluency he believed achievable. Walking down the corridor of Building 1965, he passed Kaiser's workspace and overheard Vaswani and Parmar passionately debating self-attention. He found recurrent networks annoying. The proposal to replace them seemed both correct and interesting to him.

Uszkoreit's account of Shazeer's contribution deserves careful statement because it describes something the field rarely discusses. A theoretically sound mechanism, he says, often requires a handful of experienced people to implement with extreme care before showing any signs of working. Shazeer did not debug the team's code. He read the idea, wrote his own implementation from scratch, occasionally checked in with Kaiser, disappeared for a while, and returned with a working version. His colleagues described what he did using words like magic and alchemy. Jones simply called him a wizard.

The specific additions Shazeer made will be the subject of the next three articles in this series. The paper's acknowledgments footnote credits him with scaled dot-product attention, multi-head attention, and parameter-free position representations, which serve as anchor texts for Parts 2, 3, and 4. One man, in a sprint, produced three of the four components this series will dissect.

What Exactly Was the Bet

The abstract states the position in a single sentence. The authors propose a new simple network architecture based entirely on attention, completely abandoning recurrence and convolution.

The operative word there is entirely.

Removing Recurrence

Removing recurrence was not itself the radical part. Others were trying it too. Section 2 of the paper lists them: Extended Neural GPU, ByteNet, and ConvS2S, all of which replaced recurrence with convolution to compute representations for all positions in parallel. The authors acknowledged that the goal of reducing sequential computation was the very foundation of those prior works.

But convolution traded cost for parallelism, and the paper described that cost precisely. In these models, the number of operations required to relate any two positions grows with distance: linearly for ConvS2S, logarithmically for ByteNet. Still not free. Connecting two words at opposite ends of a long sentence remained expensive, and the paper cited the established conclusion: longer paths make long-range dependencies harder to learn.

Self-attention compressed this issue to constant order. From any position to any other, in one step, regardless of distance. Table 1 displays the three layer types side by side, and the critical column is maximum path length: recurrence is O(n), convolution is O(log n), and self-attention is O(1).

This was the bet. Not that attention is useful—the industry already believed that. The bet was that the value of constant path length was so great that it could justify abandoning all other structural priors the industry had accumulated and still win.

The paper did not hide what it was sacrificing. Just two sentences after the claim of constant operations, there was an acknowledgment, rarely preserved in later summaries: the constant cost comes at the expense of reduced effective resolution, because attention averages over weighted positions. Multi-head attention existed specifically to counteract this. The paper tells you on its second page that its core mechanism introduces blurring, and that one of its most celebrated components is essentially a patch. Part 3 will return to this.

There is a second cost, recorded in Table 1 and defended in Section 4. The paper treats it as a reasonable trade-off, though the following decade would treat it as a core issue in AI infrastructure. The complexity per layer of self-attention is n squared times d. The paper's defense is that for inputs of sentence length in translation, n is typically smaller than d, which was true in 2017, but ceases to hold when someone wants to feed a book to the model. Part 7 will settle this tab.

Twelve Hours

The deadline was May 19th for a December conference. The final two weeks were spent in Building 1965, partly because some team members had desks there, and partly because the espresso machine in 1965 was better than the one in 1945. Intern Gomez described a period of relentless debugging, hardly any sleep, and systematically removing components to see if the model still worked without them. What is now called the Transformer is largely the residue of that process: the parts that couldn't be removed.

Then the numbers came in.

The base model trained for 100,000 steps on a single machine using eight NVIDIA P100 GPUs. Wall-clock time: 12 hours. This model surpassed all previously published English-to-German models and ensembles. The big model trained on the same eight-GPU machine for three and a half days, hitting 28.4 BLEU, beating the previous best results (including ensembles) by a full two points. Uszkoreit opened a bottle of champagne he had kept in his mountain expedition truck to celebrate.

Now look at Table 2, the quietest and most lethal table in the paper. It lists the training cost in floating-point operations. Competitive ensemble systems at the time ranged from 1.1×10²¹ down to 7.7×10¹⁹. The Transformer big model sat at 2.3×10¹⁹, and the base model was an order of magnitude cheaper. The new state of the art was created by the cheapest system in the table.

A field that had spent years trading scale for quality was shown a model that traded structure for quality. And because this structure maps cleanly onto parallel hardware, the exact same architecture would later become the ideal vehicle for spending compute, once compute became worth spending. The 2017 results read as a story of efficiency. Looking back, it is a story of capacity disguised as efficiency.

About two minutes remained before submission. Parmar said the English-to-French metrics arrived roughly five minutes before the deadline; she sat in the small kitchen waiting for the final number. This perhaps explains a minor inconsistency that persists in the published paper: the abstract and Table 2 report the English-to-French score as 41.8, while Section 6.1 reports 41.0. One of the most influential papers in modern machine learning contains a contradictory number because the experiment finished after the main text was written.

The title was settled a few nights before the cutoff. Welshman Jones pointed out that the team rejected industry convention and chose a single technology, and that the Beatles had already written the phrase for them. He said the idea took about five seconds, and he didn't expect anyone to actually use it.

They did.

Back to the Monkey

What the Transformer computes for each position is a weighted sum over all other positions, with weights produced by softmax. Softmax is not merely sorting. It suppresses. Raise one score, and all other scores in the distribution drop, because the sum is fixed at 1. This mechanism at the heart of the architecture is competitive; its output is defined jointly by what it drives to zero and what it elevates.

This is exactly what Moran and Desimone found in V4 neurons when two objects were present in their receptive fields. This is exactly what James described as withdrawing from some things to effectively deal with others.

To claim that the 2017 authors implemented James would be overstating it. They did not. Queries, keys, and values come from information retrieval, not psychology, and Part 2 will trace that lineage correctly. But the words they chose were not accidental; the field has been refining the underlying concept for a century. When Anne Treisman and Garry Gelade published feature integration theory in 1980, they argued that features are registered in parallel early on, and attention is the operation that binds them into objects. Parallel registration, followed by selective binding. This is a description of the Transformer layer, written thirty-seven years ago by researchers of human vision.

Whether this convergence reflects a fundamental truth about intelligence or a lexical accident will be the final question of this series. Part 8 will attempt to answer it.

Still, there is a better ending, which occurred during the poster session of the December 2017 conference. The room packed for four hours. Security eventually had to clear the space. At some point that night, a man approached the poster and told the authors he was impressed by the work. That man was Sepp Hochreiter.

Hochreiter co-invented the long short-term memory network. He came to congratulate the people who had just rendered it obsolete.

The bet. Recurrence is not necessary for sequence modeling. An architecture built entirely on attention, devoid of any recurrent or convolutional layers, can match and surpass the optimal performance. The constant path length between any two positions holds more value than all the structural priors abandoned.

The payoff. It won, and faster than the authors expected. Within roughly two years, this architecture replaced recurrent models throughout natural language processing. Every frontier language model built since is its descendant. The ambition the paper declares in its own conclusion—to extend this method to other modalities and make generation less sequential—materialized into an industry.

Status. Settled, with an asterisk. This bet was won on a trade-off the paper explicitly made and priced correctly in 2017: exchanging quadratic cost relative to sequence length for constant path length. Every long-context problem plaguing AI today represents interest payments on that trade-off. Part 7 will collect them.