Review of “Attention Is All You Need”

Overall assessment

Overall the work is a high‑impact, well‑motivated contribution that is ready for publication after focused revisions addressing reproducibility, baseline fairness, presentation clarity, and statistical reporting.

Strengths

The paper introduces a clear, novel architecture (the Transformer) with strong empirical results on standard MT benchmarks, detailed design and ablation analyses, and released code that together enable broad reproducibility.

Points to improve

The manuscript needs more complete reproducibility details (explicit data splits, full tokenization/BPE and BLEU computation protocols), clearer baseline comparisons, and statistical reporting of result variability to fully validate claimed improvements.

Recommendations

• Specify dataset splits and preprocessing in full, naming exact validation and test splits and the held-out test sets reported; fully document preprocessing and tokenization pipelines (tokenizer, vocabulary, casing, normalization) used for evaluation.
• Fully describe every baseline and ensure fair comparisons by matching preprocessing and training budget; include implementation details and hyperparameters for each baseline.
• Report a complete evaluation protocol and statistical analysis: list metrics and selection criteria, checkpointing frequency, number of independent runs/seeds, uncertainty measures (std or CIs), and formal significance tests (with p-values); specify software and versioning used for training/evaluation.
• Make numerical results self-contained and consolidate methodological details: state concise numeric takeaways in the text for each figure/table and move hyperparameters and procedural notes into a single methods paragraph or boxed note.
• Resolve numeric inconsistencies and temper broad superiority claims by fixing contradictory reported values (e.g., English–French BLEU) and specifying conditions under which claims hold (sequence length, tokenization, model scale).

Paper summary

This document introduces the Transformer, a novel network architecture that relies entirely on attention mechanisms for sequence transduction tasks, eliminating the need for recurrence and convolutions. The Transformer is evaluated on machine translation tasks, demonstrating superior performance in translation quality and training efficiency compared to traditional models. It achieves state-of-the-art BLEU scores on both English-to-German and English-to-French translation tasks while requiring significantly less training time and resources.

Main claims

• The Transformer model achieves superior performance in machine translation tasks compared to existing models.
• The Transformer is based solely on attention mechanisms, eliminating the need for recurrent and convolutional layers.
• The model establishes new state-of-the-art BLEU scores on both English-to-German and English-to-French translation tasks.
• The Transformer generalizes well to other tasks, such as English constituency parsing.

Abstract

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 Englishto-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data. * Equal contribution. Listing order is random. Jakob proposed replacing RNNs with self-attention and started the effort to evaluate this idea. Ashish, with Illia, designed and implemented the first Transformer models and has been crucially involved in every aspect of this work. Noam proposed scaled dot-product attention, multi-head attention and the parameter-free position representation and became the other person involved in nearly every detail. Niki designed, implemented, tuned and evaluated countless model variants in our original codebase and tensor2tensor. Llion also experimented with novel model variants, was responsible for our initial codebase, and efficient inference and visualizations. Lukasz and Aidan spent countless long days designing various parts of and implementing tensor2tensor, replacing our earlier codebase, greatly improving results and massively accelerating our research. † Work performed while at Google Brain. ‡ Work performed while at Google Research.