Transformer Architecture and Attention

The original Transformer introduced an attention-based encoder-decoder architecture for sequence transduction. Many later language models use related components, but not every modern model has the original encoder-decoder structure.

#Encoder and decoder

The encoder produces contextual representations of input tokens. The original decoder generates output using masked self-attention and attention over encoder representations. Decoder-only language models use a different arrangement, so do not map every product directly onto the original diagram.

#Self-attention

Each token is projected into query, key and value vectors. Attention weights determine how strongly a position combines information from other permitted positions.

textCopy
Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) V

The scaling term helps control score magnitude. A causal mask prevents a generating position from attending to future tokens. Multi-head attention learns several such projections and combines their outputs.

#Other components matter

Position information helps represent order. Feed-forward layers transform each position's representation. Residual connections and normalization help training. Attention alone does not describe the whole architecture.

Parallel training over sequence positions was an important advantage over recurrent processing. Autoregressive generation still produces dependent tokens sequentially; it does not generate every word independently at once.

#Practical consequences

Long inputs can increase computation and memory requirements. Different architectures and implementations change those costs. Attention weights are not a general proof of faithful explanations, and the architecture does not guarantee factual answers.

Use this chapter to understand the mechanism, then evaluate a selected model on actual application tasks.

Original paper: Attention Is All You Need↗