LLMs & Generative AI

Transformer Architecture

A Transformer processes tokens through repeated attention and feed-forward blocks. Attention lets each token mix information from other tokens, which made large-scale language modeling dramatically more effective.
  • Self-attention computes token-to-token relevance
  • Multi-head attention learns multiple relationship types
  • Position information is added explicitly
  • Feed-forward layers transform each token representation
  • Residual connections and normalization make deep stacks trainable
  • Decoder-only Transformers predict the next token
One Transformer block
The block repeats many times; scale comes from stacking and training on huge corpora.
Important moving parts
PartJob
EmbeddingsRepresent tokens as vectors
Position encodingExpose token order
AttentionRoute information between tokens
Feed-forward networkTransform token representations
ResidualsPreserve signal across depth
Layer normalizationStabilize training
Sources
  • Attention Is All You Need — Attention Is All You Need
  • Hugging Face LLM Course — How do Transformers work?
  • Dive into Deep Learning — Attention Mechanisms and Transformers