Playground · Self-Attention

Attention Lab.

The mechanism behind every transformer, computed for real in your browser: token embeddings, positional encodings, then softmax(QK^T / √d) — the same equation from Attention Is All You Need. Type a sentence, click a token, and see where its attention flows. The projections are randomly initialized (this head is untrained — that's what "re-roll" reshuffles); the mechanism is the real thing. Tokens with shared character patterns attend to each other; slide toward position and attention becomes locality-biased instead.

Sentence → tokens0 tokens

Attention flowclick a token to pin it

value mix — each token's colour blended by its attention row (this is what V-aggregation does)

Head controlsd=48 → dₖ=16

softmax temperature 1.0

content ◂▸ position 25%

—row entropy (bits)

—strongest link

Attention matrix · rows attend to columns

← MLP Lab Conv Lab → LLM Guide →