Query, Key, Value

Before diving into Query, Key, and Value, let's recall where we are. Each word in "The cat sat down" has been converted to an 8-dimensional embedding vector and combined with positional encodings.

These input vectors contain information about:

• •What the word means (from the learned embedding)
• •Where the word sits in the sequence (from the positional encoding)

But these vectors don't yet know about context. The embedding for "cat" is the same whether the sentence is "The cat sat down" or "The cat chased the mouse." Self-attention will fix this by letting each token gather information from all other tokens.

The question is: how exactly does a token decide what to look for and where to find it? The answer is three learned projections: Query, Key, and Value.