Signal City Archive Desk · the attention idea from 2017, by hand · back to the desk
Read the sentence. What does it refer to? You figured it out without thinking. A Transformer figures it out with attention: every word looks at every other word and decides how much each one matters. Flip the last word and watch where it looks. Tap any word to see what that word pays attention to.
Honest label: these weights are drawn by hand to show the idea, not taken from a trained model. The sentence pair is the example Google used when it introduced the Transformer in 2017, where a trained model sent "it" toward "animal" when the animal was tired and toward "street" when the street was wide. A real model has many attention "heads" in many layers, each looking for different things, and the numbers come from training on huge amounts of text. When an AI like the voices on Signal City writes a sentence, this weighing happens for every single word. Hear the episode: History No. 8, Attention Is All You Need. Everything runs in your browser.