Interactive Learning

How LLMs Learn Word Embeddings

An interactive walkthrough of prediction, backpropagation, and semantic learning

Step 1: Initialize the Dictionary

An LLM starts with a fixed vocabulary of words. Each word has a unique ID and an embedding vector (a list of numbers that will learn to represent meaning).

Important: At the beginning, all embedding vectors are filled with random numbers. These random numbers have no meaning yet—that comes later through training.

In this demo, our dictionary has 6 words. In real LLMs, this could be 50,000+ tokens.

Vocabulary Dictionary

ID 0king

ID 1queen

ID 2man

ID 3woman

ID 4apple

ID 5orange

Vector Visualization

will appear in step 3

The Big Picture

Start with random embeddings → Give model text examples → Model predicts next word → Measures error → Adjusts embeddings to reduce error → Related words naturally cluster together → Meaning emerges from math!