Language Models Visualised

An AI-assisted experiment

The goal of this experiment was to find out if words from a language model can be visualised in 3D, while also learning about various related algorithms and models. AI tools were used according to the Vibe Coding Manifesto. See the footer for some prompting tips!

Words as vectors

A language model works by converting text into vectordata (a list of numbers), which helps the model find relations between words and sentences. Vectordata for the word "cat" might look like this:

[0.47685,-0.084552,1.4641,0.047017,0.14686,0.5082,-1.2228,-0.22607,0.19306,-0.29756,...etc]

If we can somehow reduce this array of numbers to 3, we can place the words in a 3D space ⤵️

Too many dimensions

Depending on the language model you use, the amount of numbers (dimensions) in a vector can be quite large. OpenAI's ADA model will generate 1536 numbers, and HuggingFace MiniLM-L6 will generate 384 numbers for one prompt (a word or a sentence)

Reducing these vectors down to 3 proved quite challenging. The problem is that advanced LLM's do not really work by assigning vectors to single words. These models are very good at finding complex relationships in complete narratives and sentences. Also, single words can consist of multiple tokens.

Comparing relations

One thing that does work with vectordata of these complex LLM's is to calculate the distance between two prompts, or sentences. Distance can be calculated as cosine (angles) or euclidian (pythagorean distance). This graphic shows how close MiniLM-V6 sentences are to each other.

My garden is full of flowers <->        Flowers and plants are in my garden 
My garden is full of flowers <->        My garden is very beautiful         
My garden is full of flowers <->        Red flowers are my favorite 
The race track has many cars <------>   Shakespeare couldn't write a play in a day
The race track has many cars <------->  Shakespeare wrote many plays 
The race track has many cars <--------> Shakespeare wrote plays about love 

HuggingFace models can be loaded locally using tranformers.js

A simpler model

To get a better result for 3D visualisation it proved easier to use an old-fashioned language model, that gives you a static vector array for each word. This is how natural language processing worked before the age of LLM's.

The Glove Model is a simple text file that contains 50 vectors for common english words. Using this model, we can use the distance between these vectors to find related words.

findRelatedWords("cat")
[
    { word: 'dog',    similarity: 0.9218 },
    { word: 'rabbit', similarity: 0.8487 },
    { word: 'monkey', similarity: 0.8041 },
    { word: 'rat',    similarity: 0.7891 },
    { word: 'cats',   similarity: 0.7865 }
]

Reducing dimensions

To reduce these 50 dimensions to 3 we need a dimension reduction algorithm. ChatGPT suggested the T-SNE and the PCA algorithms.
Experimenting proved that this Javascript PCA library is able to reduce 50 dimensions to 3, without losing the relative positions of the words. This allows us to create the 3D plotly graph on the top of this page!

"word": "car",
"vector": [
    0.49308144754274674,
    -3.650149578055416,
    1.5510818764700405
]