Embeddings: text as numbers that carry meaning

An embedding model turns text into a list of numbers, a vector, so texts with similar meanings get similar vectors even when they share no words.

Example

from llama_index.core import Settings
from llama_index.embeddings.huggingface import HuggingFaceEmbedding

Settings.embed_model = HuggingFaceEmbedding(model_name="sentence-transformers/all-MiniLM-L6-v2")

vector = Settings.embed_model.get_text_embedding("How do I get my money back?")
print(len(vector))
print([round(x, 3) for x in vector[:5]])

Settings holds the defaults LlamaIndex uses everywhere, and embed_model is the model that makes vectors. This one returns 384 numbers for any text. Individual numbers mean nothing on their own; what matters is how close two vectors are.

Measuring closeness

Example

import numpy as np


def similarity(a, b):
    va = np.array(Settings.embed_model.get_text_embedding(a))
    vb = np.array(Settings.embed_model.get_text_embedding(b))
    return float(va @ vb / (np.linalg.norm(va) * np.linalg.norm(vb)))


question = "How do I get my money back?"
for text in ["You can get a full refund within 30 days of delivery.", "Standard delivery takes 3 to 5 working days.", "The LMP-204 desk lamp has a known cable fault."]:
    print(f"{similarity(question, text):.3f}  {text}")

Cosine similarity compares the direction of two vectors: close to 1 for the same meaning, near 0 for unrelated text. The refund sentence scores highest for the money-back question with no word in common, which is exactly what keyword search could not do.

@ multiplies two vectors into a single number, and dividing by their lengths keeps long and short texts comparable. LlamaIndex does this for you from the next lesson on.

Local model, real results

The first run downloads the model, about 90 MB, from Hugging Face; after that it loads from your disk. Hosted embedding models from OpenAI, Cohere and others work the same way through their own LlamaIndex integration packages.

Try it yourself

Compare "My parcel is late" with the three sentences.
Compare two sentences that mean the opposite, such as "refunds are allowed" and "refunds are not allowed". What does that tell you?
Embed the same sentence twice and check the similarity.

PreviousWhy retrieval: a model does not know your documents

You understood something today that you didn't yesterday.