Generating Embeddings in Practice

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Examples on this page target Python 3.12 with sentence-transformers 6.1 and Ollama 0.34, plus LangChain / langchain-core 1.4 / 1.6, LangChain4j 1.20.1 and Spring Boot 4.1 with Spring AI 2.0.1. Embeddings covers what an embedding is and how the models behind it are trained; this page is about the operational side — picking a concrete model and turning text into vectors reliably, in a pipeline that indexes and queries the DocsAssistant corpus.

Choosing a model

The Massive Text Embedding Benchmark (MTEB) leaderboard ranks embedding models across retrieval, clustering, classification and reranking tasks and is the standard starting point for picking one. Reading it well means trading off four things at once, not just picking the top score:

  • Dimension — higher-dimensional embeddings usually capture more nuance but cost more to store and search (a 1536-dim vector is 4x the storage and index memory of a 384-dim one at the same precision). Matryoshka models (Embeddings) let you truncate this after the fact.

  • Quality — MTEB’s retrieval-task scores (nDCG@10 on its BEIR-derived datasets) are the closest proxy for RAG relevance, but a model’s ranking can still shift on your own domain — a small evaluation on your own golden set (Evaluating RAG Systems) is worth more than a leaderboard rank.

  • Latency — a hosted API call adds network round-trip time on top of model inference; a local model’s latency depends on hardware (CPU vs. GPU) and batch size. Both matter more for the query path (one vector, interactive) than for ingestion (many vectors, batchable).

  • Licence — open-weight models (Apache 2.0, MIT) can be self-hosted and fine-tuned freely; some research-licensed models restrict commercial use. Hosted APIs replace a licence question with a data-residency and pricing one.

Maximum sequence length is a hard cutoff, not a trade-off: text beyond it is truncated (silently, in most clients) before embedding, so a chunking strategy (Document Ingestion & Chunking) must keep chunks under the model’s limit — typically 256 to 512 tokens for older sentence-transformers models and up to 8192 tokens for newer ones such as nomic-embed-text.

The table below is a representative, non-exhaustive snapshot of open and hosted models, stated as of 2026-09-26 — re-check each model’s own page before depending on these numbers:

Model Dimensions Max sequence Deployment Licence Notes

sentence-transformers/all-MiniLM-L6-v2

384

256 tokens

Self-hosted

Apache 2.0

Small and fast; used for standalone demos on Embeddings.

nomic-ai/nomic-embed-text-v1.5 (served by Ollama as nomic-embed-text)

768 (Matryoshka, truncatable)

8192 tokens

Self-hosted (Ollama)

Apache 2.0

The model used throughout the DocsAssistant scenario; requires task prefixes (below).

BAAI/bge-m3

1024

8192 tokens

Self-hosted

MIT

Dense + sparse + multi-vector in one model; strong multilingual retrieval.

OpenAI text-embedding-3-small

1536 (configurable, down to 256)

8191 tokens

Hosted API

Commercial

Pay-per-token; no self-hosting option.

Cohere embed-v4.0

1536 (Matryoshka, down to 256)

128000 tokens

Hosted API

Commercial

Multimodal (text and images) in one model.

See the MTEB leaderboard for current rankings and the nomic-embed-text model page and bge-m3 model card for the self-hosted rows.

Generating embeddings

sentence-transformers

For local batch embedding jobs, SentenceTransformer.encode takes a batch_size, and normalize_embeddings=True L2-normalises the output so that cosine similarity reduces to a dot product (Similarity Metrics & Semantic Search):

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2", device="cpu")  # or "cuda" / "mps"

texts = [
    "HNSW indexes trade a small amount of recall for sub-linear search time.",
    "pgvector adds a vector column type and operators to PostgreSQL.",
]
embeddings = model.encode(
    texts,
    batch_size=32,
    normalize_embeddings=True,
    device="cpu",
    show_progress_bar=False,
)

See the sentence-transformers usage guide. Choosing a small local embedding model from the Hub and running it on-device or in the browser is covered in Small LLMs & Embeddings, and running an ONNX embedding model in-process on the JVM in Java & Spring Boot.

Ollama /api/embed

Ollama exposes embedding models through a dedicated HTTP endpoint. A single call can batch multiple inputs:

curl http://localhost:11434/api/embed -d '{
  "model": "nomic-embed-text",
  "input": [
    "search_document: HNSW indexes trade recall for sub-linear search time.",
    "search_document: pgvector adds a vector column type to PostgreSQL."
  ]
}'
import ollama

response = ollama.embed(
    model="nomic-embed-text",
    input=[
        "search_document: HNSW indexes trade recall for sub-linear search time.",
        "search_document: pgvector adds a vector column type to PostgreSQL.",
    ],
)
vectors = response["embeddings"]  # list[list[float]], 768 dims each

nomic-embed-text is task-prefixed (below) — every input string is prefixed with search_document: ` or `search_query: ` depending on which side of the search it is embedding. See the `nomic-embed-text model page and the Ollama embed API reference.

A hosted API via configuration

A hosted embedding provider is normally wired in through the framework’s configuration rather than a hand-rolled HTTP call, so the model name, API key and dimensions can change per environment without touching code. With Spring AI’s OpenAI starter (spring-ai-starter-model-openai), that is a handful of properties:

spring:
  ai:
    model:
      embedding: openai          # pick the OpenAI EmbeddingModel when several starters are present
    openai:
      api-key: ${OPENAI_API_KEY}
      embedding:
        options:
          model: text-embedding-3-small
          dimensions: 768        # the model's native size is 1536; 768 fits a vector(768) column

See the official Spring AI OpenAI embeddings page. LangChain takes the same settings as constructor arguments, reading the key from the OPENAI_API_KEY environment variable:

from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(model="text-embedding-3-small", dimensions=768)
query_vector = embeddings.embed_query("how do I speed up similarity search")

See the official LangChain OpenAI embeddings page. Either way the rest of the ingestion and query code stays provider-agnostic — the same EmbeddingModel / Embeddings abstractions shown below. A matching dimension does not make vectors interchangeable, though: text-embedding-3-small vectors live in a different space from nomic-embed-text ones and must not be mixed in the same index (see the golden rule below).

LangChain OllamaEmbeddings

from langchain_ollama import OllamaEmbeddings

embeddings = OllamaEmbeddings(model="nomic-embed-text")
doc_vectors = embeddings.embed_documents([
    "HNSW indexes trade recall for sub-linear search time.",
    "pgvector adds a vector column type to PostgreSQL.",
])
query_vector = embeddings.embed_query("how do I speed up similarity search")

embed_documents and embed_query apply LangChain’s Embeddings abstraction; see database/vector-rag/integrating-with-langchain.adoc#langchain-embeddings and the LangChain Ollama embeddings guide.

Spring AI EmbeddingModel (Ollama starter)

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-ollama</artifactId>
</dependency>
spring:
  ai:
    ollama:
      base-url: http://localhost:11434
      embedding:
        options:
          model: nomic-embed-text
@Service
public class EmbeddingService {

    private final EmbeddingModel embeddingModel;

    public EmbeddingService(EmbeddingModel embeddingModel) {
        this.embeddingModel = embeddingModel;
    }

    public float[] embed(String text) {
        return embeddingModel.embed(text);
    }
}

See database/vector-rag/integrating-with-spring-ai.adoc#spring-ai-embeddings and the Spring AI embeddings reference.

LangChain4j OllamaEmbeddingModel

EmbeddingModel embeddingModel = OllamaEmbeddingModel.builder()
        .baseUrl("http://localhost:11434")
        .modelName("nomic-embed-text")
        .build();

Response<Embedding> response = embeddingModel.embed("how do I speed up similarity search");
float[] vector = response.content().vector();

See the LangChain4j Ollama embedding model docs.

Query vs. document prompts, batching, caching and error handling

Asymmetric models and task prefixes

An asymmetric embedding model is trained to place a short query and a longer matching document close in vector space only when each side is embedded with its own instruction prefix — embedding both sides the same way measurably hurts retrieval quality. nomic-embed-text uses `search_document: ` for anything going into the index and `search_query: ` for anything a user types:

def embed_document(text: str) -> list[float]:
    return ollama.embed(model="nomic-embed-text", input=f"search_document: {text}")["embeddings"][0]


def embed_query(text: str) -> list[float]:
    return ollama.embed(model="nomic-embed-text", input=f"search_query: {text}")["embeddings"][0]

See the nomic-embed-text-v1.5 model card for the task prefixes and the Ollama embed API reference.

A symmetric model (query and document text look alike, e.g. sentence-similarity or FAQ matching) has no such prefix. Always check the model card before assuming one or the other.

Batching

Embedding calls are far more efficient in batches than one text at a time, since both sentence-transformers and Ollama amortise model-loading and (on GPU) kernel-launch overhead across the batch. Ingestion pipelines should accumulate chunks into batches of a few dozen to a few hundred (bounded by the model’s max sequence length and available memory) before calling encode or /api/embed; the query path, by contrast, almost always embeds a single string and should skip batching machinery entirely.

Caching by content hash

Re-embedding unchanged text wastes compute and, for a hosted API, money. Two caches cover the two paths, and only one of them survives a restart.

For ingestion, the cache is the database itself. In the DocsAssistant schema, chunks.content_hash stores a SHA-256 hash of each chunk’s text (Document Ingestion & Chunking), so before calling the model the pipeline looks for an existing embedding of identical content under the same model. Because it lives in PostgreSQL, it survives restarts and is shared by every worker process:

import hashlib

import psycopg
from pgvector.psycopg import register_vector

MODEL = "nomic-embed-text"


def content_hash(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()


def embedding_for_chunk(conn: psycopg.Connection, text: str):
    row = conn.execute(
        """
        SELECT ce.embedding
        FROM chunks c
        JOIN chunk_embeddings ce ON ce.chunk_id = c.id
        WHERE c.content_hash = %s AND ce.model = %s
        LIMIT 1
        """,
        (content_hash(text), MODEL),
    ).fetchone()
    if row is not None:
        return row[0]  # hit: reuse the stored vector, no model call
    return embed_document(text)  # miss: one Ollama call


with psycopg.connect("host=localhost dbname=docsassistant user=docs password=docs") as conn:
    register_vector(conn)  # read vector columns back as NumPy arrays
    vector = embedding_for_chunk(conn, "HNSW indexes trade recall for sub-linear search time.")

See the official pgvector-python (Psycopg 3) and hashlib pages.

For the query path, where the same handful of questions repeat within a session, an in-process functools.lru_cache is enough. It is lost when the process exits, which is acceptable for queries but not for ingestion:

from functools import lru_cache


@lru_cache(maxsize=1024)
def embed_query_cached(text: str) -> tuple[float, ...]:
    return tuple(embed_query(text))  # a tuple, so callers cannot mutate the cached value

See the official functools.lru_cache page.

Memory management and error handling

Embedding jobs over large corpora should stream chunks and their vectors through the pipeline rather than holding the whole corpus in memory at once, and should release each model’s GPU/CPU memory (del model, explicit garbage collection, or a context-managed client) once a batch job finishes. On the network side — Ollama’s HTTP endpoint or any hosted API — calls should have a timeout, and repeated failures should retry with exponential backoff before failing the ingestion run outright rather than silently skipping chunks; skipped chunks leave permanent gaps in the index that are hard to notice later. Only transient failures are worth retrying: a connection error, a timeout or a server-side (5xx) error, but not a client error such as a model that is not installed:

import time

import httpx
import ollama

client = ollama.Client(host="http://localhost:11434", timeout=30.0)  # seconds


def embed_batch_with_retry(texts: list[str], attempts: int = 5) -> list[list[float]]:
    for attempt in range(attempts):
        try:
            return client.embed(model="nomic-embed-text", input=texts)["embeddings"]
        except ollama.ResponseError as error:
            if error.status_code < 500 or attempt == attempts - 1:
                raise  # e.g. 404 model not found: retrying will not help
        except (httpx.TransportError, ConnectionError):  # timeouts, refused connections
            if attempt == attempts - 1:
                raise
        time.sleep(2 ** attempt)  # 1 s, 2 s, 4 s, 8 s
    raise AssertionError("unreachable")

See the official Ollama Python library — Errors and Ollama embed API pages. On the JVM, Spring AI’s model clients have their own retry support for transient failures, configured with the spring.ai.retry.* properties.

The golden rule: same model, same version, for index and query

Every vector added to chunk_embeddings and every vector used to search it must come from the same embedding model and the same version of that model. Two versions of the same named model (a re-trained checkpoint, a changed default dimension, a changed tokenizer) place text in different vector spaces — cosine similarity between a vector from one version and a vector from another is not meaningful, even though both are 768-dim floats. This is why the shared schema stores model alongside every embedding rather than assuming a single global model:

CREATE TABLE chunk_embeddings (
    chunk_id  bigint PRIMARY KEY REFERENCES chunks(id) ON DELETE CASCADE,
    model     text NOT NULL,
    embedding vector(768) NOT NULL
);

See the official pgvector — Storing page, and PostgreSQL & pgvector for the full DocsAssistant schema.

A query handler should read model back out before searching (or hard-code the one model DocsAssistant is configured to serve) and refuse to compare embeddings written under a different value, rather than silently mixing spaces.

Re-embedding and migration strategies

Changing the embedding model — upgrading nomic-embed-text, switching to a higher-quality model, changing the truncated Matryoshka dimension — means every existing vector becomes stale. Three strategies handle the cut-over without downtime, in increasing order of operational weight:

  • A new column. Add embedding_v2 vector(1024) (or whatever the new model’s dimension is) next to the existing embedding column, backfill it chunk by chunk, and switch queries to it once backfill completes. Simple, but doubles storage during the migration and only works when both dimensions fit comfortably in one table.

  • A new table (shadow index). Create chunk_embeddings_v2 with the same shape but a different model value and its own HNSW index, built incrementally as chunks are re-embedded. Queries keep reading the old table until the shadow index’s coverage and quality are verified, then cut over by renaming the tables (or flipping a config flag) atomically.

  • Dual write, then cut-over. During the transition window, ingestion writes both the old and the new embedding for every chunk (new and re-embedded alike); the query path still reads the old table. Once the shadow index has full coverage and passes evaluation (Evaluating RAG Systems), the cut-over flips reads to the new table/column and a cleanup job drops the old one.

The mermaid flow below shows the shadow-index variant end to end:

flowchart LR A[Chunks table\nunchanged] --> B[Backfill job\nembeds every chunk\nwith model v2] B --> C[(chunk_embeddings_v2\nshadow HNSW index)] D[Live queries] --> E[(chunk_embeddings\nmodel v1, current)] C --> F{Coverage complete\nand evaluation passes?} F -- no --> B F -- yes --> G[Cut over:\nqueries read v2] G --> H[Drop chunk_embeddings\nmodel v1]

In LangChain and Spring AI

LangChain

LangChain’s Embeddings interface is the abstraction every embedding provider implements, so switching models means swapping the constructor, not the call sites (embed_documents / embed_query stay the same). See database/vector-rag/integrating-with-langchain.adoc#langchain-embeddings.

Spring AI

Spring AI’s EmbeddingModel plays the same role: embed(String) / embed(Document) are provider-agnostic, and switching from the Ollama starter to a hosted provider’s starter is a dependency and configuration change. See database/vector-rag/integrating-with-spring-ai.adoc#spring-ai-embeddings.

LangChain4j’s EmbeddingModel interface (used above for OllamaEmbeddingModel) fills the same role on the JVM side; database/vector-rag/integrating-with-langchain.adoc#langchain4j covers its broader RAG building blocks.

Related pages

References