Data Operations

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Written on 2026-10-01 against Python 3.12, MLflow 3.16.1, Langfuse 4.49.0 and Prometheus 3.15.0; the full list is the version baseline on the landing page. Every snippet was written from the official documentation linked after it and was not run. The data-pipeline structure is paraphrased from the data chapters of Aryan’s _LLMOps and Huyen’s AI Engineering (see the bibliography), adapted to retrieval and evaluation data rather than model-training corpora.

LLM applications consume four kinds of data: the documents behind retrieval, the traces captured in production, the golden and feedback datasets used for evaluation, and the prompts and configuration (covered in Prompt and Model Management). Each needs an owner, a lineage and a recovery plan.

A ten-step preprocessing pipeline

The sequence follows the order of Aryan’s DataOps pipeline: catalogue the sources first, settle privacy and legal questions before any content is processed, filter and deduplicate, and finish with ongoing maintenance rather than a one-off load. The step names and descriptions below are paraphrased and adapted to a retrieval index and evaluation sets; the middle steps (normalising, redacting, enriching, splitting, embedding) are this guide’s mapping of the book’s sequence onto the DocsAssistant. The same sequence applies whether the output is an index, an evaluation set or a fine-tuning corpus, and each step has an observable output so a failure can be located.

Step Name What it does for the DocsAssistant

1

Catalogue

Inventory every source (repository, wiki, tickets) with its owner, source URI, revision, licence and access class, so nothing enters the pipeline without a known origin.

2

Privacy and legal review

Decide per source whether it may be used at all, under which licence or consent, which fields count as personal or secret data, and how long derived artefacts may be kept.

3

Filter

Drop out-of-scope, low-quality, expired or unreadable content and anything the review in step 2 excluded, and count what was dropped.

4

Deduplicate

Remove exact and near-duplicate content so retrieval does not return the same passage repeatedly.

5

Normalise

Convert to a single text format, fix encoding and whitespace, preserve headings and code blocks.

6

Redact

Apply the step 2 decisions: remove or pseudonymise personal and secret data before anything is stored or sent to a model.

7

Enrich

Attach metadata: product, version, language, owner, access class and effective dates.

8

Split

Chunk along structure with the sizes the evaluation chose, keeping a pointer to the parent section.

9

Embed, load and verify

Compute embeddings with a recorded model version, write to the index with the metadata, then run sanity queries and the retrieval part of the golden set.

10

Maintain

Record counts, versions and lineage; re-run on source changes; watch the freshness objective; retire expired content and re-review sources whose licence or owner changed.

Parsing, chunking and embedding choices are treated in Document Parsing and Ingestion Pipelines and Freshness; this page covers the operational envelope around them. The verification in step 9 gates promotion in the same way as the checks in Evaluation in CI.

Dataset lineage

Lineage answers "where did this dataset or index come from, and what depends on it". Record, for every produced artefact: the input revisions (source commit or snapshot id), the pipeline code revision, the parameter set, the output identifier (index name and version, dataset version) and the time. Tools differ in how they store it, but the minimum is a manifest written by the last pipeline step.

import json, hashlib, datetime, pathlib
import mlflow

def record_lineage(index_version: str, source_rev: str, code_rev: str, params: dict, eval_path: str) -> None:
    manifest = {
        "index_version": index_version,
        "source_revision": source_rev,
        "pipeline_revision": code_rev,
        "params": params,
        "eval_set_sha256": hashlib.sha256(pathlib.Path(eval_path).read_bytes()).hexdigest(),
        "created_at": datetime.datetime.now(datetime.timezone.utc).isoformat(),
    }
    pathlib.Path("manifest.json").write_text(json.dumps(manifest, indent=2))
    with mlflow.start_run(run_name=f"index-{index_version}"):
        mlflow.log_params(params)
        mlflow.log_artifact("manifest.json")

Source: MLflow: Tracking. For versioning the data itself, a tool such as DVC stores content hashes in Git while the bytes live in object storage (see DVC: Data versioning); MLflow datasets and LangSmith datasets, described in Building Eval Sets, version evaluation data. Tracing a bad answer back through the trace to the chunk, the index version and the source revision is the practical test of whether lineage is complete.

Handling personal data

Traces and feedback contain whatever users type, which includes personal data and sometimes secrets. Controls, in order of how early they act:

  1. Minimise: do not capture content you do not need. OpenTelemetry GenAI content capture is opt-in for this reason (see Observability with OpenTelemetry).

  2. Redact before storage: detect and mask identifiers in prompts and responses at the SDK or collector, before the data leaves the process.

  3. Pseudonymise user and tenant identifiers so traces can be grouped without identifying a person.

  4. Restrict access to raw traces by role and log who reads them.

  5. Retain briefly: set a retention period per data class and delete on schedule, including copies in evaluation datasets.

  6. Honour deletion requests by knowing where a person’s data can appear: traces, feedback, golden sets built from traffic, and caches (see Cost Management for semantic caches).

from langfuse import Langfuse

def mask(data, **kwargs):
    # Replace e-mail addresses before the event leaves the process.
    import re
    if isinstance(data, str):
        return re.sub(r"[\w.+-]+@[\w-]+\.[\w.]+", "[EMAIL]", data)
    return data

langfuse = Langfuse(mask=mask)   # credentials are read from LANGFUSE_* environment variables

Source: Langfuse: Masking. A regular expression catches only the patterns you thought of; production systems usually combine pattern rules with a dedicated detection library. Access control for retrieved content is a separate problem, discussed in Access Control and Privacy. The legal bases and obligations (for example under the GDPR) are named in Operational Governance and are treated in Sensitive Information and Hidden Context and The EU AI Act; this guide does not give legal advice.

Vector-index freshness SLAs

A freshness SLO states how stale an answer may be. For the DocsAssistant: "99% of documents are searchable within 30 minutes of publication and none is older than 24 hours relative to its source". Define freshness for a document as

and the SLI as the fraction of documents with over a window. Export the lag as a metric from the ingestion job and alert on its tail, not its mean.

groups:
  - name: index-freshness
    rules:
      - record: docsassistant:index_lag_seconds:p99
        expr: histogram_quantile(0.99, sum by (le) (rate(index_lag_seconds_bucket[1h])))
      - alert: IndexFreshnessSloBreached
        expr: docsassistant:index_lag_seconds:p99 > 1800
        for: 30m
        labels:
          severity: ticket

Source: Prometheus: Recording rules. The metric name is this guide’s convention. Deletions count as freshness too: a retired page that still ranks is a worse failure than a slow new page, so track delete propagation separately. Event-driven versus scheduled ingestion, tombstones and re-embedding after a model change are in Ingestion Pipelines and Freshness.

Backups and restore tests

An index can be rebuilt from sources, but rebuilding costs embedding spend and time, and some data cannot be rebuilt. Classify before deciding:

Asset Rebuildable? Backup approach

Vector index

Yes, from sources and the manifest.

Snapshot to object storage for fast recovery; otherwise keep the manifest and rebuild.

Golden and feedback datasets

No (human labels).

Versioned storage plus off-site copy; treat as source data.

Traces and metrics

No, but disposable after retention.

Follow the retention policy; back up the aggregated reports.

Prompt registry and gateway config

Partly (Git holds exports).

Export the registry on a schedule; keep gateway config in Git.

A backup that has never been restored is an assumption. Schedule a restore test: restore the latest snapshot into an isolated environment, run the retrieval part of the golden set, and compare the scores with the last release; record the time taken against the recovery-time target in the runbook (see Operational Governance). Vector-store specifics are in Production Considerations for Vector Search and RAG.

References