Observability Platforms

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Written on 2026-10-01 against Langfuse 4.49.0 (Python SDK 4.16.0), MLflow 3.16.1, LangSmith (langsmith SDK 0.14.2), Arize Phoenix 20.18.0 (Elastic License 2.0), Grafana 13.2.3 with Tempo and Loki, OpenTelemetry Python 1.45.0 and Python 3.12; the full list is the version baseline on the landing page. No platform on this page was installed or started while writing it: every configuration snippet, endpoint path and environment variable was written from the official documentation linked after it and was not run. Where a detail (an image tag, a header name, a port, a licence boundary) could not be confirmed it is marked as unverified in the text, and the reader should check the linked page before copying it. No prices or benchmark figures are quoted; commercial terms change often and are linked, not repeated.

The previous page, Observability with OpenTelemetry, produced spans that follow the GenAI conventions. A span is only useful if someone can open it, search it, attach a score to it and compare it with last week’s. That is the job of an observability platform, and the DocsAssistant has to pick one. The choice matters less than it looks if the application emits standard OpenTelemetry, because the backend then becomes a configuration value; it matters more for the features around the traces (evaluation, datasets, prompt versions) that are not portable. The general argument for tracing as the foundation of LLM operations is made by Aryan in LLMOps and, for evaluation-driven development, by Huyen in AI Engineering; the platform descriptions below are written from each project’s own documentation instead, because the books pre-date the current releases.

What a platform adds on top of raw traces

Any OTLP-capable store can hold the spans. What the LLM-specific platforms add is a layer of features that the DocsAssistant team would otherwise have to build:

  • Trace views that understand LLM calls. Prompts, completions, token usage and tool calls rendered as a conversation, not as anonymous span attributes.

  • Scores attached to traces. Human labels, user feedback and judge results stored next to the trace they describe, so that Online Evaluation and Feedback has somewhere to write.

  • Datasets and experiments. Turning a bad production trace into a golden-set row and re-running the set against a new prompt.

  • Prompt management. Versioned prompts with labels, fetched at run time; the release side of this is on Prompt and Model Management.

  • Cost views. Token counts multiplied by a model price table that the platform maintains.

A Grafana stack gives none of the first four out of the box, and that is a legitimate trade for teams that already run it.

LangSmith

LangSmith is the observability and evaluation service from the LangChain company. It traces LangChain and LangGraph applications with environment variables alone, and also accepts traces from other code through its SDK wrappers and through OpenTelemetry. It offers datasets, experiments, online evaluators, annotation queues and a prompt hub. The LangChain-specific setup is already covered on Observability with LangSmith, so this page only places it in the comparison.

For the DocsAssistant the relevant points are:

  • It is a hosted product first; a self-hosted deployment exists but is documented as an enterprise offering, and its terms are not verified here. Check the LangSmith documentation for the current hosting options before assuming data can stay inside your network.

  • The SDKs are MIT-licensed (see the version baseline), but the service itself is proprietary. A licence column that says "MIT" for the SDK must not be read as "the platform is open source".

  • Its evaluation tooling pairs with openevals and agentevals, used on LLM-as-a-Judge and RAG and Agent Evaluation.

  • Trace ingestion through OpenTelemetry is documented; the exact endpoint and header names were not checked for this page, so the wiring section below covers Langfuse and Phoenix only.

Langfuse

Langfuse is an LLM engineering platform that combines tracing, prompt management, evaluations, datasets and a playground. It is the default backend in this sub-section’s running scenario because it is self-hostable, accepts OpenTelemetry directly and bundles prompt management with evaluation. Licence: the repository states that everything outside its ee/ directories is MIT and that the enterprise directories carry a separate licence, so a self-hosted install is free to run but some features sit behind a commercial licence key. Which features those are changes between releases and was not verified here; read the self-hosting licence page before promising a feature to a team.

Self-hosted deployment

The self-hosted architecture is not a single container. It has two Langfuse services (a web application and an asynchronous worker) and four backing stores: PostgreSQL for transactional data, ClickHouse for traces and analytics, Redis for queues and caching, and S3-compatible blob storage (MinIO in the reference setup) for raw event payloads. The official Docker Compose file is a quick-start for a single machine; the documentation positions Kubernetes with Helm for production. Everything below is a sketch of the shape, with every secret taken from the environment, not a replacement for that file.

# docker-compose.yml (sketch; the official file defines more variables, healthchecks and volumes)
services:
  langfuse-web:
    image: docker.langfuse.com/langfuse/langfuse:${LANGFUSE_IMAGE_TAG}
    depends_on: [postgres, clickhouse, redis, minio]
    ports:
      - "3000:3000"
    environment:
      DATABASE_URL: postgresql://postgres:${POSTGRES_PASSWORD}@postgres:5432/postgres
      NEXTAUTH_URL: ${LANGFUSE_PUBLIC_URL}
      NEXTAUTH_SECRET: ${NEXTAUTH_SECRET}
      SALT: ${LANGFUSE_SALT}
      ENCRYPTION_KEY: ${LANGFUSE_ENCRYPTION_KEY}
      CLICKHOUSE_URL: http://clickhouse:8123
      CLICKHOUSE_MIGRATION_URL: clickhouse://clickhouse:9000
      CLICKHOUSE_USER: ${CLICKHOUSE_USER}
      CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD}
      REDIS_HOST: redis
      REDIS_AUTH: ${REDIS_AUTH}
      LANGFUSE_S3_EVENT_UPLOAD_BUCKET: ${LANGFUSE_S3_BUCKET}
      LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT: http://minio:9000
      LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID: ${MINIO_ROOT_USER}
      LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY: ${MINIO_ROOT_PASSWORD}
  langfuse-worker:
    image: docker.langfuse.com/langfuse/langfuse-worker:${LANGFUSE_IMAGE_TAG}
    depends_on: [postgres, clickhouse, redis, minio]
    environment:
      # same DATABASE_URL, SALT, ENCRYPTION_KEY, CLICKHOUSE_*, REDIS_* and LANGFUSE_S3_* values as langfuse-web
      DATABASE_URL: postgresql://postgres:${POSTGRES_PASSWORD}@postgres:5432/postgres
  postgres:
    image: docker.io/postgres:${POSTGRES_IMAGE_TAG}
    environment:
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
  clickhouse:
    image: docker.io/clickhouse/clickhouse-server
    environment:
      CLICKHOUSE_USER: ${CLICKHOUSE_USER}
      CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD}
  redis:
    image: docker.io/redis:${REDIS_IMAGE_TAG}
    command: ["redis-server", "--requirepass", "${REDIS_AUTH}"]
  minio:
    image: cgr.dev/chainguard/minio
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}

The variables come from an untracked .env file or from the CI secret store, never from the compose file; generate each value with a password manager or openssl rand. The image names (docker.langfuse.com/langfuse/langfuse and langfuse-worker, docker.io/clickhouse/clickhouse-server, cgr.dev/chainguard/minio, docker.io/redis, docker.io/postgres) were checked on 2026-10-01 against the official docker-compose.yml linked from the guide below, which pins the Langfuse images to major version 4. That file also defines healthchecks on PostgreSQL, ClickHouse, Redis and MinIO, makes the Langfuse containers wait for them, and shares one long environment block between web and worker; the sketch leaves all of that and the tags out on purpose, so copy the official file rather than this one. For anything beyond a single-host evaluation, use the Helm chart and managed PostgreSQL, ClickHouse and Redis, as the self-hosting documentation recommends. Derived from the Langfuse self-hosting guide for Docker Compose and the self-hosting configuration reference.

Operational consequences worth planning for: ClickHouse and the blob store hold prompts and completions, so they inherit the retention and PII rules from Data Operations; the Langfuse public port should sit behind your normal TLS and authentication front end; and upgrades run database migrations, so back up PostgreSQL and ClickHouse before changing the image tag.

MLflow 3 GenAI tracing

MLflow began as an experiment tracker and model registry and, from version 3, adds GenAI features: tracing with automatic instrumentation for many client libraries, LLM-as-a-judge scorers, evaluation datasets and a prompt registry. It is Apache-2.0 licensed, runs as a server you host (with a database for metadata and an artifact store), and its documentation describes OpenTelemetry compatibility, including export of traces to other OTLP backends and ingestion of OTLP traces.

A minimal use is two lines of Python that turn on automatic tracing for an OpenAI-compatible client:

import os

import mlflow
from openai import OpenAI

mlflow.set_tracking_uri("http://mlflow.internal:5000")   # the MLflow server of your team
mlflow.set_experiment("docsassistant-prod")
mlflow.openai.autolog()                                  # traces every chat.completions call

client = OpenAI(
    base_url="http://litellm.internal:4000/v1",
    api_key=os.environ["LITELLM_KEY"],                   # key from the environment
)

Because the DocsAssistant calls its models through LiteLLM, which speaks the OpenAI protocol, mlflow.openai.autolog() is the obvious integration. Derived from the MLflow GenAI tracing documentation; the exact signature of autolog and the set of supported libraries were read from that page’s index only and were not exercised.

MLflow is the natural pick when a data-science team already uses it for classical models and wants one registry for both worlds (see Prompt and Model Management). It is less natural as a pure operations console for an engineering team, because its trace UI sits inside an experiment-tracking product.

Arize Phoenix

Phoenix is an open-source-style observability and evaluation tool from Arize, built on OpenTelemetry and the OpenInference instrumentation conventions. It runs locally from a single Python package or container, stores traces in SQLite by default or in PostgreSQL, and offers trace exploration, datasets, experiments, a prompt playground and evaluators. The licence needs care: Phoenix is released under the Elastic License 2.0, which is source-available and not an OSI-approved open source licence. Its restrictions (for example on offering the software as a hosted managed service) rarely affect an internal deployment, but legal review is the right path if your product exposes it to customers.

Two traits matter for the DocsAssistant. First, Phoenix is attractive for local development and for notebooks, because it starts in seconds without the multi-store architecture that Langfuse requires. Second, its native span vocabulary is OpenInference, which overlaps with but is not identical to the OpenTelemetry gen_ai.* attributes; how well each backend renders spans from a given instrumentation library should be tested with real traces before it is relied on.

A Grafana, Tempo and Loki stack

Teams that already run Grafana for infrastructure can keep LLM traces in the same place: Tempo stores traces, Loki stores logs (including prompt and completion events if you choose to log them), Prometheus or Mimir stores the metrics, and Grafana joins them in dashboards and Explore. All the components are open source; their licences are not all the same (Grafana, Tempo and Loki are published under AGPL-3.0 at the time of the last check, which has obligations if you modify and expose them, and the exact current terms should be read from each repository).

What this stack does well is the operations half of the story: latency percentiles, error rates, dashboards, alerts, and correlation between a slow answer and the vLLM or gateway span underneath it. The metrics side is the subject of Metrics, Dashboards and Alerting. What it does not do is understand an LLM conversation: there are no built-in scores, datasets, or prompt versions, and rendering a prompt means reading span attributes or log lines. A common pattern is therefore to run both: OpenTelemetry Collector fans the same spans out to Tempo for operations and to Langfuse or Phoenix for quality work. Derived from the Tempo documentation and the Loki documentation; no Grafana stack was started for this page.

Comparison

The table summarises claims from each project’s documentation as of the written date. Cells that say "not verified" are cases where the page’s author could not confirm the point from a primary source.

Platform Licence Self-host OpenTelemetry Evaluations Prompt management

LangSmith

Proprietary service; the langsmith, openevals and agentevals SDKs are MIT.

Documented as an enterprise option; terms not verified here.

Ingestion documented; endpoint details not verified here.

Datasets, experiments, online evaluators, annotation queues.

Prompt hub with versions and commits.

Langfuse

MIT outside the ee/ directories, which carry a separate commercial licence; check which features fall inside it.

Yes, Docker Compose for a single host and Helm for Kubernetes; needs PostgreSQL, ClickHouse, Redis and blob storage.

Native OTLP ingestion documented over HTTP (protobuf and JSON); gRPC documented as not supported.

LLM-as-a-judge, datasets, experiments, human annotation, scores through API.

Yes, versioned prompts with labels and a client cache.

MLflow 3

Apache-2.0.

Yes, a server plus a database and artifact store.

Documented compatibility, with OTLP ingestion and export; details not verified here.

Scorers and judges, evaluation datasets, comparison runs.

Yes, prompt registry with versions and aliases.

Arize Phoenix

Elastic License 2.0: source-available, not OSI open source, with limits on offering it as a managed service.

Yes, one container or Python package; SQLite or PostgreSQL.

OpenTelemetry based, with OpenInference conventions; gen_ai.* rendering should be tested.

Evaluators, datasets, experiments.

Yes, a prompt playground and versioned prompts; depth not verified here.

Grafana, Tempo, Loki

Open source; the main components are AGPL-3.0 at the last check, read each repository for current terms.

Yes, plus the operational cost of running several services.

Native OTLP.

None built in; scores must be modelled as your own metrics or span attributes.

None built in.

None of the platforms is a drop-in replacement for another, because the evaluation and prompt features use their own data models. The portable layer is the trace, which is why the GenAI conventions matter.

Wiring the DocsAssistant through OTLP

The DocsAssistant already emits spans through the OpenTelemetry SDK, as built on Observability with OpenTelemetry. Pointing it at a different backend is then an environment change, not a code change. The OpenTelemetry SDKs read the standard OTEL_EXPORTER_OTLP_* variables, so the same container image can be started with different values per environment.

To Langfuse

Langfuse’s OpenTelemetry endpoint is under /api/public/otel on the Langfuse host and authenticates with HTTP Basic using the project’s public and secret key. The documentation supports OTLP over HTTP with protobuf or JSON and states that gRPC is not supported. The header value is the Base64 encoding of public:secret, so build it at start-up from the two keys held as secrets, and never write it into the repository or the image:

# Keys come from the secret store; nothing below contains a literal key.
export LANGFUSE_AUTH="$(printf '%s:%s' "${LANGFUSE_PUBLIC_KEY}" "${LANGFUSE_SECRET_KEY}" | base64 | tr -d '\n')"

export OTEL_SERVICE_NAME="docsassistant"
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="${LANGFUSE_HOST}/api/public/otel/v1/traces"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="Authorization=Basic ${LANGFUSE_AUTH},x-langfuse-ingestion-version=4"

Derived from the Langfuse OpenTelemetry integration page, checked on 2026-10-01: it documents both /api/public/otel (for exporters that append the signal path themselves) and /api/public/otel/v1/traces (for a signal-specific endpoint variable, as used here), and it asks for the x-langfuse-ingestion-version: 4 header so that data appears in real time on the version 4 data model instead of with a delay. Header values containing spaces must be passed as the OpenTelemetry specification requires (percent-encoding in some SDKs); if the SDK rejects Authorization=Basic …​, follow the encoding note in the SDK documentation. The Langfuse Python SDK can also be used instead of raw OTLP when its prompt-management and score APIs are wanted; it is configured with LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY and LANGFUSE_HOST, and for the self-hosted deployment LANGFUSE_HOST is the URL of the langfuse-web service.

To Phoenix

Phoenix listens for OTLP over HTTP on its web port and for gRPC on a second port: the tracing documentation, checked on 2026-10-01, gives 6006 for the UI and OTLP HTTP and 4317 for OTLP gRPC. A local or private-network Phoenix usually needs no authentication; an instance with authentication enabled expects an API key, which again comes from the environment:

export OTEL_SERVICE_NAME="docsassistant"
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="${PHOENIX_COLLECTOR_ENDPOINT}/v1/traces"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="http/protobuf"
# only when Phoenix authentication is enabled:
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="authorization=Bearer ${PHOENIX_API_KEY}"

Derived from the Phoenix tracing documentation and the OpenTelemetry OTLP exporter configuration. The PHOENIX_COLLECTOR_ENDPOINT and PHOENIX_API_KEY variables are from the tracing page, and the lowercase authorization: Bearer header from the Phoenix authentication documentation, both checked on 2026-10-01; that page asks for the lowercase header name so the same value also works over gRPC. Phoenix also ships its own phoenix.otel helper that registers a tracer provider; the raw-variable route above is shown because it needs no Phoenix package in the application image.

Both at once

Exporting from the application to two vendors doubles the places that need keys and increases request-path overhead. The cleaner pattern is a single export to an OpenTelemetry Collector inside your network, with the Collector holding the backend credentials and fanning out to Langfuse, Phoenix and Tempo through its own pipelines. Then the application only needs OTEL_EXPORTER_OTLP_ENDPOINT pointing at the Collector, and rotating a Langfuse key never touches the application.

How to choose

Decide in this order, and expect the first two questions to remove most of the options.

  1. Where may the data live? Prompts and completions contain user questions, and possibly retrieved text. If they cannot leave your network, hosted-only options are out, and the self-hostable ones compete on operating cost: Phoenix is the lightest, Langfuse needs four stores, MLflow needs a database and artifact store, and Grafana needs whatever your platform team already runs.

  2. Which licence can your organisation accept? Phoenix’s Elastic License 2.0 and Langfuse’s enterprise directories are the two cases that need a conscious decision; AGPL in the Grafana stack matters mostly if you modify and expose the software.

  3. What is the framework? A team fully on LangChain and LangGraph gets the least friction from LangSmith; a team on Spring AI or plain OpenAI-compatible clients gains little from that tie.

  4. Who uses it? Engineers debugging a slow answer want trace and metric correlation (Grafana, Langfuse). Product and evaluation owners want scores, datasets and prompt versions in a UI (Langfuse, LangSmith, Phoenix, MLflow). A data-science group that already runs MLflow should check whether its GenAI features are enough before adding a second system.

  5. Can you keep the exit open? Instrument with OpenTelemetry and the GenAI conventions, send through a Collector, and keep prompts in a registry whose export format you control. Then switching platform is a configuration and migration task, not a rewrite.

For the DocsAssistant the default recommendation of this sub-section is: OpenTelemetry to a Collector, Langfuse for traces, scores and prompts, Prometheus and Grafana for operational dashboards, and Phoenix on developer laptops. That is a default, not a ranking; a team with different constraints will reach a different answer, and the table above is there so the reasoning can be re-run.

References