LLMOps & Evaluation Cheat Sheet

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

A single-page index of quick back-links to every page in this sub-section, grouped the same way as the navigation. It was written on 2026-10-01 against vLLM 0.30.0, SGLang 0.5.21, KServe 0.21.0, LiteLLM 1.103.2, MLflow 3.16.1, Langfuse 4.49.0, LangSmith SDK 0.14.2 with openevals 0.2.0 and agentevals 0.0.9, Arize Phoenix 20.18.0, RAGAS 0.4.3, DeepEval 4.2.7, OpenTelemetry Python 1.45.0 with the GenAI semantic conventions at Development stability, Spring AI 2.0.1, LangChain 1.4.3, Prometheus 3.15.0 and Grafana 13.2.3; the full list is the version baseline on the landing page.

The downloadable PDF is a separate, single A4 page meant for printing. It states the version baseline and covers:

  • the lifecycle loop and the three maturity levels;

  • the metric catalogue with formulas (pass@k, BLEU, ROUGE, BERTScore, embedding similarity, perplexity, nDCG@k, MRR, Cohen’s kappa, Elo, RAG metrics);

  • the judge rubric, the judge bias checklist and comparative evaluation (Elo, Bradley-Terry);

  • eval-set rules and sample-size arithmetic;

  • the CI gate: Wilson lower bound, paired sign test and a gate snippet;

  • online-evaluation sampling and feedback;

  • serving metrics (TTFT, TPOT, goodput, MBU) and autoscaling signals;

  • release patterns: shadow, canary, A/B, blue-green and the rollback ladder;

  • the OpenTelemetry gen_ai.* attributes and metrics;

  • the observability platform comparison;

  • alert rules and drift measures;

  • the cost formula and its levers;

  • the governance checklist.

Overview — LLMOps & Evaluation.

Evaluation — Evaluation Methodology, LLM-as-a-Judge, Comparative Evaluation and Arenas, Building Eval Sets, RAG and Agent Evaluation, Evaluation in CI and Online Evaluation and Feedback.

Serving and release — Serving at Scale and Deployment and Release Patterns.

Observability — Observability with OpenTelemetry, Observability Platforms, Metrics, Dashboards and Alerting and Drift and Regression Monitoring.

Cost and governance — Cost Management, Prompt and Model Management, Data Operations and Operational Governance.

The one formula the cost section of the sheet builds on — the cost of one request, explained on Cost Management:

Source: OpenTelemetry: Semantic conventions for generative AI and vLLM: Metrics for the signals the sheet lists.

Download the LLMOps & Evaluation Cheat Sheet (PDF)

References

The PDF summarises the pages above; each source is cited in full on the page that covers it, and in the bibliography.