LLMOps & Evaluation

Serving. LLMOps (Aryan) leans on Jenkins and Triton-centred practice; current stacks use GitHub Actions with vLLM, SGLang and KServe, and TGI is in maintenance mode.

Experiment tracking and evaluation. MLflow 3 now covers GenAI tracing, scorers, judges and a prompt registry; LangSmith judges come from openevals and agentevals rather than judge prompts pulled from the Hub; Langfuse is self-hostable and OpenTelemetry-based.

Observability conventions. The gen_ai.* conventions post-date the books, are still at Development stability, and moved to their own repository.

Benchmarks. The Hugging Face Open LLM Leaderboard is retired and MMLU is saturated; evaluation centres on task-specific evals, arenas and agent evals.

Spring AI. Spring AI in Action targets Spring AI 1.0; the pages use the 2.0.1 evaluator and observability APIs.

Aryan, Abi. LLMOps: Managing Large Language Models in Production. O’Reilly Media, 1st ed., July 2025. ISBN 978-1-098-15420-2. Publisher page: https://www.oreilly.com/library/view/llmops/9781098154196/. No official code repository; supplementary material: https://oreil.ly/LLMOps. Attribution note: the copyright page reads "(c) 2025 Abi Aryan and MeyerPerin Inc." and Lucas Meyer appears only in the acknowledgments, so this bibliography cites Abi Aryan as the author. Chapters consulted: 2 (maturity model), 3, 4, 6, 7, 8 and 9.

Huyen, Chip. AI Engineering: Building Applications with Foundation Models. O’Reilly Media, December 2024. ISBN 978-1-098-16630-4. Publisher page: https://www.oreilly.com/library/view/ai-engineering/9781098166298/. Official code repository: https://github.com/chiphuyen/aie-book. Chapters consulted: 3, 4, 9 and 10.

Albada, Michael. Building Applications with AI Agents. O’Reilly Media, September 2025. ISBN 978-1-098-17650-1. Publisher page: https://www.oreilly.com/library/view/building-applications-with/9781098176495/. Official code repository: https://github.com/michaelalbada/BuildingApplicationsWithAIAgents. Chapters consulted: 9, 10 and 11.

Mendelevitch, Ofer, and Forrest Sheng Bao. Hands-On RAG for Production. O’Reilly Media, May 2026. ISBN 979-8-341-62171-8. Publisher page: https://www.oreilly.com/library/view/hands-on-rag-for/9798341621701/. Official code repository: https://github.com/ofermend/hands-on-rag. Chapters consulted: 4 and 6.

Oshin, Mayo, and Nuno Campos. Learning LangChain. O’Reilly Media, 2025. ISBN 978-1-098-16728-8. Publisher page: https://www.oreilly.com/library/view/learning-langchain/9781098167271/. Official code repository: https://github.com/langchain-ai/learning-langchain. Chapter 10 (testing and evaluation with LangSmith) consulted.

Walls, Craig. Spring AI in Action. Manning Publications, 2026. ISBN 978-1-63343-611-4. Publisher page: https://www.manning.com/books/spring-ai-in-action. Official code repository: https://github.com/habuma/spring-ai-in-action-samples. Chapters 2 (evaluators) and 9 (observability) consulted.

RAGAS: https://docs.ragas.io/

DeepEval: https://deepeval.com/docs/getting-started

openevals: https://github.com/langchain-ai/openevals

agentevals: https://github.com/langchain-ai/agentevals

LangSmith evaluation: https://docs.langchain.com/langsmith/evaluate-llm-application

MLflow GenAI: https://mlflow.org/docs/latest/genai/

Spring AI testing and evaluators: https://docs.spring.io/spring-ai/reference/api/testing.html

Spring AI LLM-as-a-judge guide: https://docs.spring.io/spring-ai/reference/guides/llm-as-judge.html

Langfuse: https://langfuse.com/docs

Arize Phoenix: https://arize.com/docs/phoenix

LangSmith observability: https://docs.langchain.com/langsmith/observability

Spring AI observability: https://docs.spring.io/spring-ai/reference/observability/index.html

vLLM: https://docs.vllm.ai/

SGLang: https://docs.sglang.io/

KServe: https://kserve.github.io/website/

LiteLLM proxy: https://docs.litellm.ai/docs/simple_proxy

Prometheus: https://prometheus.io/docs/

Grafana: https://grafana.com/docs/

OpenTelemetry GenAI semantic conventions: https://opentelemetry.io/docs/specs/semconv/gen-ai/ (this page now only points to the dedicated repository)

The conventions repository: https://github.com/open-telemetry/semantic-conventions-genai

OpenTelemetry GenAI observability blog post: https://opentelemetry.io/blog/2026/genai-observability/

W3C Trace Context: https://www.w3.org/TR/trace-context/

Model Context Protocol specification: https://modelcontextprotocol.io/specification/

NIST AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework

Regulation (EU) 2024/1689 (Artificial Intelligence Act), official text on EUR-Lex: https://eur-lex.europa.eu/eli/reg/2024/1689/oj

Chen et al., Evaluating Large Language Models Trained on Code (HumanEval, pass@k): https://arxiv.org/abs/2107.03374

Papineni et al., BLEU: a Method for Automatic Evaluation of Machine Translation (ACL 2002): https://aclanthology.org/P02-1040/

Lin, ROUGE: A Package for Automatic Evaluation of Summaries (ACL Workshop 2004): https://aclanthology.org/W04-1013/

Post, A Call for Clarity in Reporting BLEU Scores (SacreBLEU, WMT 2018): https://aclanthology.org/W18-6319/

Zhang et al., BERTScore: https://arxiv.org/abs/1904.09675

Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena: https://arxiv.org/abs/2306.05685

Liu et al., G-Eval: https://arxiv.org/abs/2303.16634

Shankar et al., Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences: https://arxiv.org/abs/2404.12272

Es et al., RAGAS: Automated Evaluation of Retrieval Augmented Generation: https://arxiv.org/abs/2309.15217

Yao et al., tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains: https://arxiv.org/abs/2406.12045

Chiang et al., Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference: https://arxiv.org/abs/2403.04132

Bradley, R. A., and Terry, M. E. (1952). Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika 39(3/4): doi:10.2307/2334029

Hunter, D. R. (2004). MM algorithms for generalized Bradley-Terry models. The Annals of Statistics 32(1): doi:10.1214/aos/1079120141

Russo et al., A Tutorial on Thompson Sampling: https://arxiv.org/abs/1707.02038

Agrawal and Goyal, Analysis of Thompson Sampling for the Multi-armed Bandit Problem: https://arxiv.org/abs/1111.1797

Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 22(158): doi:10.1080/01621459.1927.10502953

Lin, J. (1991). Divergence measures based on the Shannon entropy. IEEE Transactions on Information Theory 37(1): doi:10.1109/18.61115

NIST/SEMATECH e-Handbook of Statistical Methods (control charts): https://www.itl.nist.gov/div898/handbook/pmc/section3/pmc31.htm

Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention: https://arxiv.org/abs/2309.06180

Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs: https://arxiv.org/abs/2312.07104

Zhong et al., DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving: https://arxiv.org/abs/2401.09670

Google, Site Reliability Engineering (the SRE Book): https://sre.google/sre-book/table-of-contents/ — chapters on monitoring distributed systems, managing incidents and postmortem culture.

Google, The Site Reliability Workbook: https://sre.google/workbook/table-of-contents/ — in particular Implementing SLOs and Alerting on SLOs.

Mitchell et al., Model Cards for Model Reporting (FAT* 2019): https://arxiv.org/abs/1810.03993