LLM Foundations & Prompting

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Examples across this sub-section target Python 3.12 with the current anthropic and openai Python SDKs (anthropic 1.8.x), Hugging Face transformers 5.17 (with tokenizers 0.23.2) and tiktoken 0.14.0, TRL 1.14, DSPy 3.4.0, Ollama 0.34 serving llama3.1:8b (and the thinking-capable qwen3:8b), and Java 21 with Spring Boot 4.1 / Spring AI 2.0.1, calling the hosted models claude-sonnet-5, claude-haiku-4-5, claude-opus-5-5 and claude-fable-5-1, alongside OpenAI’s current flagship line (gpt-6-luna, gpt-6-sol, gpt-6-astra), with gpt-6-sol as the OpenAI model the reasoning and general examples call. This baseline was checked and re-verified as current against each vendor’s own documentation on 2026-09-28; every page states any narrower version it targets in its own intro, and this landing page’s baseline is the one to trust if an individual page and this page ever appear to disagree after a later update.

Most pages reuse DocsAssistant, the same running example the Vector Databases & RAG section uses: an agent that answers questions about this site’s own AsciiDoc pages, using one embedding model and one chat model across every stack so the examples stay interoperable (see that page’s scenario table for the exact models, corpus and indexes). This sub-section picks the story up one level below retrieval: what the chat model itself is, how it turns text into tokens and those tokens into an answer, how to prompt it well, and how to choose which model to call in the first place.

Reading order

The 19 pages below build up in three stages, plus a cheat sheet. How LLMs work explains the model itself, from what an LLM is down to why it sometimes gets things wrong. Prompt and context engineering covers how to talk to that model reliably, and how to manage what occupies its context window as an application grows. Choosing models is about picking among today’s actual models and estimating what a call will cost. Read them in order if you are new to LLMs; jump straight to a page if you already know the basics and need one topic.

What’s covered

How LLMs work

What Is an LLM? — language model to LLM to foundation model, the three-layer AI stack, and what self-supervised pre-training on next-token prediction actually produces.
Tokens & Tokenization — BPE and byte-level tokenizers, chat templates, counting tokens, and the token-to-cost-and-latency formula.
Transformer Architecture — the decoder-only forward pass, self-attention, RoPE, MQA/GQA, MoE and Mamba-style alternatives.
How LLMs Are Trained — pre-training, scaling laws, SFT, preference tuning (RLHF, DPO) and RL for reasoning (RLVR/GRPO).
Sampling & Decoding — temperature, top-k/top-p, stop sequences, logprobs, and which parameters reasoning models ignore.
Context Windows & Memory — the context window as working memory, the KV cache, long-context degradation, and prompt caching economics.
Reasoning Models — extended thinking, effort and budget parameters, and when reasoning is worth its extra latency and cost.
Multimodal Models — ViT patches, CLIP, and native image/PDF and audio input and output.
Hallucinations & Limitations — why models hallucinate, related failure modes, and the mitigations that actually help.

Prompt and context engineering

Prompt Engineering Fundamentals — roles, the anatomy of a prompt, the specificity checklist, and few-shot examples.
Prompting Techniques — chain-of-thought, prompt chaining, self-consistency, tree-of-thought, self-critique, role prompting and prefilling.
Structured Output & Tool Calling — JSON-schema structured outputs, constrained decoding, Pydantic/Jackson and Spring AI’s entity(), and tool calling.
Context Engineering — context budgets, just-in-time retrieval, compaction, structured note-taking and progressive disclosure.
Managing Prompts — prompts as versioned code, golden-set regression testing, prompt improvers, and DSPy-style optimisation.

Choosing models

The Model Landscape — proprietary and open-weight families, model tiers and modalities.
Choosing a Model — a selection workflow, the build-vs-buy criteria table, and routing/cascade patterns.
Benchmarks & Leaderboards — what MMLU-Pro, GPQA, SWE-bench and friends measure, and how to read arena-style leaderboards.
Cost, Latency & Throughput — TTFT, TPOT, token pricing and batch APIs, and the trade-offs behind picking a smaller model or effort level.
AI Application Architecture — the end-to-end reference architecture these pieces form once wired together: context enhancement, guardrails, routing, caching, agent patterns, monitoring and the feedback loop.

Cheat sheet

Cheat Sheet (PDF) — a single-page, printable summary of every concept in this sub-section.

mindmap root((LLM Foundations & Prompting)) How LLMs work What Is an LLM? Tokens & Tokenization Transformer Architecture How LLMs Are Trained Sampling & Decoding Context Windows & Memory Reasoning Models Multimodal Models Hallucinations & Limitations Prompt and context engineering Prompt Engineering Fundamentals Prompting Techniques Structured Output & Tool Calling Context Engineering Managing Prompts Choosing models The Model Landscape Choosing a Model Benchmarks & Leaderboards Cost, Latency & Throughput AI Application Architecture Cheat sheet

Bibliography

The following sources were consulted while preparing this sub-section’s pages. Every code example was verified against the official documentation and current SDKs — never copied from a book — and where a book’s example was outdated, the page in question says so in prose and shows the current API.

Requester-provided books, each a consulted reference for concepts and the practitioner’s angle only:

Huyen, Chip. AI Engineering: Building Applications with Foundation Models. O’Reilly Media, 1st ed., Dec 2024 (copyright 2025). ISBN 978-1-098-16630-4. Publisher page: https://www.oreilly.com/library/view/ai-engineering/9781098166298/ . Official code repository (resources, not runnable code): https://github.com/chiphuyen/aie-book .
Alammar, Jay, and Maarten Grootendorst. Hands-On Large Language Models: Language Understanding and Generation. O’Reilly Media, 1st ed., Sep 2024. ISBN 978-1-098-15096-9. Publisher page: https://www.oreilly.com/library/view/hands-on-large-language/9781098150952/ . Official code repository: https://github.com/HandsOnLLM/Hands-On-Large-Language-Models .
Taulli, Tom. AI-Assisted Programming: Better Planning, Coding, Testing, and Deployment. O’Reilly Media, 1st ed., Apr 2024. ISBN 978-1-098-16456-0. Publisher page: https://www.oreilly.com/library/view/ai-assisted-programming/9781098164553/ . Official code repository: https://github.com/ttaulli/AI-Assisted-Programming-Book .
Osmani, Addy. Beyond Vibe Coding: From Coder to AI-Era Developer. O’Reilly Media, 1st ed., Aug 2025. ISBN 979-8-341-63475-6. Publisher page: https://www.oreilly.com/library/view/beyond-vibe-coding/9798341634749/ ; no official code repository. Companion web edition: https://beyond.addy.ie/ .

oreilly.com answers automated fetches with an HTTP 403 response, but every publisher page above is a live, reachable page when opened in a browser.

Official documentation:

Anthropic:
Models overview, List models, Choosing a model and Optimizing for cost and intelligence, Claude models overview, Claude Sonnet 5 overview and Claude Haiku 4.5 migration guide.
Prompt engineering overview and Claude prompting best practices, Use XML tags to structure your prompts and Use examples (multishot prompting).
Thinking, Extended thinking and Effort.
Structured outputs, Tool use overview, Define tools and Strict tool use.
Prompt caching, Context windows, Token counting and the count_tokens API reference.
Pricing, Batch processing, Streaming Messages and the Messages API reference.
Vision, PDF support and Reduce hallucinations.
Agent Skills documentation, the Claude Console and the Claude Cookbooks metaprompt notebook.
OpenAI:
Prompt engineering guide, Models and Reasoning models and Prompt Optimizer.
Structured model outputs, Chat Completions API reference, Chat Completions object and Streaming API responses.
Pricing, Batch API and Prompt caching.
GitHub Copilot — Prompt engineering.
Google — Gemini models page.
Hugging Face:
transformers documentation, pipeline tutorial and LLM generation tutorial, text generation (GenerationMixin.generate()) reference, model outputs reference and CLIP model documentation.
AutoTokenizer / PreTrainedTokenizer reference and chat templates guide.
LLM inference optimization, TRL documentation and TRL SFTTrainer.
the Hugging Face Hub and its organization pages for Meta Llama, Mistral AI, Qwen, DeepSeek, Google, Microsoft and OpenAI.
tiktoken repository and README.
Ollama:
API reference, generate a response, list local models, Thinking and Structured outputs.
Spring AI (the Spring AI reference documentation):
ChatClient API, Structured output converter and Prompts reference.
DSPy documentation.
Python templating: Python string.Template and Jinja2 API documentation.
Java: Jackson — Migrating to Jackson 3 and Hibernate Validator reference guide.
SciPy — minimize (BFGS method).
llama.cpp — Grammars (GBNF).

Leaderboards:

Engineering articles:

Anthropic, Effective context engineering for AI agents and Building effective agents.

Specifications and standards:

OpenTelemetry — Semantic conventions for generative AI systems.

Papers:

Vaswani et al., Attention Is All You Need.
Hoffmann et al., Training Compute-Optimal Large Language Models (Chinchilla).
Ouyang et al., Training language models to follow instructions with human feedback (InstructGPT/RLHF).
Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model (DPO).
Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (GRPO).
Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.
Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models.
Yao et al., Tree of Thoughts: Deliberate Problem Solving with Large Language Models.
Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models.
Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding (RoPE).
Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness.
Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.
Jiang et al., Mixtral of Experts.
Gu & Dao, Mamba: Linear-Time Sequence Modeling with Selective State Spaces.
Madaan et al., Self-Refine: Iterative Refinement with Self-Feedback.
Zhou et al., Large Language Models Are Human-Level Prompt Engineers (APE).
DeepSeek-AI, DeepSeek-V3 Technical Report.
Yang et al., Qwen3 Technical Report.
Liu et al., Lost in the Middle: How Language Models Use Long Contexts.
Holtzman et al., The Curious Case of Neural Text Degeneration (nucleus/top-p sampling).
Chiang et al., Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference.
Wang et al., MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark.
Rein et al., GPQA: A Graduate-Level Google-Proof Q&A Benchmark.
Liu et al., Is Your Code Generated by ChatGPT Really Correct? (EvalPlus / HumanEval+).
Zhou et al., Instruction-Following Evaluation for Large Language Models (IFEval).
Muennighoff et al., MTEB: Massive Text Embedding Benchmark.
Enevoldsen et al., MMTEB: Massive Multilingual Text Embedding Benchmark.
Chen et al., FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Accuracy.
Huang et al., A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions.
Sennrich et al., Neural Machine Translation of Rare Words with Subword Units (BPE).
Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (ViT).
Radford et al., Learning Transferable Visual Models From Natural Language Supervision (CLIP).
Bommasani et al., On the Opportunities and Risks of Foundation Models.

These books are consulted references, not the primary source of any example on this sub-section’s pages. On any discrepancy between a book and a tool’s own official documentation, that documentation is authoritative, and every code example was verified against the versions stated at the top of this page when the pages were written.