A gentle guide to evals, observability and AgentOps

LLM Evaluation & Reliability Engineering

Your AI agent shipped. Is it still working? Learn to measure it, see inside it, change it safely and keep it under control.

Get it on Amazon See what's inside

Paperback, 7 × 10 in, 460 pages. Companion code is free and open source.

Cover of LLM Evaluation and Reliability Engineering: a cream cover with a staircase of black squares and one red square.

460pages

21chapters in 6 Parts

101illustrations, diagrams and charts

20chapters with runnable code

800+automated tests in the repo

6appendices, glossary and index

22-second trailer

Your AI agent shipped. Is it still working?

A quick look at what the book covers, if you would rather watch than read.

Evals that hold up
Traces you can read
SLOs and error budgets
Canaries and kill switches

Made with Remotion. Music generated with ElevenLabs. AI-generated content, labelled as such on YouTube.

Get it on Amazon

Why this book

Shipping was the easy part.

"It works. Great. So whose is it?"

On Black Friday morning Sam's dashboard is green. Every request returned a clean 200. Yet for forty minutes a critical outage waited in the low-priority queue, filed by the AI assistant itself. Nothing crashed, so nothing alerted. Building it was one job. Owning it is another.

This book is for the developer who has called an LLM API, shipped a feature, and is now the person responsible for what it does when real people rely on it.

A developer stands in a bright open-plan office looking at a wall of dashboards while the pager on his belt lights up. — Chapter 0. Launch morning in the office: a wall of green panels, a pager at the belt, and the first uneasy question about who owns this system.

Silent

Nothing crashes. The output is simply wrong.

A good ticket in the wrong queue

Slow

It works, too slowly to be useful.

A 30-second wait before any reply

Statistical

Fine on average, bad for a subset, or drifting.

One segment slides while the global number holds

Costly

It works, and the bill doubles.

An agent loop burns the month's budget overnight

Unsafe

It does something it was tricked into.

A forged "supervisor note" triggers a credit

Four skills, one story

Together they are what people call AgentOps.

One fictional company, Crateway Market, and one support assistant called Relay, run through every chapter. Each chapter opens with something that went wrong, explains the idea in plain English, shows a small piece of Python you can run, and ends with one artifact to make for your own system.

Evaluate

Measure whether an LLM system is any good: evals, honest statistics, judges, agent grading, datasets and RAG.

Keep it running

Reliability and observability: retries, failover, traces, SLOs and error budgets, and the quiet decay you would never see.

Change it safely

Cost and capacity, prompts as code, evals in CI/CD, canaries, rollbacks and model upgrades.

Secure and govern

Prompt injection, least privilege, audit trails, regulation, incident response and running the program.

The structure

Six Parts. Six questions. Four levels of responsibility.

Each Part answers one question and ends with a short level-up check: five yes-or-no questions about your own system.

Part I

Stepping Into Production

“Can I trust what comes back, and what it does?”

Level 1: BuilderChapters 0 to 2

Part II

Know If It Is Good

“Is it any good, and can I prove it?”

Level 2: Quality ownerChapters 3 to 8

Part III

Keep It Running

“Does it stay up and stay good, and will I know when it does not?”

Level 3: Reliability and operations ownerChapters 9 to 13

Part IV

Run It Well

“Can I afford it, and can I change it safely?”

Level 3: Reliability and operations ownerChapters 14 to 16

Part V

Secure and Govern

“Is it safe, and is it allowed?”

Level 4: Program ownerChapters 17 and 18

Part VI

Respond and Run the Program

“What do we do when it breaks, and how does the team run this without heroes?”

Level 4: Program ownerChapters 19 and 20

The role ladder

Titles vary and are not settled. What matters is what you are accountable for, and that grows in a rough order. The book is built to climb it.

Level 1

Builder

"It works." The output is checked and the tools are safe.

Part I

Level 2

Quality owner

"It is good, and I can prove it."

Part II

Level 3

Reliability and operations owner

"It stays up, stays good, stays affordable, and changes safely."

Parts III and IV

Level 4

Program owner

"It is secure, governed and recoverable, and it runs without heroes."

Parts V and VI

Chapter map

Twenty-one chapters, from "it works" to "it runs without heroes".

Open a Part to see its chapters. Every chapter ends with questions and answers, and a deliverable you can take back to your own system.

IStepping Into ProductionCan I trust what comes back, and what it does?

00
You Own It Now
What changes between “it works” and “it is in production”, the five new kinds of failure, the role ladder and the six incidents at Crateway.
01
Anatomy of a Production LLM System
A model call as a metered function, how it becomes a feature, the agent loop, and a map of twelve ways it fails.
02
Trusting What the Model Returns and Does
The validation gateway, a repair loop that stops, tool contracts, and idempotency keys so a retry never pays twice.

IIKnow If It Is GoodIs it any good, and can I prove it?

03
From Vibe Checks to Evals
Cases, pass criteria and graders you can trust. Build and run a 30-case eval in plain Python.
04
Numbers You Can Trust
Why one run lies: pass@k, standard errors, 95% intervals, the bootstrap and paired comparison.
05
LLM-as-a-Judge
Judge prompts with anchored rubrics, five biases that make a judge lie politely, and validating a judge with Cohen's kappa.
06
Evaluating Agents
Grade the outcome, not the reply: state checks, trajectories, trigger cases, simulated customers and sandboxes.
07
Datasets, Labels, and Feedback Loops
Golden sets, dataset cards, double-labeling, a contamination guard, synthetic data and the feedback flywheel.
08
Evaluating RAG
Judge retrieval and generation separately, triage a wrong answer with two questions, and treat chunking as an experiment.

IIIKeep It RunningDoes it stay up and stay good, and will I know when it does not?

09
When Calls Fail
Why retries multiply, backoff with jitter, shared retry budgets, deadlines, circuit breakers and step guards.
10
When Providers Fail
Routers and adapters, failover that cannot flood the backup, service tiers, streams that die midway, cancellation.
11
Seeing Inside
Why status codes miss quiet failures: span types, a tracer in a page of Python, OpenTelemetry and handoffs.
12
Healthy Means More Than Up
Golden signals for an agent, SLIs and SLOs, quality SLIs from a judged sample, error budgets and burn-rate alerts.
13
Catching Quiet Decay
Baselines and segment-aware bands, three kinds of drift, PSI, chi-square and KS tests, and alert routing.

IVRun It WellCan I afford it, and can I change it safely?

14
Cost, Latency, and Capacity
Token economics, FinOps in small pieces, cascades and caching, and how to prove a saving without losing quality.
15
Prompts as Code, Evals in CI/CD
Versioned prompts, eval tiers on every pull request, and a non-inferiority gate that blocks or warns.
16
Releasing and Upgrading Safely
The release ladder, shadow testing, a self-rolling-back canary, rehearsing failure and the life of a model.

VSecure and GovernIs it safe, and is it allowed?

17
Securing Agents
Prompt injection, excessive agency, layered guardrails, risk tiers, and safety evals that report over-refusal too.
18
Governing Agents
Least privilege, audit trails, change control, a dated map of the EU AI Act, SOC 2 and NIST (not legal advice), human review.

VIRespond and Run the ProgramWhat do we do when it breaks, and how does the team run this without heroes?

19
Incident Response and Postmortems
When to declare, the containment ladder with a kill switch, compensating actions and the blameless postmortem.
20
Running the Program
A RACI sheet, on-call and runbooks, readiness reviews and game days, a maturity ladder and a 90-day plan.

What you'll make along the way

A 30-case eval from your own real failures
A validation gateway and a tool contract card
A retry policy, an SLO sheet and burn-rate alerts
A CI gate for prompt changes, and a canary plan
A kill-switch runbook and a blameless postmortem
A RACI sheet and a 90-day plan

What it does not cover

Training, fine-tuning and self-hosting models
Voice and multimodal systems
Legal advice (Chapter 18's regulation map is dated and marked as such)
Any one vendor's product as "the answer": tools appear as dated examples in skippable Tool Boxes

Inside the book

101 illustrations and diagrams. Every label is correct.

Each chapter is built around a memorable picture: a suitcase for the context window, a fuse for the circuit breaker, a flight recorder for traces. Diagrams and charts are drawn by code from the same numbers the text uses. Click any image to enlarge it.

Scene illustrations

Diagrams and charts

Companion code

Everything runs offline. No API key. No bill.

A scripted stand-in for the Relay assistant means you can run every example, break it, and watch the tests catch it. Every code listing printed in the book exists in the repository, every printed output is produced by that code, and the tests run on every change.

ch09_when_calls_fail/retry.py (excerpt)

# Classify, back off with full jitter, honour Retry-After, respect a shared budget and a deadline.
RETRYABLE = {408, 429, 500, 502, 503, 504, 529}   # timeouts, throttling, transient server trouble

def amplification(retries, layers):
    """Attempts that reach the provider for ONE request when every layer retries `retries` times."""
    return (retries + 1) ** layers

# ... inside call_with_retry:
        window = min(cap, base * 2 ** n)
        sleep = (retry_after or 0) + rng.uniform(0, window)   # floor + jitter
        if clock.now() + sleep >= deadline:
            return f"{status}: no time left before the deadline"
        if not budget.allow_retry():            # asked last: a refusal costs nothing
            return f"{status}: retry budget is empty"

$ python3 -m venv .venv
$ .venv/bin/pip install -r requirements-dev.txt
$ .venv/bin/python -m pytest -q

No key, no network, no cost

A scripted Relay stand-in and a fake clock, so a 30-second cooldown takes zero seconds in tests.

One folder per chapter

From a metered model call and a validation gateway to a canary controller, an audit trail and a quarterly program report.

✓

Checked, not promised

Printed listings must exist in the repo, printed outputs must be real program output, and CI runs on Python 3.10 to 3.13.

MIT

Free to use in your own work

Open source under the MIT License. Vendor-library snippets live in clearly marked optional/ folders.

Browse the code on GitHub

retries & jittercircuit breakerspass@kCohen's kappaLLM judgeRAG metricstraceparentburn-rate alertsPSI / KS driftnon-inferiority gatecanary rollbackkill switchaudit trailRACI

How to read it

You do not have to read straight through.

You need to have called an LLM API and be willing to read short Python. No data-science background required. Pick the path that matches your job.

Fast path

A busy developer

About 190 pages that give you the core: measure, validate, retry, set targets, defend, respond, own.

Chapters 0, 1, 2, 3, 4, 9, 12, 17, 19, 20

Evals first

Quality owners and eval writers

Everything about measuring quality, then drift, CI gates, safe releases and safety evals.

Chapters 0 to 8, 12, 13, 15, 16, then the safety evals in 17

Ops first

Reliability and operations owners

Keep it up first, then measure it, then watch it, change it and respond when it breaks.

Chapters 0, 1, 2, 9, 10, then 3 to 8, then 11 to 16, 19, 20

Back matter you will actually use

APPENDIX A

Production-readiness checklist

Every deliverable in one place, with blocking groups, owners and evidence dates.

APPENDIX B

Templates

Five forms for the artifacts the chapters ask for.

APPENDIX C

Symptom guide

"If something is wrong right now": find the sentence closest to what you see, then what to do first.

APPENDIX D

Formulas and numbers

One row per formula you will want at the desk, each pointing to the chapter that shows the working.

APPENDIX E

Tooling snapshot

The libraries, standards and services the book names, with version, date and status as of 5 October 2026.

APPENDIX F

Regulatory quick map

A developer's map of EU law, SOC 2 and NIST. Dated, and not legal advice.

Plus a glossary of 174 terms, an index of more than 300 terms, and a further-reading list.

Good to know

Questions, answered plainly.

How this book was made

It grew out of three video courses the author made on LLM evaluation, production reliability engineering and AgentOps. The author planned it, chose its running example and its incidents, and is responsible for what it says. The text and the code were generated with AI language models from the course materials and the author's plan, then directed, edited and checked by the author. The scene illustrations, the pictograms in the diagrams and the cover composition were generated with an AI image model; diagram layouts, labels and charts were drawn by code.

The book checks itself instead of asking for trust: listings must exist in the companion repository, printed outputs must be real, and the tests must pass on every change. Mistakes may still remain, and corrections are posted on the books page.

Who is this book for?

Developers who have called an LLM API, built a prototype or shipped a feature, and are now responsible for what it does in production. You need a little Python and a rough idea of an LLM API call. No data-science background is needed.

Do I need an API key, or to pay for anything, to run the code?

No. Every example outside the optional Tool Boxes runs offline against a small scripted stand-in for the Relay assistant, with no network and no key. You replace one function with a call to your own model when you are ready.

Is it tied to one vendor or framework?

No. Tools appear as examples in clearly dated, skippable Tool Boxes, each with the version and the date it was checked. The ideas outlast the tools, and Appendix E lists them all in one snapshot.

Can I read just the parts I need?

Yes. Chapter 0 lays out three reading paths: a fast path of about 190 pages, an evals-first path and an ops-first path. If something is wrong right now, start with Appendix C, the symptom guide.

Was AI used to write it?

Yes, and the book says so plainly. The text and the code were generated with AI language models from the author's course materials and plan, then directed, edited and checked by the author. The scene illustrations and cover composition used an AI image model; diagrams and charts were drawn by code. The book also checks itself: listings and outputs are verified against the public code.

What format is it, and where do I buy it?

A 460-page paperback (7 × 10 inches), published independently and sold on Amazon. ISBN 979-8160444727.

Where do I report a mistake or find corrections?

On the author's books page, https://omkamal.github.io/books/index.html. Tools, models and regulations move quickly, which is why each one carries a version and a date.

The author

Omar Kamal Hosney

More than thirty years of experience across teaching, software testing, data science and applied AI. He built and led a software testing team; since 2014 the work has been data science and machine learning, and more recently applied generative AI. This book grew out of his three video courses on LLM evaluation, production reliability engineering and AgentOps.

Home All books Companion code LinkedIn

Your agent shipped. Now own it.

The practical, gentle guide to knowing it works, seeing inside it, changing it safely and keeping it under control.

Get it on Amazon