AI-103 Lab: Evaluate and Monitor AI Agents in Python

14. Evaluation and Observability is a 17-minute hands-on lab from the free AI-103 Developing AI Apps and Agents on Azure course, in which you measure the quality of AI answers and monitor an agent once it is running. It is for developers preparing for AI-103 and for anyone who has shipped an AI feature, seen an evaluator score of 4.2 and had no idea whether that was good.

The lesson’s main message: a judge you have not checked is just a number generator. You learn to score answers on separate metrics, calibrate the judge against human labels, trace where time and tokens go, and set signals that warn you before users complain. It is part of the course on the AI-103 course page.

What this lab covers

The lab has two exercises.

Exercise 1: evaluate, then calibrate the judge.

• Four separate evaluators. Groundedness checks whether every claim comes from the retrieved context. Relevance checks whether the answer addresses the question. Coherence checks readability and internal consistency. Safety scans for harmful content. The lab runs all four on four answers to the same question: a good one, an ungrounded one, an irrelevant one and an incoherent one.
• Why one blended score is dangerous. Averaging the four metrics would hide the ungrounded answer’s fabrication. The lesson’s rule is to report metrics separately and gate on the weakest one. It also notes that a fabrication is a groundedness failure, not a safety failure, so safety alone will not catch it.
• Calibrating a model-as-judge. A small golden set of five human-labelled answers goes through a naive judge. The judge marks a correct paraphrase as bad, so it agrees with the humans four times out of five. The lesson explains why 80% agreement means a reported score is not yet evidence, and why you should report agreement alongside the score.

Exercise 2: trace, measure and set quality signals.

• Traces. A trace is the structured record of one run: every span, its kind, input, output and timing. You build one by hand by wrapping tool calls, then print the spans, the agent’s step log and a token breakdown. The breakdown shows input tokens dominating cost, so shortening context matters more than shortening answers.
• Five signals. Completion rate, steps per run, tokens per run, tool calls per run and tool error rate. A script compares today’s numbers to a baseline and flags any change over 15%.
• Reading alerts together. In the example, tool calls per run collapse while steps and tokens rise. The lesson reads this as one cause, confident hallucination: the agent stopped using tools and is answering from its own knowledge, and no error rate shows it.
• Slicing and drift. Cut metrics by deployment, prompt version, tenant and entry point, monitor a fixed golden set on a schedule, and alert on percentiles rather than averages.

What you will do

These are the steps the instructor takes in the video, in order, with the moment each one starts. Watch the video for the full walkthrough.

1. You run an evaluator and get a score of 4.2. (0:25)
2. We start by using built-in evaluators to assess AI quality, so you can automate quality checks with invoke AI evaluation. (0:44)
3. Let’s start by writing the evaluators. (1:48)
4. We run four evaluators on four answers to the same F-21 question. (2:34)
5. We’ll define a context and a question, then run four hand-crafted answers through our evaluators. (3:15)
6. We start by importing helper modules for display and safety moderation. (3:31)
7. Then we set a fixed context about a firmware fault and a question asking what it means and how to clear it. (3:35)
8. We are going to run a small golden set of five human labeled answers through a naive judge. (6:38)
9. We start with a list of five labeled answers. (6:56)
10. Then we create a judge using a small GPT model with a set of rules. (7:04)
11. Let’s start by capturing the trace and the breakdown. (9:18)
12. We create a knowledge index and an empty spans list. (10:48)
13. We’ll run a Python script that compares a baseline snapshot to today’s metrics. (13:57)
14. We start by importing our shared lab utilities for banners, bullets, and tables. (14:11)
15. Then we define two dictionaries. (14:16)

Where it fits in the AI-103 course

This is module 14 of 23. Just before it, 13. Autonomy with Safeguards bounds an autonomous agent with stopping rules and recovery levels; this lab measures the quality of what that agent produces. Groundedness, first introduced in 07. Retrieval-Augmented Generation, returns here as a score you compute.

After it, the course moves to 15. Image and Video Generation and the other modalities. The monitoring ideas come back in 22. Deployment, CI/CD, Monitoring, Cost, where quality and cost are watched in production.

How to study this lab

1. Watch the opening to see why a lone score like 4.2 tells you nothing.
2. Run the four evaluators and check that each failing answer fails on the metric you expect.
3. Calculate the blended average yourself and see how it hides the fabrication.
4. Run the judge calibration. Find the paraphrase the judge gets wrong, then try fixing its rule.
5. Build the hand-made trace and look at the prompt share of tokens.
6. Change the baseline numbers in the signals script and see which alerts fire.
7. Check yourself. The course’s mock exam has an area on evaluating, deploying and operating; the 10-question sample is free, and the full mock comes with a subscription.

Frequently asked questions

Why not use one overall quality score?

The lesson shows that an answer can be safe and fluent but made up, or grounded but off-topic. Averaging hides those failures, so it reports each metric separately and gates on the weakest.

Can I trust an LLM judge?

Only after checking it against human labels. In the lab, a naive judge agrees with humans on four of five answers and misses a correct paraphrase, which is why the lesson says to report agreement alongside any score.

Which metrics should I monitor for an agent?

The lab uses five: completion rate, steps per run, tokens per run, tool calls per run and tool error rate, compared against a baseline.

Is this lab free?

Yes. It is part of the free AI-103 course on YouTube; follow all modules from the AI-103 course page.