AI Engineer's How to Write Your First AI Eval - SallyAnn DeLucia, Arize AI: skim's analysis identifies 11 key moments, with 1 potential conflict of interest flagged. SallyAnn DeLucia, Head of Product at Arize AI, leads a hands-on workshop on replacing unscalable vibe checks with structured AI evaluations.
Category: Education. Format: Educational. YouTube video analyzed by skim.
skim AI Analysis
Credibility assessment: Practitioner Workshop Guide. SallyAnn DeLucia speaks from four years of direct product leadership at Arize AI, including building their internal agent Alex and evaluation suite. Claims are grounded in live demonstrations of trace trees, span filtering, and prompt playground testing, though examples naturally center on Arize's own platform.
Bias assessment: Observability-First Product. As Head of Product at Arize AI, SallyAnn DeLucia centers the entire workflow around Arize AX, OpenInference, and proprietary tools like Alex and Signal. However, she emphasizes framework-agnostic tracing and teaches foundational evaluation design principles that apply across any LLM stack.
Originality: 76% — Hands-On Eval Recipe. Rather than staying at high-level theory, the session walks through an end-to-end practitioner loop: inspecting multi-agent travel planner traces, translating vague goals into six-part pass/fail rubrics, mapping span variables, and iterating on an LLM judge live.
Depth: 78% — Practical Engineering Depth. Clearly distinguishes AI evals from deterministic unit tests, breaks down root, agent, tool, and LLM spans inside multi-agent architectures, compares six evaluation paradigms including newly released decision models, and demonstrates variable mapping and human alignment.
Key Points (11)
1. Why Vibe Checks Fail and Evals Differ from Tests
Timestamp: 00:02:06 to 00:05:15
Relying on informal vibe checks after changing a prompt makes it impossible to detect regressions, compare model versions objectively, or enforce quality gates in CI. Unlike traditional software unit tests that check one input against one output in readable code, AI agents have multiple valid outputs and reveal their execution logic only through traces.
Significance (High): Frames observability traces as the mandatory foundation for AI testing, showing engineering teams why traditional deterministic unit testing breaks down on multi-step agents.
Sources in support: SallyAnn DeLucia (Head of Product at Arize AI)
2. Comparing LLM Judges, Decision Models, and Code
Timestamp: 00:05:15 to 00:07:29
SallyAnn DeLucia categorizes six evaluation approaches across development and production: LLM as a judge, decision models, deterministic code checks, dataset annotations, agent as a judge, and production user feedback or business metrics. While LLM as a judge pairs a label with a reasoning explanation, new decision models like OpenAI's recently released API return fast, low-cost probability labels without explanations.
Significance (High): Gives practitioners a clear selection matrix for choosing between explainable LLM judges, fast decision models, deterministic code checks, and full agent judges.
Sources in support: SallyAnn DeLucia (Head of Product at Arize AI)
3. Connecting Model Provider API Keys in Arize AX
Timestamp: 00:07:29 to 00:10:16
To set up the workshop environment in Arize AX, users log into the platform and add an API key from any preferred model provider such as Anthropic, OpenAI, or Gemini, or configure a custom endpoint. Using the test integration button confirms the API key can make live LLM calls before opening the demo project traces.
Significance (Low): Demonstrates how evaluation platforms decouple judge execution from a single vendor by letting teams plug in any LLM provider or custom endpoint.
Sources in support: SallyAnn DeLucia (Head of Product at Arize AI)
4. Inspecting Travel Agent Traces and User Inputs
Timestamp: 00:10:16 to 00:14:06
DeLucia introduces the demo travel planning agent, which takes user requests containing destination, trip duration, budget, and travel style to generate a day-by-day itinerary. By examining a sample trace for a three-day, $800 trip to Prague requesting local, less touristy options, she shows how to locate inputs, outputs, and span attributes before deciding what to measure.
Significance (Medium): Grounds evaluation design in real user data, showing that engineers must inspect actual agent inputs and outputs before writing grading rubrics.
Sources in support: SallyAnn DeLucia (Head of Product at Arize AI)
5. Reading Root, Agent, Tool, and LLM Spans
Timestamp: 00:14:10 to 00:17:27
A trace tree begins at the root span, defined by parent ID equals null, which holds the initial user input and final itinerary output. Below the root span, individual spans capture units of work across sub-agents like a research agent, external tool calls fetching weather or destination data, and sequential LLM calls where the first selects a tool and the second summarizes its output.
Significance (High): Demystifies multi-agent trace hierarchies so developers can pinpoint whether a failure happened during tool selection, data retrieval, or final synthesis.
Sources in support: SallyAnn DeLucia (Head of Product at Arize AI)
6. Agent Architectures and OpenInference Support
Timestamp: 00:17:34 to 00:18:54
Responding to audience questions on orchestration, DeLucia explains that multi-agent systems can use a supervisor agent to plan sub-agent tasks or rely on modular graph architectures like LangGraph where agents handle distinct tasks step by step. Arize supports these varied setups through OpenInference, a framework-agnostic package offering auto-instrumenters alongside manual and hybrid customization.
Significance (Medium): Clarifies that trace-based evaluation works identically whether an application uses a central supervisor agent or modular LangGraph nodes.
Sources in support: SallyAnn DeLucia (Head of Product at Arize AI)
7. DeLucia Advises Judging One Quality Per Eval
Timestamp: 00:19:03 to 00:22:02
Defining what good means requires replacing vague goals like 'helpful, accurate response' with concrete checks such as matching requested trip days and staying within budget. Because research shows LLM-as-a-judge performs best when evaluating a single criterion at a time, teams should start with a small set of specific evals and expand iteratively, just as Arize grew its internal agent Alex to 32 production evaluations.
Significance (High): Prevents a common practitioner mistake—stuffing multiple rubrics into one LLM judge prompt—and shows how evaluation suites scale in production.
Sources in support: SallyAnn DeLucia (Head of Product at Arize AI)
8. Structuring a Six-Part Pass and Fail Rubric
Timestamp: 00:22:02 to 00:25:19
Before writing an evaluation prompt, practitioners should draft six concrete elements: the task being evaluated, a one-line definition of success, a one-line definition of failure, the specific target being judged, and three bullet points each detailing exact pass and fail conditions. Detailed bullet points give the LLM judge explicit boundaries, much like explaining task expectations to a coworker.
Significance (High): Provides a repeatable six-part template that turns subjective product expectations into structured instructions ready for an LLM judge prompt.
Sources in support: SallyAnn DeLucia (Head of Product at Arize AI)
9. Filtering Trace Spans to Build Eval Datasets
Timestamp: 00:25:19 to 00:27:23
To test an evaluation, engineers must filter traces to isolate the exact span type their rubric targets and save those spans into a dataset. Evaluating overall itinerary adherence requires filtering for root spans where parent ID equals null, whereas evaluating tool retrieval accuracy requires filtering spans by the specific tool name.
Significance (Medium): Connects trace architecture directly to dataset curation, ensuring judges receive only the span inputs and outputs relevant to their specific check.
Sources in support: SallyAnn DeLucia (Head of Product at Arize AI)
10. Testing Judges and Mapping Variables in Playground
Timestamp: 00:27:23 to 00:32:08
Inside Arize's prompt playground, DeLucia converts the pass/fail rubric into an LLM judge template, uses variable mapping to bind prompt placeholders to dataset span attributes, and configures classification labels such as correct, incorrect, or partially correct. Running the judge with a model like 4.1 mini produces labels and explanations across the dataset, allowing rapid prompt iteration before saving the evaluator.
Significance (High): Shows the complete mechanics of assembling, variable-mapping, and running an LLM-as-a-judge prompt against real production trace data.
Sources in support: SallyAnn DeLucia (Head of Product at Arize AI)
11. Scaling with Alignment, Online Evals, and Signal
Timestamp: 00:32:08 to 00:35:09
After building an initial judge on roughly 20 traces, teams should align it against human annotations, schedule it as an online evaluation running continuously on incoming traces, and involve stakeholders across product, engineering, support, and security. DeLucia also highlights how AI assistants like Alex can iterate on eval prompts automatically and how Arize's managed agent Signal consumes evaluation results to surface and fix trace issues.
Significance (Medium): Maps out the mature production lifecycle where evaluations move from manual playground experiments to continuous automated monitoring and agent-assisted debugging.
Sources in support: SallyAnn DeLucia (Head of Product at Arize AI)
This analysis was generated by skim (skim.plus), an AI-powered content analysis platform by Credible AI. Scores and classifications represent the platform's AI-generated assessment and should be considered alongside other sources.
