AI Engineer's How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI: skim's analysis identifies 15 key moments, with 1 potential conflict of interest flagged. Laurie Voss of Arize AI leads a practical workshop on evaluating non-deterministic AI agents using TypeScript, Claude, and Arize Phoenix.
Category: Education. Format: Educational. YouTube video analyzed by skim.
skim AI Analysis
Credibility assessment: Hands-On Live Engineering. Laurie Voss demonstrates every evaluation concept live against an executable TypeScript repository, showing real failures, raw trace data, and exact evaluation scores rather than relying on abstract claims.
Bias assessment: Arize Phoenix Advocacy. As Head of Developer Relations at Arize AI, Laurie Voss builds the entire tutorial around Arize Phoenix, though he notes the underlying OpenTelemetry standards work across competing frameworks and models.
Originality: 82% — Practical Eval Loop. Instead of speaking theoretically about AI testing, the workshop provides a deliberately flawed financial agent and walks through a complete debugging and evaluation cycle including meta-evaluating the judge and model upgrades.
Depth: 88% — End-to-End Methodology. The session systematically dissects deterministic code checks, LLM-as-a-judge rubrics, XML prompt boundaries, precision-recall trade-offs, pass@k vs pass^k reliability metrics, and CI integration.
Key Points (15)
1. Laurie Voss explains why eyeballing fails
Timestamp: 00:02:47 to 00:08:33
Laurie Voss argues that checking an LLM app a few times by hand fails to catch regressions because non-deterministic prompts return different strings on every run. Traces act like detailed logs of every span, while evals run automated tests over those traces to quantify quality.
Significance (High): Framing traces as logs and evals as tests demystifies ML terminology for software engineers and establishes why manual spot-checking breaks down on prompt edits or model upgrades.
Sources in support: Laurie Voss (Head of Developer Relations at Arize AI, co-founder of npm)
2. Code, LLM judges, and humans solve distinct flaws
Timestamp: 00:08:33 to 00:15:22
Code evals run in milliseconds for deterministic checks, LLM-as-a-judge evals grade semantic meaning against a rubric, and human experts catch novel domain failures. Multi-step agents raise the stakes because an early retrieval error can cascade into a polished report on the wrong subject.
Significance (High): Separating deterministic formatting checks from semantic judgment prevents teams from overpaying for LLM calls while guarding against agents that fail or succeed in unexpected ways.
Sources in support: Laurie Voss (Head of Developer Relations at Arize AI, co-founder of npm)
3. Instrumenting a two-turn Claude agent in Phoenix
Timestamp: 00:16:31 to 00:26:01
Using OpenTelemetry and OpenInference via the Phoenix OTEL register call, a two-turn Claude Agent SDK financial analyst is traced automatically in TypeScript. Wrapping the research and report-writing turns in a parent span groups all tool calls, while isolation settings block local Claude Code skills.
Significance (Medium): Automatic OpenTelemetry instrumentation captures every web search and prompt turn without manual log statements, though ES module exports in TypeScript require importing the instrumented SDK copy.
Sources in support: Laurie Voss (Head of Developer Relations at Arize AI, co-founder of npm)
4. Traces expose permission errors, says Laurie Voss
Timestamp: 00:26:02 to 00:31:26
When Laurie Voss runs the financial agent live on Tesla, the agent asks for permission to write a file instead of printing the report because its prompt is ambiguous and permission mode is set to don't ask. Inspecting the Phoenix trace reveals the exact failed file-write tool call.
Significance (Medium): Without trace spans showing internal tool calls, an unattended agent failing on file permissions just returns a cryptic refusal message.
Sources in support: Laurie Voss (Head of Developer Relations at Arize AI, co-founder of npm)
5. Designing adversarial batch queries and criteria
Timestamp: 00:31:26 to 00:36:36
To test the agent beyond happy-path tickers, the 15-query batch includes a multi-company comparison, a fake ticker ZZZQ, and slang queries like asking if Tesla is a buy right now. Defining what constitutes a good report requires consulting non-technical domain stakeholders before writing evals.
Significance (Medium): Synthetic edge cases and slang queries expose how an agent handles invalid inputs early, while involving domain experts prevents engineers from encoding their own biases as quality standards.
Sources in support: Laurie Voss (Head of Developer Relations at Arize AI, co-founder of npm)
6. Reading traces reveals the 116% consensus bug
Timestamp: 00:36:40 to 00:45:30
Reading traces manually and via the Phoenix error analysis skill surfaces unexpected failure modes across the batch, including missing buy/sell/hold calls on Coca-Cola and emojis on Rivian. In the Tesla report, the agent combines percentages from two separate web searches to claim 116% of analysts rated the stock.
Significance (High): Inspecting traces distinguishes retrieval failures from reasoning errors, proving that the 116% analyst consensus bug came from combining two valid search results from different datasets.
Sources in support: Laurie Voss (Head of Developer Relations at Arize AI, co-founder of npm)
7. Open coding and stacking Swiss cheese evals
Timestamp: 00:45:30 to 00:50:32
Laurie Voss advises writing plain-language notes on traces first (open coding) before grouping them into failure categories (axial coding), such as the harness leakage category his coding agent found. Stacking multiple narrow code, judge, and human evals follows the Swiss cheese security model rather than relying on one giant evaluator.
Significance (Medium): Guessing failure categories in advance blinds teams to real bugs like harness leakage, and multiplying failure frequency by severity sets the right fix order.
Sources in support: Laurie Voss (Head of Developer Relations at Arize AI, co-founder of npm)
8. Instant TypeScript code evals catch missing tickers
Timestamp: 00:50:37 to 00:56:12
A deterministic TypeScript code evaluator parses the structured JSON input and checks whether the requested stock ticker appears in the final report, logging span annotations back to Phoenix. Running the check instantly catches two failures where the Tesla report omitted its ticker and the Amazon report only discussed AWS.
Significance (High): Even a basic zero-cost string check catches scope drift immediately, while grading the final outcome rather than the sequence of tool calls preserves agent flexibility.
Sources in support: Laurie Voss (Head of Developer Relations at Arize AI, co-founder of npm)
9. Why faithfulness beats correctness on live data
Timestamp: 00:56:13 to 01:05:35
Using Claude Sonnet 5.5 as a judge, Phoenix's built-in correctness evaluator scores all 16 reports as zero because the judge lacks live October market data past its January knowledge cutoff. Switching to the faithfulness evaluator with raw web search spans and research notes as context scores 14 of 16 as unfaithful with specific explanations.
Significance (High): Evaluating live-search agents with a static correctness prompt fails every run, whereas passing trace search results into a faithfulness eval pinpoints fabricated targets and conflated numbers.
Sources in support: Laurie Voss (Head of Developer Relations at Arize AI, co-founder of npm)
10. Grading tool selection exposes prompt ambiguity
Timestamp: 01:05:50 to 01:10:41
Phoenix's built-in tool selection evaluator checks whether the agent chose appropriate tools from its allowed list for each research and writing turn. Although 13 of 16 writing turns tried to save a file to disk, the judge passed most single attempts because the prompt's instruction to write a report was genuinely ambiguous.
Significance (Medium): When a tool-selection judge sides with an agent's unwanted file-write call, it reveals that the agent's system prompt failed to specify where the output should go.
Sources in support: Laurie Voss (Head of Developer Relations at Arize AI, co-founder of npm)
11. Laurie Voss's five-part custom judge rubric
Timestamp: 01:10:42 to 01:18:23
To check whether reports give concrete buy, sell, or hold advice, Laurie Voss builds a custom actionability judge with five parts: expert role and context, explicit criteria derived from trace errors, XML-tagged data boundaries, tested examples, and constrained tool-call output labels. Removing labeled examples slightly increased agreement with his own loose human labels on Microsoft.
Significance (High): Structuring custom rubrics with XML tags and testing them with and without examples prevents overfitting and clarifies whether the judge or the human grader is being stricter.
Sources in support: Laurie Voss (Head of Developer Relations at Arize AI, co-founder of npm)
12. Treating eval prompts as code and avoiding god evals
Timestamp: 01:19:22 to 01:24:46
Running the custom actionability evaluator yields a 5-to-11 pass/fail split across the 16 reports, creating a measurable hill to climb. Omitting XML tags around examples caused the judge to confuse examples with data, demonstrating why eval prompts must be versioned as code and split into single-dimension checks rather than one god evaluator.
Significance (Medium): Separating evaluators by dimension and distinguishing release-blocking guardrails from aspirational north stars keeps test failures interpretable.
Sources in support: Laurie Voss (Head of Developer Relations at Arize AI, co-founder of npm)
13. Meta-evaluating the judge on a golden dataset
Timestamp: 01:24:47 to 01:38:55
To verify that the custom judge can be trusted, Laurie Voss labels all 16 reports by hand in Phoenix to create a golden dataset split 75/25 into dev and test sets. Running a meta-evaluation experiment on the 12 dev reports achieves 12-for-12 agreement, whereas tightening the rubric to require all four criteria lowers agreement to 11 of 12.
Significance (High): Treating an LLM judge as a classifier measured by precision, recall, and human agreement prevents teams from deploying overly strict rubrics or tuning on their test split.
Sources in support: Laurie Voss (Head of Developer Relations at Arize AI, co-founder of npm)
14. Experiments prove prompt fixes and Haiku 5.5 trade-offs
Timestamp: 01:38:56 to 01:50:53
Mapping each eval failure to a prompt edit raises ticker mentions and actionability to 48 out of 48 across three 16-query experiment runs, while faithfulness stays low at 3 of 48. Upgrading the agent in an experiment from Claude Haiku 4.5 to Haiku 5.5 triples faithfulness and improves consistency at four times the cost, but reveals that Haiku 5.5 conservatively recommends 'hold' on 41 of 45 calls.
Significance (High): Running multi-repetition experiments exposes hidden model behaviors—like Haiku 5.5 achieving consistency by defaulting to 'hold'—that single runs or unmeasured prompt edits would miss.
Sources in support: Laurie Voss (Head of Developer Relations at Arize AI, co-founder of npm)
15. Gating CI builds with Vitest and Phoenix
Timestamp: 01:50:55 to 01:56:04
Once capability evals pass reliably, they become regression evals inside a Vitest test suite (`npm test`) that runs in GitHub Actions with Phoenix as a service container. The suite gates releases on a 100% guardrail for ticker mentions and an 0.80 average threshold for the noisier actionability LLM judge.
Significance (High): Embedding deterministic guardrails and averaged LLM judge thresholds into standard CI pipelines locks in quality gains and enables eval-driven development.
Sources in support: Laurie Voss (Head of Developer Relations at Arize AI, co-founder of npm)
This analysis was generated by skim (skim.plus), an AI-powered content analysis platform by Credible AI. Scores and classifications represent the platform's AI-generated assessment and should be considered alongside other sources.
