Evals in AI: A Deep Dive — Tejas Kumar, IBM

1. Tejas Kumar Pairs Evals With Runtime Harnesses

Timestamp: 00:02:03 to 00:06:06

Tejas Kumar argues that AI evaluations provide ahead-of-time reliability before deployment, whereas agent harnesses provide just-in-time reliability at runtime. He illustrates runtime harnesses with a Claude Code example that caught an HTML prompt injection by treating tool results as non-instructional data. Together, evals and harnesses act like two pedals on a bicycle, enabling teams to run cheaper models safely.

Significance (High): Separating build-time evaluation from runtime harness enforcement gives engineering teams a clear two-layer defense architecture, reducing reliance on expensive frontier models.

Sources in support: Tejas Kumar (AI Engineer at IBM (watsonx.data team))

2. Preventing Agent Exploits Like Instagram's

Timestamp: 00:06:07 to 00:08:30

Evals help security teams discover dangerous agent behavior before attackers exploit it in production. Kumar recounts an incident where attackers used a VPN to bypass location checks and convinced an Instagram support chatbot with elevated API permissions to attach a secondary email to 20,225 accounts. A solid ahead-of-time eval suite or runtime harness would have blocked unverified account mutations before launch.

Significance (High): Connecting elevated API mutations to an uncalibrated chatbot exposes user accounts to trivial social engineering unless pre-deployment evals test adversarial permission scenarios.

Sources in support: Tejas Kumar (AI Engineer at IBM (watsonx.data team))

3. Defining Evals as Fuzzy Unit Tests

Timestamp: 00:09:47 to 00:15:20

From first principles, evals act as reliability instrumentation for non-deterministic systems, functioning like fuzzy unit tests across scenarios. Every production-grade eval suite shares four core components: test cases, expected behavior, a judge, and result aggregation. These components work together to verify whether an agent followed policy, called the proper tools, and resolved the user's issue.

Significance (Medium): Framing evaluations around four concrete components demystifies AI testing for software engineers accustomed to deterministic unit tests.

Sources in support: Tejas Kumar (AI Engineer at IBM (watsonx.data team))

4. Four Biases That Make LLM Judges Lie

Timestamp: 00:15:21 to 00:21:10

Kumar outlines four evaluation techniques: exact match, schema validation, pairwise comparison, and LLM-as-a-judge, noting that pairwise comparison is roughly 39% more vulnerable to attack vectors than pointwise evaluation. Furthermore, LLM judges mislead teams through four specific biases: position bias, sycophancy toward polite or pleasing answers, self-preference for outputs from their own model family, and a preference for verbose responses.

Significance (High): Unchecked judge biases create false confidence in broken agents; switching from pairwise to pointwise scoring and separating model families directly reduces evaluation distortion.

Sources in support: Tejas Kumar (AI Engineer at IBM (watsonx.data team))

Neutral sources: Lenny (Conference Attendee)

5. Calibrating Judges and Gating CI at 80%

Timestamp: 00:21:11 to 00:27:50

Training an LLM judge means calibrating its verdicts against a golden dataset of roughly 30 human-scored question-and-answer scenarios until it reaches 80% to 85% agreement, according to Kumar. He warns that 100% agreement is a red flag for overfitting or sycophancy, since even human support teams only agree about 80% of the time. Once calibrated, the judge acts as a CI gate with pinned model versions while teams skim live traffic and retrieve living policy via tools like OpenRAG.

Significance (High): Setting an explicit 80% to 85% human-agreement band turns subjective model grading into an enforceable CI deployment gate without overfitting to static test cases.

Sources in support: Tejas Kumar (AI Engineer at IBM (watsonx.data team))

6. Why Substring Matching Passes Bad Answers

Timestamp: 00:28:55 to 00:32:07

In a live coding demonstration using Cursor and Vitest, Kumar transitions from a deterministic addition unit test to a customer support scenario where a buyer asks to return headphones after 20 days. When testing an assertion that expects the response to contain the substring 'cannot', a bad response stating 'Yes, you can return them for free, you cannot pay for the shipping' passes the test.

Significance (Medium): Demonstrating a false-positive test pass with a single word change proves why keyword assertions fail on generative text.

Sources in support: Tejas Kumar (AI Engineer at IBM (watsonx.data team))

7. Triggering Self-Preference Bias Live in GPT-3.5

Timestamp: 00:32:08 to 00:36:18

Kumar replaces the substring check with a pairwise LLM judge using GPT-3.5 Turbo, which initially selects the correct refund denial across five runs. However, when he uses GPT-3.5 Turbo itself to generate the wrong, policy-violating approval option, the GPT-3.5 Turbo judge flips and consistently picks the wrong answer because it recognizes its own model family's phrasing. Injecting the explicit 14-day return policy string into the judge's prompt immediately eliminates the self-preference failure.

Significance (High): Reproducing self-preference bias live in the terminal shows that ungrounded judges prioritize stylistic familiarity over factual correctness until anchored by explicit policy.

Sources in support: Tejas Kumar (AI Engineer at IBM (watsonx.data team))

8. Iteratively Bridging Policy Gaps Across 30 Cases

Timestamp: 00:36:19 to 00:42:59

Moving from a single test to a 30-scenario dataset with human pass/fail verdicts, Kumar switches the GPT-3.5 Turbo judge to a pointwise prompt and observes high initial disagreement. After normalizing case sensitivity, he inspects specific disagreements, such as a 10-day yoga mat return, and adds missing policy rules requiring change-of-mind returns to be unused and in original packaging.

Significance (Medium): Walking through failed test cases illustrates that calibrating a judge is primarily an exercise in uncovering implicit human rules and writing them into explicit policy.

Sources in support: Tejas Kumar (AI Engineer at IBM (watsonx.data team))

9. Upgrading to GPT-4o Mini and Enforcing CI Gates

Timestamp: 00:43:00 to 00:48:07

When GPT-3.5 Turbo continues failing on the yoga mat scenario despite having the right context, Kumar upgrades the judge to GPT-4o mini, noting that low-tier models eventually hit a capability floor. He resolves a final sycophancy failure where a loyal customer asks for an exception at six weeks by stating in the policy that return rules are non-negotiable regardless of customer loyalty, reaching 27 out of 30 passing cases (90% agreement).

Significance (High): Combining a modest model upgrade with anti-sycophancy policy language and an 80% threshold assertion turns a flaky test file into a stable CI gate.

Sources in support: Tejas Kumar (AI Engineer at IBM (watsonx.data team))

10. Live Policy Retrieval With OpenRAG and Docling

Timestamp: 00:48:08 to 00:51:52

Kumar demonstrates a visual web app and IBM's open-source OpenRAG platform to show how production judges retrieve living policy rather than relying on hardcoded strings. OpenRAG uses Docling to parse unstructured files like PDFs, spreadsheets, audio, and video into LLM-ready chunks indexed in OpenSearch.

Significance (Medium): Separating policy storage from test code via document ingestion allows non-engineers to update company rules without breaking or rewriting evaluation prompts.

Sources in support: Tejas Kumar (AI Engineer at IBM (watsonx.data team))

11. Controlling Eval Costs With a Testing Hierarchy

Timestamp: 00:51:53 to 00:54:35

Recapping the workshop, Kumar lays out a strict cost-control hierarchy for running evaluations at scale. Teams should start with free deterministic unit tests, move to regular expressions or pattern matching, and then inspect observability traces to validate JSON message envelopes and tool call arguments. Only when deterministic trace checks cannot answer the question should teams invoke an LLM judge, starting with the cheapest viable model before scaling up.

Significance (High): Following a tiered progression from deterministic unit tests and trace assertions to cheap LLM judges keeps CI evaluation costs manageable.

Sources in support: Tejas Kumar (AI Engineer at IBM (watsonx.data team))

12. Synthetic Red-Teaming and Shared Agent Policy

Timestamp: 00:54:36 to 00:59:30

During audience Q&A, Kumar explains that teams lacking live customer traffic can expand their eval datasets using synthetic data and red-team agents prompted with past failures to generate adversarial inputs. Asked whether developers should write evals for personal coding agent skills in tools like Cursor or Claude Code, he says no, because evals belong at the harness-building layer while user-written skills are deterministic instructions.

Significance (Medium): Clarifying synthetic data generation, skill testing boundaries, and shared policy retrieval resolves practical deployment bottlenecks for teams adopting evals.

Sources in support: Tejas Kumar (AI Engineer at IBM (watsonx.data team))

IBM Employment and OpenRAG Promotion (Medium severity)

Type: Professional

Tejas Kumar is employed as an AI engineer on the watsonx.data team at IBM, which develops and distributes OpenRAG and Docling.

Significance: His employment incentivizes framing IBM's OpenRAG and Docling as the ideal production architecture for retrieving living policy into LLM judges.