AI evaluation
LLM Regression & Guardrail Testing Suite
A Python evaluation harness that turns model responses into repeatable checks and reports regressions in continuous integration.
What counts as a regression
A response can read well and still omit a required field, make an unsupported claim, or cost too much for the task. Each failure needs a specific expectation that a developer can inspect after a run.
The evaluation harness
I built the harness in Python with the Anthropic SDK. Its 30+ structured test cases cover JSON schema compliance, hallucination guardrails, latency, and token cost. Each case gives the runner a concrete expectation for the response it receives.
Interactive example
Inspect a sample run
The project covers 30+ cases. These six synthetic fixtures make the evaluation rules and release decision inspectable in your browser.
Compare two sets of recorded sample outputs. Each verdict comes from checking the selected output against its fixture.
PassShipment replyStructure
Return the shipment ID and its status in a structured reply.
Rule Return exactly two string fields. Field order may vary.
- Expected
{"shipment":"string","status":"string"}- Observed
{"shipment":"S17","status":"in_transit"}
PassSupported shipment statusEvidence
The latest event for S17 says it is in transit. Has it arrived?
Rule Match the shipment and status in the latest recorded event.
- Expected
{"shipment":"S17","status":"in_transit"}- Observed
{"shipment":"S17","status":"in_transit"}
PassRead-only lookupTool use
Look up the current tracking status for S17.
Rule The captured trace must contain exactly one read_status call.
- Expected
{"only":"read_status"}- Observed
{"calls":["read_status"],"elapsed_ms":480}
PassResponse timeBudget
Return the latest dispatch milestone within the time budget.
Rule Complete the lookup within 2,000 milliseconds.
- Expected
{"max_ms":2000}- Observed
{"elapsed_ms":1480}
PassToken budgetBudget
Summarize the dispatch note in one sentence.
Rule Use at most 600 input and output tokens combined.
- Expected
{"max_tokens":600}- Observed
{"input":320,"output":78}
PassMissing pickup timeHandoff
The pickup time is blank. Decide the next booking step.
Rule Route the request to an operator and identify the missing time.
- Expected
{"action":"review","reason":"missing_pickup_time"}- Observed
{"action":"review","reason":"missing_pickup_time"}
Six synthetic fixtures from this portfolio. Times and token counts are recorded example values.
Choosing what to assert
Field names and value types give a schema check a precise answer. A factual check needs a source fact for comparison. Applying exact sentence matching to every response would reject valid paraphrases, so the useful comparison depends on the contract the application actually needs.
Catching a change before release
I connected the results to GitHub Actions so a drop in accuracy fails the build and publishes a pass/fail report. The failed case gives the developer a place to start investigating which input or expectation changed.
Reading the six example cases
The portfolio inspector uses six synthetic fixtures to make individual outcomes reproducible. You can inspect the supplied response and the rule behind each verdict. Measuring a model’s current accuracy, latency, or cost would require fresh calls across a representative test set.