MIS 752 · Lab 7 · kenneth-holstad
Catch the AI Lying: an audited pass/fail harness for one small free model
Ten gold clinical question/answer pairs, one small free model, three rulers (exact match, a lenient contains check, and an LLM-as-judge run twice), and the measured reliability of the ruler itself. Built in Google Colab on OpenRouter's free tier: 40 requests, $0, no PHI. These numbers are this run's and will vary.
The 30-second read
THE GOLD STANDARD
items : 10 (4 numeric, 3 code-lookup, 1 short-phrase, 1 yes-no, 1 one-word)
written before any model output was looked at -- that ordering is what makes it a
standard instead of a rationalisation
file : data/gold_qa.csv
THE MODEL UNDER TEST
model : liquid/lfm-2.5-2.6b:free at temperature 0.0
judge : nvidia/nemotron-3-ultra-550b-a55b:free, two passes at 0.7 and 1.1
requests used : 40 this session, of ~50 free per day
API cost : $0.000000 (every model here is on
OpenRouter's free tier, so the cost column is $0 by construction --
which is exactly why cost is in this table and not on a chart axis)
ACCURACY ON THE SAME 10 ITEMS, THREE RULERS
exact match : 6/10 = 60% deterministic, free, brittle
contains gold : 6/10 = 60% lenient, false PASSES
judge, pass A : 7/10 = 70% reads meaning
judge, pass B : 7/10 = 70% same questions, new dice
IS THE RULER ITSELF RELIABLE? (the part almost nobody ships)
judge vs itself : agreement 1.0, kappa 1.0 = almost perfect
exact vs judge : agreement 0.9, kappa 0.7826 = substantial
mean |score gap| : 0.00 points, over the 10 of 10 items scored by both passes
unparseable : 0/10 judge replies ignored the output format
resolution : one flipped verdict moves accuracy by 10 points at n=10
DISAGREEMENT DIRECTION
judge passed what exact match failed : 1
judge failed what exact match passed : 0
STEP 5 -- YOUR WEEK 4 ANSWERS, RE-SCORED
10 of your Lab 4 answers re-scored; Week 4's keyword rubric averaged 76/100 and this week's judge 43/100, with 5 row(s) more than 25 points apart.
Your judge repeated itself on every item
One dot per judge pass per gold item, joined by a connector, sorted so the biggest self-disagreement is on top. The number at the right of each row is pass B minus pass A. Row labels are prefixed EM+ / EM- for whether exact match passed that item, so three facts sit on one row without relying on colour alone. A judge that cannot repeat itself is not a measuring instrument, and this is the cheapest way to find out.
Exact match is the brittle one
One dot per gold item on the judge’s own 0–10 score, sorted ascending. Marker colour and shape both carry the exact-match verdict, so the chart still reads in greyscale. The dotted line is the score at which a judge verdict normally flips to PASS. Dots far from that line on the wrong side are items where the judge’s verdict and its own score do not agree — a second, free thing to audit.
The pass/fail table
The week’s deliverable: every gold item, what the model said, and what each ruler decided. A red bar on the left edge of a row marks an item to read first: the judge flipped between passes, the two scorers disagreed, or the judge returned no verdict. Judge A ran at temperature 0.7, judge B at 1.1. Hover an answer for its full text.
| Item | Gold | Model said | Exact | Contains | Judge A | Judge B |
|---|
| icd10_t2dmcode lookup | E11.9 | E11.9 | ✓ PASS | ✓ PASS | ✓ PASS10/10 | ✓ PASS10/10 |
| icd10_amicode lookup | I21.9 | I20.0 | ✗ FAIL | ✗ FAIL | ✗ FAIL0/10 | ✗ FAIL0/10 |
| snomed_t2dmcode lookup | 44054006 | <|tool_call_start|>[google(query='SNOMED CT Diabetes mellitu… | ✗ FAIL | ✗ FAIL | ✗ FAIL0/10 | ✗ FAIL0/10 |
| acetaminophen_max_gnumeric | 4 | <|tool_call_start|>[google(query='maximum total daily dose o… | ✗ FAIL | ✗ FAIL | ✗ FAIL0/10 | ✗ FAIL0/10 |
| a1c_8_eag_mgdlnumeric | 183 | 183 | ✓ PASS | ✓ PASS | ✓ PASS10/10 | ✓ PASS10/10 |
| hypo_level1_mgdlnumeric | 70 | 70 | ✓ PASS | ✓ PASS | ✓ PASS10/10 | ✓ PASS10/10 |
| hipaa_safe_harbornumeric | 18 | 18 | ✓ PASS | ✓ PASS | ✓ PASS10/10 | ✓ PASS10/10 |
| warfarin_reversal_agentshort phrasescorers differ | 4-factor prothrombin complex concentrate | Prothrombin complex concentrate (PCC) | ✗ FAIL | ✗ FAIL | ✓ PASS10/10 | ✓ PASS10/10 |
| sjw_warfarin_yesnoyes no | NO | NO | ✓ PASS | ✓ PASS | ✓ PASS10/10 | ✓ PASS10/10 |
| atorva80_intensityone word | high | high | ✓ PASS | ✓ PASS | ✓ PASS10/10 | ✓ PASS10/10 |
| Passed, out of 10 | 6/10 | 6/10 | 7/10 | 7/10 |
Auditing the instrument
| Comparison | Agreement | Disagree | Unparseable | Kappa | Kappa reads as | Mean |score gap| |
|---|
| exact match vs judge (pass A) | 0.90 9/10 | 1 | 0 | 0.78 | substantial | n/a (two different rulers) |
| judge pass A vs judge pass B | 1.00 10/10 | 0 | 0 | 1.00 | almost perfect | 0.00 points, over the 10 of 10 items scored by both passes |
Confusion matrix: rows are the exact-match verdict, columns the judge pass A verdict. Blue cells are agreement, red cells are disagreements; UNPARSEABLE is a column, not a dropped row, so an instrument failure can never flatter the accuracy.
| Exact ↓ Judge A → | PASS | FAIL | UNPARSEABLE | Total |
|---|
| Exact: PASS | 6 | 0 | 0 | 6 |
| Exact: FAIL | 1 | 3 | 0 | 4 |
Your Week 4 answers, re-scored
The same answers you graded in Lab 4 with a keyword rubric, now graded by this week’s judge against a written reference answer. Both series are on a 0–100 scale, sorted so the biggest gap is on top, with the gap in points written on each row. 10 of your Lab 4 answers re-scored; Week 4's keyword rubric averaged 76/100 and this week's judge 43/100, with 5 row(s) more than 25 points apart.
How this was measured
Ten gold question/answer pairs were written before any model output was looked at and saved as data/gold_qa.csv. One model (liquid/lfm-2.5-2.6b:free) answered all ten at temperature 0.0 through OpenRouter, with a fixed system prompt that is identical for every item, because the prompt is part of the instrument.
Three rulers scored the same answers. Exact match normalizes case, Unicode look-alikes, Markdown emphasis, answer labels, surrounding quotes and trailing punctuation, then requires the whole string to equal the gold answer or one of its accepted variants. Contains-gold is a deliberately lenient diagnostic that produces false passes. The LLM judge (nvidia/nemotron-3-ultra-550b-a55b:free) reads the question, the gold answer and the candidate, and returns VERDICT / SCORE / REASON in three lines; the parser clamps the score to 0-10 and returns UNPARSEABLE rather than guessing when the format was not respected.
The judge was run twice, at temperature 0.7 and again at 1.1, and the two passes are compared with raw agreement and Cohen's kappa (chance-corrected, read on the Landis and Koch scale). Kappa bands are a convention, not a law, and at n=10 a single flipped item moves accuracy by 10 points, so treat every number here as a pilot measurement with its denominator attached.
Every model called is on OpenRouter's free tier at $0 per token, verified against the live catalog at run time; prices are fetched, never hard-coded. Answers and verdicts are cached to disk, so re-running a cell never spends quota twice. The harness counts requests against a daily budget and refuses to exceed it.
This is a ten-item teaching pilot, not a clinical validation. The gold items are self-contained vignettes with no PHI and no real patients, and nothing here is medical advice or a cleared clinical device.