ThinkThen

Beatles Bench pages
Start
The ten functions
Scripts
Bindings
What Jev knows
Tune your bar
Agents
Run it
Beatles Bench · Tune your bar

diff shows what changed.

The talk's slide: diff shows what changed.
Open the slide at full size.

Ask the same question twice, and diff prints only the answers that changed. A summary comes last. With an answer key, each change says whether it fixed a mistake, made one, or settled a not-sure answer.

Here the page reads two saved runs for the 10 songs on the audit page. The first run asked from the title alone. The second run gave Jev each song's catalog entry too. The slide's last column shows the answers with context. The warning is right. The two runs asked different questions.

Run it

At the band 0.2:0.8, 7 of the 10 answers changed, and all 10 end right.
thinkthen diff memory.jsonl context.jsonl \
  --threshold 0.2:0.8 \
  --key key.jsonl |
jq -c 'if .summary
  then .summary | {changed, right_a, right_b, lost}
  else [.id, .from, .to, .effect]
  end'
Output
thinkthen: diff: warning: the question digest differs in 10 of 10 paired answers. A different question, threshold, or profile gives a different digest.
["A Day in the Life","yes","no","gained"]
["Blackbird","unsure","no","resolved"]
["Glass Onion","unsure","no","resolved"]
["In My Life","unsure","no","resolved"]
["Something","unsure","yes","resolved"]
["Taxman","unsure","no","resolved"]
["The Long and Winding Road","yes","no","gained"]
{"changed":7,"right_a":3,"right_b":10,"lost":0}
exit 0

Change the bar

At a bar of 0.5, 5 changed. Each was a wrong yes that turned right.
thinkthen diff memory.jsonl context.jsonl \
  --threshold 0.5 \
  --key key.jsonl |
jq -c 'if .summary
  then .summary | {changed, right_a, right_b, lost}
  else [.id, .from, .to, .effect]
  end'
Output
thinkthen: diff: warning: the question digest differs in 10 of 10 paired answers. A different question, threshold, or profile gives a different digest.
["A Day in the Life","yes","no","gained"]
["Glass Onion","yes","no","gained"]
["In My Life","yes","no","gained"]
["Taxman","yes","no","gained"]
["The Long and Winding Road","yes","no","gained"]
{"changed":5,"right_a":5,"right_b":10,"lost":0}
exit 0

The lesson

gained marks a wrong answer that turned right. resolved marks a not-sure answer that settled, right or wrong. lost counts right answers that turned wrong, and none did. At the band 0.2:0.8, Come Together, Here Comes the Sun, and Octopus's Garden kept their answers. diff leaves them out.

diff shows what a change fixed and what it broke.

On GitHub: github.com/botassembly/beatles-bench/tree/main/examples/audit