# `diff` shows what changed.

Ask the same question twice, and `diff` prints only the answers that changed. A summary comes last. With an answer key, each change says whether it fixed a mistake, made one, or settled a not-sure answer.

Here the page reads two saved runs for the 10 songs on the [audit page](/learn/beatles-bench/audit/). The first run asked from the title alone. The second run gave Jev each song's catalog entry too. The slide's last column shows the answers with context. The warning is right. The two runs asked different questions.

## Run it

*At the band 0.2:0.8, 7 of the 10 answers changed, and all 10 end right.*

```
thinkthen diff memory.jsonl context.jsonl \
  --threshold 0.2:0.8 \
  --key key.jsonl |
jq -c 'if .summary
  then .summary | {changed, right_a, right_b, lost}
  else [.id, .from, .to, .effect]
  end'
```

*Output*

```
thinkthen: diff: warning: the question digest differs in 10 of 10 paired answers. A different question, threshold, or profile gives a different digest.
["A Day in the Life","yes","no","gained"]
["Blackbird","unsure","no","resolved"]
["Glass Onion","unsure","no","resolved"]
["In My Life","unsure","no","resolved"]
["Something","unsure","yes","resolved"]
["Taxman","unsure","no","resolved"]
["The Long and Winding Road","yes","no","gained"]
{"changed":7,"right_a":3,"right_b":10,"lost":0}
```

*exit 0*

## Change the bar

*At a bar of 0.5, 5 changed. Each was a wrong yes that turned right.*

```
thinkthen diff memory.jsonl context.jsonl \
  --threshold 0.5 \
  --key key.jsonl |
jq -c 'if .summary
  then .summary | {changed, right_a, right_b, lost}
  else [.id, .from, .to, .effect]
  end'
```

*Output*

```
thinkthen: diff: warning: the question digest differs in 10 of 10 paired answers. A different question, threshold, or profile gives a different digest.
["A Day in the Life","yes","no","gained"]
["Glass Onion","yes","no","gained"]
["In My Life","yes","no","gained"]
["Taxman","yes","no","gained"]
["The Long and Winding Road","yes","no","gained"]
{"changed":5,"right_a":5,"right_b":10,"lost":0}
```

*exit 0*

## The lesson

`gained` marks a wrong answer that turned right. `resolved` marks a not-sure answer that settled, right or wrong. `lost` counts right answers that turned wrong, and none did. At the band 0.2:0.8, Come Together, Here Comes the Sun, and Octopus's Garden kept their answers. `diff` leaves them out.

diff shows what a change fixed and what it broke.

On GitHub: [github.com/botassembly/beatles-bench/tree/main/examples/audit](https://github.com/botassembly/beatles-bench/tree/main/examples/audit)