Skip to content
ThinkThen

Measure answers against your own labels

At the end, you know how often ThinkThen agrees with the answers you wrote yourself, and what a higher bar changes.

1. Label some messages yourself

Section titled “1. Label some messages yourself”

Write one message per line, with the answer you expect in label. Use real messages. The more you label, the steadier the counts; this example uses six.

labeled.jsonl
{"id":"a1","text":"Please refund my order.","label":true}
{"id":"a2","text":"Where is my parcel?","label":false}
{"id":"a3","text":"I was charged twice for one invoice.","label":true}
{"id":"a4","text":"Can I have my money back for the late delivery?","label":true}
{"id":"a5","text":"Thanks for the quick help yesterday!","label":false}
{"id":"a6","text":"Cancel my subscription and return the last payment.","label":true}

2. Answer each message beside its label

Section titled “2. Answer each message beside its label”

--field /text sends only the text, so the model never sees your label. --into answer adds the answer to each line:

Terminal window
thinkthen decide --field /text --into answer \
  'Does the customer request a refund?' \
  labeled.jsonl > answered.jsonl
jq -c '{id, label, answer}' answered.jsonl

Output

{"id":"a1","label":true,"answer":true}
{"id":"a2","label":false,"answer":false}
{"id":"a3","label":true,"answer":true}
{"id":"a4","label":true,"answer":true}
{"id":"a5","label":false,"answer":false}
{"id":"a6","label":true,"answer":true}

Each line now holds your label and the model's answer side by side.

3. Count the agreements and the mistakes

Section titled “3. Count the agreements and the mistakes”

A wrong yes is a yes where you wrote no. A missed yes is a no where you wrote yes.

Terminal window
jq -s -c '{
  agree: map(select(.answer == .label)) | length,
  wrong_yes: map(select(.answer == true
    and .label == false)) | length,
  missed_yes: map(select(.answer == false
    and .label == true)) | length
}' answered.jsonl

Output

{"agree":6,"wrong_yes":0,"missed_yes":0}

The model agrees with you on all six messages, with no wrong yeses and no missed yeses.

4. Raise the bar and count again

Section titled “4. Raise the bar and count again”

--threshold 0.8 says yes only when the chance of yes is at least 0.8. The default bar is 0.5.

Terminal window
thinkthen decide --field /text --threshold 0.8 \
  --into strict 'Does the customer request a refund?' \
  answered.jsonl > strict.jsonl
jq -s -c '{
  wrong_yes: map(select(.strict == true
    and .label == false)) | length,
  missed_yes: map(select(.strict == false
    and .label == true)) | length
}' strict.jsonl

Output

{"wrong_yes":0,"missed_yes":1}

One missed yes: the model is less sure about the double charge than the new bar asks, so it is now a no. A higher bar gives fewer wrong yeses and more missed yeses.

Check the result

Section titled “Check the result”
  • Pick the bar from what a mistake costs. A wrong yes in a refund queue costs a person a minute. A missed yes costs a customer their refund, so take a lower bar.
  • When neither mistake is cheap, set a low and a high bar, and send the not-sure messages to a person, as Triage a support inbox does.
  • Keep labeled.jsonl. Run these steps again when you change the question or the model, and compare the counts.

Next

Section titled “Next”

Screen papers for a review puts two questions with bars to work on research abstracts.