Beatles Bench · Start
Jev set the standard.
Jev is the model behind ThinkThen. TypeSafe makes it. It reads a text and a question and returns a probability for every answer. It writes no text, and you train nothing.
Roger Bannister's mile set a standard for runners. Jev answers in about 60 milliseconds. That is the median time at Jev's server, measured on 2026-09-26. On Beatles Bench, a thousand answers cost 0.016 dollars.
The numbers come from the bench's record for this slide.
Run it
At a bar of 0.8, Jev picks Ringo.
is_a_song="The text is the title of a song by the Beatles."
question="$is_a_song Who sings the lead vocal on it?"
singers=(
John
Paul
George
Ringo
"John and Paul duet"
)
printf '%s' "Octopus's Garden" |
thinkthen choose "$question" "${singers[@]}" \
--threshold 0.8 \
--replay recordingOutput
"Ringo"
exit 0: an option came back
Change the bar
A bar of 0.9 asks for more. The answer is not sure.
is_a_song="The text is the title of a song by the Beatles."
question="$is_a_song Who sings the lead vocal on it?"
singers=(
John
Paul
George
Ringo
"John and Paul duet"
)
printf '%s' "Octopus's Garden" |
thinkthen choose "$question" "${singers[@]}" \
--threshold 0.9 \
--replay recordingOutput
null
exit 3: not sure
The lesson
Ringo is right. Jev's probability for Ringo clears 0.8 and falls short of 0.9. A bar of 0.9 asks for more than Jev gives.
Jev answers with a probability, and your bar decides what counts.
On GitHub: github.com/botassembly/beatles-bench/tree/main/examples/choose