Beatles Bench · The ten functions
tag applies labels to data.
tag names every label that fits. Each label gets its own probability. A label counts when it clears the bar, and the default bar is 0.5.
Run it
At 0.5, Octopus's Garden gets three labels.
is_a_song="The text is the title of a song by the Beatles."
question="$is_a_song Which of these describe it?"
labels=(
"love song"
sad
psychedelic
"about a place"
"about the sea"
)
cat <<'EOF' |
Yesterday
Penny Lane
Octopus's Garden
EOF
thinkthen tag "$question" "${labels[@]}" \
--batch 1 \
--lines \
--threshold 0.5 \
--replay recording |
jq .Output
{
"input": "Yesterday",
"value": [
"love song",
"sad"
]
}
{
"input": "Penny Lane",
"value": [
"psychedelic",
"about a place"
]
}
{
"input": "Octopus's Garden",
"value": [
"psychedelic",
"about a place",
"about the sea"
]
}exit 0
Change the bar
At 0.7, two songs lose psychedelic.
is_a_song="The text is the title of a song by the Beatles."
question="$is_a_song Which of these describe it?"
labels=(
"love song"
sad
psychedelic
"about a place"
"about the sea"
)
cat <<'EOF' |
Yesterday
Penny Lane
Octopus's Garden
EOF
thinkthen tag "$question" "${labels[@]}" \
--batch 1 \
--lines \
--threshold 0.7 \
--replay recording |
jq .Output
{
"input": "Yesterday",
"value": [
"love song",
"sad"
]
}
{
"input": "Penny Lane",
"value": [
"about a place"
]
}
{
"input": "Octopus's Garden",
"value": [
"about a place",
"about the sea"
]
}exit 0
The lesson
Penny Lane and Octopus's Garden are psychedelic at 0.5 and lose the label at 0.7. The bench has no answer key for tag.
A higher bar keeps fewer labels.
On GitHub: github.com/botassembly/beatles-bench/tree/main/examples/tag