Skip to content

filter

filter asks one yes-or-no question about every line of its input and prints the lines that answer yes, exactly as they came in. It drops the lines that answer no, and, when you set a low and a high bar, the lines it is not sure about.

Terminal window
thinkthen filter \
'Does the customer request a refund?' messages.txt

Output

Please refund my order.
I was charged twice for one invoice.
messages.txt
Please refund my order.
Where is my parcel?
I was charged twice for one invoice.

The refund request and the double charge answer yes, so filter prints them. The parcel question answers no, so filter drops it.

Set a low bar and a high bar. filter keeps a line at or above the high bar, and drops it below the low bar or in between, where it is not sure.

Terminal window
thinkthen filter --low 0.2 --high 0.8 \
'Does the customer request a refund?' messages.txt

Output

Please refund my order.

Only the refund request is kept. The double charge falls between the bars, so filter is not sure and drops it. Send not sure to a person. Never treat it as a no.

See why

--json prints one line of details for every input line, kept or not. This jq keeps three of them.

Terminal window
thinkthen filter --low 0.2 --high 0.8 --json \
'Does the customer request a refund?' messages.txt |
jq -c '{item, status, kept}'

Output

{"item":"Please refund my order.","status":"answered","kept":true}
{"item":"Where is my parcel?","status":"answered","kept":false}
{"item":"I was charged twice for one invoice.","status":"unsure","kept":false}

status says whether filter answered or was not sure, and kept says whether it printed the line. JSON output lists every field, including probability, the chance of yes.

Move the bar

With one bar, --threshold, filter keeps a line at or above it and drops the rest. Raise the bar to 0.8 and the double charge, at 0.7, is dropped.

Terminal window
thinkthen filter --threshold 0.8 \
'Does the customer request a refund?' messages.txt

Output

Please refund my order.

The default bar is 0.5. Lower it to keep more lines. Raise it to keep only the clear ones.

Use the answer in a script

--check turns the answer into the exit code: 0 when filter keeps at least one line.

Terminal window
if thinkthen filter --check \
'Does the customer request a refund?' \
messages.txt > /dev/null
then
echo 'refund requests are waiting'
else
status=$?
case "$status" in
1|3) ;;
*) exit "$status" ;;
esac
fi

Output

refund requests are waiting
Answer Exit code with --check
at least one line kept 0
none kept, none not sure 1
none kept, at least one not sure 3

Without --check, filter exits 0 whenever it answers. Exit codes lists every code.

Keep the yes rows of a CSV file
messages.csv
id,text
a1,Please refund my order.
a2,Where is my parcel?
a3,I was charged twice for one invoice.
Terminal window
thinkthen filter 'Does the customer request a refund?' \
--from csv --column text --into is_refund < messages.csv

Output

id,text,is_refund
a1,Please refund my order.,TRUE
a3,I was charged twice for one invoice.,TRUE

--column names the column to read, and --into names the new answer column. filter writes only the rows it keeps. With --in-place, a row that fails leaves your file as it was: filter exits 6 and says how many rows failed, so no row is lost. File types covers TSV, Excel and other files.

Options
OptionWhat it does
<QUESTION>Question to ask
[FILE]...Input files; default: standard input
--name <NAME>Optional execution identity
--text <TEXT>Judge this text instead of reading input
--threshold <THRESHOLD>Required probability
--low <LOW>Low probability bar
--high <HIGH>High probability bar
--check [=<CHECK>]Return the verdict as an exit code
--true <TEXT>Meaning of a true answer
--false <TEXT>Meaning of a false answer
--column <COLUMN>Field column; repeat; replaces a saved pointer
--details [=<DETAILS>]Add status, probability, error, detail and source columns
--in-place [=<IN-PLACE>]Replace source with backup
--into <INTO>Answer field for JSON Lines; column for tables
--output <OUTPUT>Destination path
--sheet <SHEET>Worksheet

Shared options

If filter can't answer a line, it prints an error and exits with code 2, 4, 5 or 6. Troubleshooting lists each error and its fix.

In code: Python · TypeScript · Go · Java · Kotlin · C# · R · Ruby · Rust · Swift · Zig · C · C++ · DuckDB · SQLite · PostgreSQL

Next: tag gives a text every label that fits.