Bash · Recipes

Set bad records aside first

jq reads each raw line and keeps only JSON objects with a string body. It writes every other line to an aside file before decide sees the good records. A record run otherwise stops at the first bad record.

queue.jsonl, a file the script reads

{"id":"A","body":"Arrived a day early. Thank you!"}
not-json
{"id":"B","body":"The zipper broke the first time I used it."}
{"id":"C","body":42}
{"id":"D","body":"The strap snapped on day two."}

The malformed line and the numeric body stay aside. decide judges the three text records.

rule='def ok:
  (try fromjson catch null)
  | type == "object" and (.body | type) == "string";'
jq -Rr "$rule select(ok | not)" queue.jsonl > aside.jsonl
jq -Rr "$rule select(ok)" queue.jsonl > good.jsonl
printf 'aside:\n'
cat aside.jsonl
printf 'judged:\n'
thinkthen decide 'Is this a complaint?' \
  --jsonl --field /body --batch 1 < good.jsonl |
  jq -c '{id:.input.id, complaint:.value}'

Output

aside:
not-json
{"id":"C","body":42}
judged:
{"id":"A","complaint":false}
{"id":"B","complaint":true}
{"id":"D","complaint":true}

exit 0

On GitHub: github.com/botassembly/thinkthen