Screen papers for a review

At the end, every abstract is marked include, exclude or person, and the papers ThinkThen is not sure about wait for a person to read.

1. Put the abstracts in a file

Write one paper per line as JSON, each with an id and the abstract in text:

papers.jsonl

{"id":"p1","text":"A randomized controlled trial of chemotherapy with or without immunotherapy in 412 adults with advanced lung cancer."}
{"id":"p2","text":"We reviewed the charts of 90 patients treated for colon cancer at one center from 2015 to 2020."}
{"id":"p3","text":"Patients with lung cancer were assigned to a new follow-up schedule or usual care. We report results for the first 60."}

2. Write each criterion as a question

Each criterion of the review becomes one yes-or-no question in a toolbox. Each has a low and a high bar, so an abstract that doesn't settle the question comes back not sure.

screen.json

{
  "field": "/text",
  "questions": {
    "trial": {
      "function": "decide",
      "question": "Is this a randomized controlled trial?",
      "threshold": {"low": 0.3, "high": 0.7}
    },
    "lung": {
      "function": "decide",
      "question": "Does the study enroll lung cancer patients?",
      "threshold": {"low": 0.3, "high": 0.7}
    }
  }
}

3. Ask every criterion about every paper

annotate asks both questions about each abstract. --into screen adds the answers to each line:

thinkthen annotate --into screen \
  screen.json papers.jsonl > screened.jsonl
jq -c '{id, screen}' screened.jsonl

Output

{"id":"p1","screen":{"lung":true,"trial":true}}
{"id":"p2","screen":{"lung":false,"trial":false}}
{"id":"p3","screen":{"lung":true,"trial":"unsure"}}
Copy

The first paper is a randomized trial in lung cancer. The second is neither. The third says patients were assigned to a schedule but not how, so the model is not sure it is randomized.

4. Include, exclude or send to a person

Your code decides the route, in screen.jq. A paper that fails any criterion is excluded. A paper that meets every criterion is included. Anything else, a not-sure answer or a failed one, goes to a person.

screen.jq

[.screen.trial, .screen.lung] as $a
| if any($a[]; . == false) then "\(.id) exclude"
  elif all($a[]; . == true) then "\(.id) include"
  else "\(.id) person"
  end

jq -r -f screen.jq screened.jsonl

Output

p1 include
p2 exclude
p3 person
Copy

The first paper is included, the second is excluded, and the third goes to a person.

Check the result

• Screen a sample by hand, excluded papers included, and count where you and the model disagree on each criterion. A criterion with many disagreements needs a clearer question. Measure answers against your own labels shows the counts.
• Every paper sent to a person needs a person's decision before the review uses it.

Next

Find the same company in two lists asks a question about pairs of lines.