Break a problem into decisions

At the end, a fuzzy job is a few small questions with fixed answers, each tried on a real message and saved in one toolbox.

1. Write down the decisions a person makes

"Handle this support message" has no answer you can check. A person doing the job makes three decisions:

• Does the customer request a refund? A yes or no, from decide.
• What is the message about? One of billing, shipping or other, from choose.
• How urgent is it? A number from routine to urgent, from score.

Each has a small answer, so you can check it against what you would have said.

2. Ask the yes-or-no decision

Try each question on one real message with --text. Set a low and a high bar, so a message the model can't settle comes back not sure:

thinkthen decide --low 0.2 --high 0.8 \
  --text 'I was charged twice for one invoice.' \
  'Does the customer request a refund?'

Output

unsure
Copy

Not sure: a double charge might be a refund request, or might not. That is the message a person should read.

3. Ask the choice

thinkthen choose \
  --choice billing --choice shipping --choice other \
  --text 'I was charged twice for one invoice.' \
  'What is the message about?'

Output

billing
Copy

Billing: a double charge is about an invoice.

4. Ask the scale

thinkthen score --level routine --level urgent \
  --text 'I was charged twice for one invoice.' \
  'How urgent is the request?'

Output

0.45
Copy

The answer is a number from 0 for routine to 1 for urgent.

5. Put the questions in one toolbox

Save the three questions in a toolbox, with the same bars:

inbox.json

{
  "questions": {
    "refund": {
      "function": "decide",
      "question": "Does the customer request a refund?",
      "threshold": {"low": 0.2, "high": 0.8}
    },
    "topic": {
      "function": "choose",
      "question": "What is the message about?",
      "choices": ["billing", "shipping", "other"]
    },
    "urgency": {
      "function": "score",
      "question": "How urgent is the request?",
      "levels": ["routine", "urgent"]
    }
  }
}

annotate asks all three at once and prints each answer by its name:

thinkthen annotate \
  --text 'I was charged twice for one invoice.' \
  inbox.json

Output

refund=unsure	topic=billing	urgency=0.45
Copy

The same three answers come back in one call: not sure, billing and the urgency.

Check the result

• Each answer is one you could have given yourself. If you can't say whether an answer is right, the question is still too big; split it again.
• Name everything the model needs in the question itself: the allowed values, and any date or amount rule.
• Code routes, counts and acts on the answers. The model only answers. A person reads the not-sure cases.

Next

Triage a support inbox asks these questions about a whole inbox and sends each message to its queue.