---
title: Toolboxes
description: Save ThinkThen questions in a toolbox file, run one by name with thinkthen run, chain a question with your own shell commands, and look up the file format.
goal: The reader saves a question in a toolbox, runs it by name, chains it with a shell step safely, and can write the file from its format.
reader: A command-line user who asks the same questions again and wants them in a file under version control
kind: how-to
verified: 2026-11-03
---

# Toolboxes

A toolbox is a JSON file of saved questions. `thinkthen run` asks a saved question by its name, and a toolbox can also pass the answers on to your own commands.

This toolbox saves two questions, `refunds` and `topic`:

```json title="toolbox.json"
{
  "entries": {
    "refunds": {
      "question": {
        "function": "filter",
        "question": "Does the customer request a refund?",
        "field": "/text"
      }
    },
    "topic": {
      "question": {
        "function": "choose",
        "question": "What is the message about?",
        "choices": ["billing", "shipping", "other"],
        "field": "/text"
      }
    }
  }
}
```

```json title="messages.jsonl"
{"id":"a1","text":"Please refund my order."}
{"id":"a2","text":"Where is my parcel?"}
{"id":"a3","text":"I was charged twice for one invoice."}
```

Name the toolbox, then the entry, then the file:

```sh
thinkthen run toolbox.json refunds messages.jsonl
```

Output

```text
{"id":"a1","text":"Please refund my order."}
{"id":"a3","text":"I was charged twice for one invoice."}
```


`refunds` is a filter, so it prints the two messages that ask for a refund. The entry runs as if you had typed its function and options. `run` reads JSON Lines unless you say otherwise, such as `--from lines` or `--column text` for a CSV file.

## Keep each question in its own file

A folder named `thinkthen/` works as a toolbox, with one entry per file, named after the file. `thinkthen run NAME` looks for `thinkthen.json` or a `thinkthen/` folder in the current folder.

```json title="thinkthen/refunds.json"
{
  "question": {
    "function": "filter",
    "question": "Does the customer request a refund?",
    "field": "/text"
  }
}
```

```json title="thinkthen/topic.json"
{
  "question": {
    "function": "choose",
    "question": "What is the message about?",
    "choices": ["billing", "shipping", "other"],
    "field": "/text"
  }
}
```

```json title="messages.jsonl"
{"id":"a1","text":"Please refund my order."}
{"id":"a2","text":"Where is my parcel?"}
{"id":"a3","text":"I was charged twice for one invoice."}
```

```sh
ls thinkthen
thinkthen run refunds messages.jsonl
```

Output

```text
refunds.json
topic.json
{"id":"a1","text":"Please refund my order."}
{"id":"a3","text":"I was charged twice for one invoice."}
```


`ls` shows the two entries, and `run refunds` asks the one in `refunds.json`. A file that holds one entry runs without a name.

## Ask every saved question at once

[annotate](/functions/annotate/) asks every question in a toolbox about each message and gives back one answer per question. Its toolbox holds a `questions` map instead of `entries`; the format is below.

## Pass the answers to your own command

An entry can list steps. Each step reads the answers of the step before it, one JSON object per line. This entry keeps the refund requests, then `jq` prints the `id` and chance of yes of each:

```json title="refund-ids.json"
{
  "run": [
    {
      "question": {
        "function": "filter",
        "question": "Does the customer request a refund?",
        "field": "/text"
      }
    },
    {"shell": ["jq", "-c", "select(.answer == true) | {id, probability}"]}
  ]
}
```

```json title="messages.jsonl"
{"id":"a1","text":"Please refund my order."}
{"id":"a2","text":"Where is my parcel?"}
{"id":"a3","text":"I was charged twice for one invoice."}
```

```sh
thinkthen run --allow-shell refund-ids.json messages.jsonl
```

Output

```text
{"id":"a1","probability":0.99}
{"id":"a3","probability":0.7}
```


A shell step runs only with `--allow-shell`. It names a program and its words, such as `["jq", "-c", "..."]`, and ThinkThen runs it as written, with no shell in between. To use shell syntax, name the shell: `["sh", "-c", "..."]`. A shell step that fails stops the run with its own exit code.

**Read a toolbox before you add `--allow-shell`.** A shell step can run any program on your computer, so whoever wrote the toolbox decides what runs. A bare `thinkthen run NAME` uses the toolbox of the folder you are in, so in a folder you didn't write, such as a cloned project, read its `thinkthen.json` or `thinkthen/` folder first, or name the toolbox file you have read.

ThinkThen removes `THINKTHEN_API_KEY` and `OPENAI_API_KEY` from shell steps. A step gets a key only when it lists it, as in `"env": ["THINKTHEN_API_KEY"]`, and the command then prints a notice that names it. Every other variable in your shell still reaches the step.

## The toolbox format

A file for `run` holds an `entries` map from name to entry, or one entry by itself. A `thinkthen/` folder holds one entry in each `NAME.json`.

| An entry holds | What it is |
| --- | --- |
| `question` | One saved question, as in the table below. |
| `run` | Steps, in order, instead of `question`. Nothing else sits beside `run`. |
| `name` | An optional name for the answers, as `--name` gives. |

A step is `{"question": ...}`, with an optional `name`, or `{"shell": ["PROGRAM", "WORD", ...]}`, with an optional `env` list of key names to pass.

A file for annotate holds a `questions` map from name to question, and an optional `field` for every question that names none. rank and annotate can't sit inside it.

Every question names its `function`, and every question takes an optional `field`, such as `"/text"`:

| Function | Fields |
| --- | --- |
| decide, filter | `question`; optional `threshold`, `true`, `false` |
| choose | `question`, `choices`; optional `threshold` |
| tag | `question`, `labels`; optional `threshold` |
| score | `question`, `levels`, lowest first |
| rank | `question`, `levels`, lowest first; optional `top` |
| find | `question`; optional `threshold` |
| recognize | `kinds`; optional `threshold` |
| relate | `rules`; optional `threshold` |
| annotate | `questions`, a map from name to question |

`threshold` is one bar, a number above 0 and at most 1, such as `0.8`; the default is `0.5`. decide, filter and choose also take a low and a high bar, `{"low": 0.2, "high": 0.8}`. A relate rule holds `relation`, `source` and `target`, and optional `either` and `reads`. A field the format doesn't list stops the command with the closest name it knows.

## Next

[Agents](/command-line/agents/) shows how an AI agent calls ThinkThen.