ThinkThen

Beatles Bench pages
Start
The ten functions
Scripts
Bindings
What Jev knows
Tune your bar
Agents
Run it
Beatles Bench · Run it

Bring your own backend.

The talk's slide: Bring your own backend.
No run made the slide's console. It follows the check specification at thinkthen d0aa1b0b, except the model line, which shows the model a server names. Open the slide at full size.

Any server with the same interface as Jev can answer. Name it with --url.

thinkthen check sends four fixed requests to check that a server works. With --plan, it prints its plan and sends nothing.

The numbers come from the check specification.

Run it

The check names the address and the model it would ask.
thinkthen check \
  --plan \
  --url https://your-server/v1 |
head -n 4
Output
url https://your-server/v1/systemone
provider systemone
model asked unspecified
model sent jev-1.13.0
exit 0

The lesson

A bar tuned on one model does not carry to another. Run audit again on your labeled records before you trust a new backend.

A new model needs its own bar.

On GitHub: github.com/botassembly/beatles-bench