Fail a CI job on a yes-or-no question

At the end, your CI job passes when a pull request description says how the change was tested, fails when it doesn't, and asks a reviewer when ThinkThen is not sure.

1. Save the description in a file

Your CI job writes the pull request description to pr.md. This one doesn't say how the change was tested:

pr.md

Fix the rounding of invoice totals.

Totals now round half up to two decimals, as the tax rules require.

2. Ask the question with --check

--check makes the answer the exit code: 0 for yes, 1 for no and 3 for not sure. The low and high bars send an unclear description to not sure.

thinkthen decide --check --low 0.3 --high 0.7 \
  'Does the description say how the change was tested?' \
  pr.md
status=$?
echo "exit code $status"
case "$status" in
  0|1|3) ;;
  *) exit "$status" ;;
esac

Output

false
exit code 1
Copy

No: the description says what changed, not how it was tested, so the exit code is 1.

3. Turn each exit code into the job's result

check-pr.sh reads the exit code and prints a note for the author or the reviewer. Any other code means the check itself didn't run, so the job fails with 2 rather than passing.

check-pr.sh

#!/bin/sh
# Pass on yes. Fail on no, not sure or an error.
thinkthen decide --check --low 0.3 --high 0.7 \
  'Does the description say how the change was tested?' \
  "$1" > /dev/null
case $? in
  0) echo 'passed: the description has a test plan' ;;
  1) echo 'failed: say how you tested the change'
     exit 1 ;;
  3) echo 'not sure: a reviewer must read the test plan'
     exit 1 ;;
  *) echo 'error: the check did not run'
     exit 2 ;;
esac

sh check-pr.sh pr.md

Output

failed: say how you tested the change
Copy

The job fails, and the note tells the author what to add.

4. Add the script to your CI job

Run sh check-pr.sh pr.md as a step after the step that writes pr.md. The job needs thinkthen installed and your key in a CI secret. Installation and Configuration cover both.

Check the result

Run the script on a description that says how the change was tested:

pr-tested.md

Fix the rounding of invoice totals.

Totals now round half up to two decimals, as the tax rules require.

Tested: I added unit tests for totals that end in 5, and ran
the full test suite, which passes.

sh check-pr.sh pr-tested.md

Output

passed: the description has a test plan
Copy

Yes: the description names the tests, so the job passes.

A writer who knows the question can word a description to pass it. Treat the check as a reminder, and keep a reviewer on the not-sure branch. Exit codes lists every code the script can see.

Next

Measure answers against your own labels checks your bars against answers you wrote yourself.