Scripted multi-step AI workflows for repeatable code tasks
Define a fixed sequence of AI and shell steps in a file so tasks like grading tests or generating fixtures run the same way each time.
The use case
Free-form agent chats are hard to repeat. Shopify open-sourced Roast, a framework for writing AI workflows as files: each step is either a model prompt or a plain command, run in order with outputs passed along. Shopify used it for jobs such as grading the quality of test files across a large codebase. Because the steps are fixed, results are easier to compare between runs and to review. Engineers still check outputs on a sample before acting on them, and keep deterministic steps like running tests as plain commands rather than asking the model.
Example workflow
- Write the task as ordered steps in a workflow file
- Mix prompt steps with shell or test commands
- Run the workflow on a few targets
- Check results on a sample
- Run across the codebase and review findings
Tools
- Roast
- Ruby
Risks and guardrails
- Model steps still vary between runs; sample results
- Workflows with write access need review gates
- Keep secrets out of prompts and logs
Sources
Written by Agents Lookup. Credits and official pages for the tools this use case names:
- Shopify on GitHub: Roast: github.com/Shopify/roast