Mutation-guided tests for a specific class of bug
Seed realistic bugs of one type into code, then have a model write tests that catch them, so tests target real risks.
The use case
Coverage numbers say little about whether tests catch the bugs a team actually worries about. Meta described ACH, which starts from a plain description of a concern, such as privacy leaks, generates realistic faulty versions of the code, and then writes tests that fail on those faults but pass on the real code. Engineers review the proposed tests before they land. The approach suits teams with a clear risk area and a working CI. Engineers still judge whether the injected faults are realistic and whether the tests are readable enough to maintain.
Example workflow
- Describe the type of bug you care about
- Generate plausible faulty versions of target code
- Have the model write tests that catch those faults
- Keep tests that pass on real code and fail on mutants
- Engineer reviews and merges the tests
Tools
- LLM API
- Mutation testing
- CI
Risks and guardrails
- Unrealistic mutants produce tests of little value
- Generated tests may be brittle to refactoring
- Keep seeded faults out of shipped code
Sources
Written by Agents Lookup. Credits and official pages for the tools this use case names:
- Engineering at Meta: ACH mutation-guided test generation: engineering.fb.com/2025/02/05/security/revolutionizing-software-testing-llm-powered-bug-catchers-meta-ach