Agents Lookup

Use cases / Data & analytics

Build and edit data pipelines with an agent

Describe a pipeline in plain language and an agent drafts the transformation steps in BigQuery for a data engineer to review.

DifficultyIntermediate
Setup timeHalf a day
CategoryData & analytics

The use case

Data engineers spend time writing similar ingest-clean-join pipelines. BigQuery's data engineering agent takes a description such as 'load these files, clean dates, join to customers and build a daily summary' and drafts the pipeline in BigQuery pipelines and Dataform, and can modify existing ones. The engineer sees the generated SQL and steps, edits them, and tests on sample data. The engineer still owns schema decisions, data quality checks and scheduling, and reviews changes to pipelines that feed production reports. Starting with a new, low-stakes pipeline is a sensible way to learn how much editing the drafts need.

Example workflow

  1. Describe sources, transforms and target tables
  2. Agent drafts pipeline steps and SQL
  3. Review and edit each step
  4. Test on sample data and add quality checks
  5. Schedule after review

Tools

Risks and guardrails

Sources

Written by Agents Lookup. Credits and official pages for the tools this use case names: