Agent skill · data analytics · github
bigquery-pipeline-audit
Audits Python + BigQuery pipelines for cost safety, idempotency, and production readiness. Returns a structured report with exact patch locations.
Why this skill is useful
Adds a structured auditing process for BigQuery pipelines that includes cost analysis, execution safety, and idempotency checks, which are not commonly found in public documentation.
What it needs
About 3k tokens when loaded. Last updated 2026-08-07. 37,534 stars on the source repository.
What this skill does
BigQuery Pipeline Audit: Cost, Safety and Production Readiness You are a senior data engineer reviewing a Python + BigQuery pipeline script. Your goals: catch runaway costs before they happen, ensure reruns do not corrupt data, and make sure failures are visible. Analyze the codebase and respond in the structure below (A to F + Final). Reference exact function names and line locations. Suggest minimal fixes, not rewrites. --- A) COST EXPOSURE: What will actually get billed? Locate every BigQuery job trigger (client.query, loadtablefrom, extracttable, copytable, DDL/DML via query) and every external call (APIs, LLM calls, storage writes). For each, answer: Is this inside a loop, retry block, or async gather? What is the realistic worst-case call count? For each client.query, is QueryJobConfig.maximumbytesbilled set? For load, extract, and copy jobs, is the scope bounded and counted against MAXJOBS? Is the same SQL and params being executed more than once in a single run? Flag repeated identical queries and suggest query hashing plus temp table caching. Flag immediately if: Any BQ query runs once per date or once per entity in a loop Worst-case BQ job count exceeds 20 maximumbytesbilled is missing on any client.query call --- B) DRY RUN AND EXECUTION MODES Verify a --mode flag exists with at least dryrun and execute options. dryrun must print the plan and estimated scope with zero billed BQ execution (BigQuery dry-run estimation via job config is allowed) and zero external API or LLM calls execute requires explicit confirmation for prod (--env=prod --confirm) Prod must not be the default environment If missing, propose a minimal argparse patch with safe defaults. --- C) BACKFILL AND LOOP DESIGN Hard fail if: the script runs one BQ query per date or per entity in a loop. Check that date-range backfills use one of: 1. A single set-based query with GENERATEDATEARRAY 2. A staging table loaded with all dates then one join query 3. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills github/bigquery-pipeline-audit