Agent skill · data analytics · onewave-ai
data-pipeline-builder
Designs and builds ETL/ELT data pipelines. Takes data sources, destination, transformation requirements. Generates pipeline code (Python/SQL), scheduling config, error handling, monitoring setup, and data quality checks. Outputs data-pipeline-spec.md + implementation files.
Why this skill is useful
Generates complete ETL/ELT pipeline code, configuration, and monitoring setups that significantly extend the AI's capabilities in data engineering.
What it needs
Requires airflow, python installed locally. About 2k tokens when loaded. Last updated 2026-07-15. 242 stars on the source repository.
What this skill does
Data Pipeline Builder Design and implement production-grade ETL/ELT data pipelines: take data sources, a destination, and transformation requirements, then produce a complete pipeline specification plus all implementation files needed to run it. Contents references/project-structure.md -- output file layout, architecture pattern selection, component selection. references/python-patterns.md -- Python code standards and base extractor/transformer/loader/retry patterns. references/quality-checks.md -- composable data quality check framework and built-in checks. references/orchestration-config.md -- Airflow DAG, pipeline config YAML, and monitoring/alerting patterns. references/spec-template.md -- the data-pipeline-spec.md output template. Workflow 1. Gather requirements. If the user gave clear requirements, proceed to design. Otherwise ask targeted questions: data sources (databases, APIs, files, streams); destination (warehouse, lake, database); transformations (joins, aggregations, filters, business rules); freshness requirement (real-time, hourly, daily); technology preferences (Airflow, dbt, Spark, cloud provider); data quality and compliance requirements. 2. Analyze and design. Catalog each source (connection type, auth, schema, volume, CDC availability, rate limits). Define the destination (platform, schema design, partitioning, clustering, access patterns). Map transformations (field mappings, business logic, type conversions, joins, aggregations, deduplication, SCD handling, derived fields). Establish non-functional requirements (freshness SLA, processing window, failure tolerance, retention, compliance). Select an architecture pattern and components per references/project-structure.md. 3. Present the design before generating code. Confirm architecture, sources, destination, schedule, key transformations, and quality gates with the user, then proceed on approval. 4. Generate implementation. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills onewave-ai/data-pipeline-builder