Agent skill · data analytics · erichowens
data-pipeline-engineer
Expert data engineer for ETL/ELT pipelines, streaming, data warehousing. Activate on: data pipeline, ETL, ELT, data warehouse, Spark, Kafka, Airflow, dbt, data modeling, star schema, streaming
Why this skill is useful
Provides specific patterns and scripts for ETL/ELT pipelines that the AI wouldn't reliably generate on its own.
What it needs
Requires airflow, dbt, kafka, spark installed locally. About 3k tokens when loaded. Last updated 2026-07-14. 180 stars on the source repository.
What this skill does
Data Pipeline Engineer Expert data engineer specializing in ETL/ELT pipelines, streaming architectures, data warehousing, and modern data stack implementation. Quick Start 1. Identify sources - data formats, volumes, freshness requirements 2. Choose architecture - Medallion (Bronze/Silver/Gold), Lambda, or Kappa 3. Design layers - staging → intermediate → marts (dbt pattern) 4. Add quality gates - Great Expectations or dbt tests at each layer 5. Orchestrate - Airflow DAGs with sensors and retries 6. Monitor - lineage, freshness, anomaly detection Core Capabilities Capability Technologies Key Patterns ------------ -------------- -------------- Batch Processing Spark, dbt, Databricks Incremental, partitioning, Delta/Iceberg Stream Processing Kafka, Flink, Spark Streaming Watermarks, exactly-once, windowing Orchestration Airflow, Dagster, Prefect DAG design, sensors, task groups Data Modeling dbt, SQL Kimball, Data Vault, SCD Data Quality Great Expectations, dbt tests Validation suites, freshness Architecture Patterns Medallion Architecture (Recommended) Lambda vs Kappa Lambda: Batch + Stream layers → merged serving layer (complex but complete) Kappa: Stream-only with replay → simpler but requires robust streaming Reference Examples Full implementation examples in ./references/: File Description ------ ------------- dbt-project-structure.md Complete dbt layout with staging, intermediate, marts airflow-dag.py Production DAG with sensors, task groups, quality checks spark-streaming.py Kafka-to-Delta processor with windowing great-expectations-suite.json Comprehensive data quality expectation suite Anti-Patterns (10 Critical Mistakes) 1. Full Table Refreshes Symptom: Truncate and rebuild entire tables every run Fix: Use incremental models with isincremental(), partition by date 2. Tight Coupling to Source Schemas Symptom: Pipeline breaks when upstream adds/removes columns Fix: Explicit source contracts, select only needed columns in staging 3. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills erichowens/data-pipeline-engineer