Agent skill · data analytics · borghei
senior-data-engineer
Data engineering for batch and streaming pipelines with Airflow, dbt, Spark, and Kafka. Use when designing data architectures, building pipelines, adding data-quality checks, optimizing ETL/ELT, or troubleshooting pipeline failures.
Why this skill is useful
Adds 3 executable scripts for pipeline orchestration, data quality validation, and ETL performance optimization that extend the AI's capabilities.
What it needs
Requires python installed locally. About 3k tokens when loaded. Last updated 2026-07-21. 462 stars on the source repository.
What this skill does
Senior Data Engineer Generate pipeline configurations (Airflow, Prefect, Dagster), validate data quality with profiling and anomaly detection, and optimize SQL/Spark performance with actionable recommendations. Core Capabilities Pipeline generation — Airflow/Prefect/Dagster DAG code for batch and incremental loads, with DAG validation. Data quality — schema validation, profiling, anomaly detection, data contracts, and Great Expectations suite generation. ETL/ELT optimization — SQL and Spark analysis, partition strategy, and query cost estimation per warehouse. Architecture decisions — batch vs streaming and warehouse vs lakehouse trade-off frameworks. Reliability patterns — incremental watermarks, dead letter queues, freshness checks, and schema-drift detection. When to Use Designing a data architecture or choosing batch vs streaming / warehouse vs lakehouse. Building or generating Airflow/Spark/dbt pipelines. Adding data-quality checks or data contracts. Optimizing slow ETL/ELT queries or troubleshooting pipeline failures. Clarify First Before generating pipelines, confirm these inputs. If any is unknown or vague, ASK — do not assume: [ ] Orchestrator — Airflow / Prefect / Dagster (--type; changes the generated DAG code) [ ] Source, destination & load mode — systems involved and batch vs incremental (--source/--destination/--mode; shapes the pipeline) [ ] Data-quality expectations — the schema and contracts to enforce (drives the Great Expectations suite generation) Stop rule: ask only the 2-3 that most change the output. If the user says "just draft it," proceed and list your assumptions at the top of the artifact. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills borghei/senior-data-engineer