Agent skill · data analytics · alirezarezvani
senior-data-engineer
Data pipelines: Airflow DAGs, Spark, dbt, Kafka, ETL/ELT, data quality checks. Load whenever a DAG or batch job times out or fails, data arrives wrong, or the user designs an ingestion pipeline.
Why this skill is useful
Adds 3 executable scripts for data pipeline orchestration, data quality validation, and ETL performance optimization that are not commonly found in public documentation.
What it needs
Requires python installed locally. About 3k tokens when loaded. Last updated 2026-08-05. 23,939 stars on the source repository.
What this skill does
Senior Data Engineer Production-grade data engineering skill for building scalable, reliable data systems. Table of Contents 1. Trigger Phrases 2. Quick Start 3. Workflows 4. Architecture Decision Framework 5. Tech Stack 6. Reference Documentation 7. Troubleshooting --- Trigger Phrases Activate this skill when you see: Pipeline Design: "Design a data pipeline for..." "Build an ETL/ELT process..." "How should I ingest data from..." "Set up data extraction from..." Architecture: "Should I use batch or streaming?" "Lambda vs Kappa architecture" "How to handle late-arriving data" "Design a data lakehouse" Data Modeling: "Create a dimensional model..." "Star schema vs snowflake" "Implement slowly changing dimensions" "Design a data vault" Data Quality: "Add data validation to..." "Set up data quality checks" "Monitor data freshness" "Implement data contracts" Performance: "Optimize this Spark job" "Query is running slow" "Reduce pipeline execution time" "Tune Airflow DAG" --- Quick Start Core Tools --- Workflows → See references/workflows.md for details Architecture Decision Framework Use this framework to choose the right approach for your data pipeline. Batch vs Streaming Criteria Batch Streaming ---------- ------- ----------- Latency requirement Hours to days Seconds to minutes Data volume Large historical datasets Continuous event streams Processing complexity Complex transformations, ML Simple aggregations, filtering Cost sensitivity More cost-effective Higher infrastructure cost Error handling Easier to reprocess Requires careful design Decision Tree: Lambda vs Kappa Architecture Aspect Lambda Kappa -------- -------- ------- Complexity Two codebases (batch + stream) Single codebase Maintenance Higher (sync batch/stream logic) Lower Reprocessing Native batch layer Replay from source Use case ML training + real-time serving Pure event-driven When to choose Lambda: Need to train ML models on historical data Complex batch transformations not feasible in streaming Exist …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills alirezarezvani/senior-data-engineer--7a6d64