Agent skill · data analytics · aradotso
data-engineering-medallion-pipeline
End-to-end data engineering pipeline using MinIO, Airbyte, PostgreSQL, DBT, and Airflow with medallion architecture (Bronze/Silver/Gold layers)
Why this skill is useful
Adds executable scripts and a complete end-to-end data pipeline setup that AI cannot generate from knowledge alone.
What it needs
Requires docker, docker-compose installed locally. About 8k tokens when loaded. Last updated 2026-08-05. 5 stars on the source repository.
What this skill does
Data Engineering Medallion Pipeline Skill Skill by ara.so — Data Skills collection. This skill enables AI agents to work with a complete data engineering pipeline implementing the Medallion Architecture (Bronze → Silver → Gold) using modern open-source tools: MinIO (S3-compatible storage), Airbyte (data ingestion), PostgreSQL (data warehouse), DBT (transformations), Apache Airflow (orchestration), and Grafana (monitoring). What This Project Does The data-engineering-medallion project provides a complete end-to-end data pipeline that: 1. Ingests raw data from MinIO object storage into PostgreSQL using Airbyte 2. Transforms data through three layers (Bronze/Silver/Gold) using DBT 3. Orchestrates the entire pipeline with Apache Airflow DAGs 4. Validates data quality with automated DBT tests 5. Monitors infrastructure health with Prometheus and Grafana 6. Visualizes business metrics in Power BI dashboards The architecture follows ELT (Extract-Load-Transform) pattern with clear separation of concerns: Bronze: Raw immutable data from sources (JSONB format) Silver: Cleaned, validated, and typed data Gold: Business-ready aggregated metrics and KPIs Installation & Setup Prerequisites Clone and Initialize Service URLs After startup, access these interfaces: Airflow: http://localhost:8080 (admin/admin) MinIO: http://localhost:9001 (minioadmin/[from .env]) Airbyte: http://localhost:8000 (create account on first visit) Grafana: http://localhost:3000 (admin/admin) Prometheus: http://localhost:9090 DBT Docs: http://localhost:8085 (after make dbt-docs) Key Commands (Makefile) Project Structure DBT Model Development Bronze Layer (Raw Extraction) Bronze models extract JSONB data from Airbyte into typed columns: Silver Layer (Cleaned & Validated) Silver models clean, cast types, deduplicate, and add calculated fields: Gold Layer (Business Metrics) Gold models aggregate data for business consumption: DBT Testing & Data Quality Schema Tests (schema.yml) Custom Tests Running Tests Airflo …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills aradotso/data-engineering-medallion-pipeline