Agent skill · data analytics · aradotso
retail-etl-pipeline-medallion
End-to-end retail ETL pipeline using PySpark, SQL Server, and Medallion Architecture (Bronze/Silver/Gold layers) for data warehousing
Why this skill is useful
Adds executable SQL scripts and a complete ETL pipeline setup for retail data processing that aren't available in public documentation.
What it needs
Requires apache-airflow, docker, docker-compose, pyspark, python, sql-server installed locally. About 7k tokens when loaded. Last updated 2026-08-05. 5 stars on the source repository.
What this skill does
Retail ETL Pipeline - Medallion Architecture Skill Skill by ara.so — Data Skills collection Overview The Retail ETL Pipeline project implements a complete data engineering solution for retail operations using the Medallion Architecture pattern (Bronze → Silver → Gold layers). It handles complex retail scenarios including: Inventory shrinkage resolution Recipe conversions for meat/poultry products Supplier rebate tier tracking Multi-branch sales consolidation Stock level management across locations The pipeline processes raw CSV data from CRM/ERP systems through three progressive quality layers, ultimately delivering a "Single Version of Truth" for business intelligence. Architecture Layers Bronze Layer (Raw Ingestion) Raw data ingestion from CSV files Minimal transformation, preserving source format Audit columns: loadedat, sourcefile Silver Layer (Cleaned & Standardized) Data type enforcement Deduplication Standardization (dates, currencies, product codes) Business rule validation Gold Layer (Business-Ready Analytics) Aggregated metrics Calculated KPIs (inventory turnover, shrinkage %) Dimensional models for BI tools Installation & Setup Prerequisites Infrastructure Setup Database Initialization Key SQL Scripts Execution Order The pipeline consists of 13+ SQL scripts that must be executed sequentially: Core ETL Patterns Pattern 1: Bronze Layer Ingestion (Raw CSV → SQL) Pattern 2: Silver Layer Cleansing Pattern 3: Gold Layer Aggregations Pattern 4: Inventory Shrinkage Calculation PySpark Integration (Optional) For large-scale data processing, integrate PySpark for Silver/Gold transformations: Airflow DAG Example Orchestrate the entire pipeline with Apache Airflow: Configuration Environment Variables Docker Compose Configuration Common Patterns & Use Cases Use Case 1: Multi-Branch Sales Consolidation Use Case 2: Recipe Yield Tracking (Meat/Poultry) Use Case 3: Supplier Rebate Tier Tracking Troubleshooting Issue 1: CSV Bulk Insert Fails Issue 2: Duplicate Records in S …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills aradotso/retail-etl-pipeline-medallion