Agent skill · data analytics · aradotso
spark-data-engineering-pipeline
Production PySpark ETL pipeline with AWS S3, PostgreSQL, schema validation, data quality checks, and automated testing
Why this skill is useful
Adds a complete production-grade ETL pipeline with executable scripts for data extraction, transformation, and loading, which the AI cannot generate from general knowledge.
What it needs
Requires pip, python3 installed locally. Requires aws, postgresql account access. About 7k tokens when loaded. Last updated 2026-08-05. 5 stars on the source repository.
What this skill does
Spark Data Engineering Pipeline Skill by ara.so — Data Skills collection. Overview A production-grade ETL pipeline built with PySpark that extracts JSON data from AWS S3, performs schema validation and data quality checks, transforms the data, writes it as Parquet, and loads it into PostgreSQL. This project demonstrates end-to-end data engineering best practices including logging, testing, and CI/CD. Project Structure Installation Configuration Create a .env file in the project root: Running the Pipeline Core Components 1. Data Extraction from S3 2. Schema Validation 3. Data Quality Checks 4. Data Transformation 5. Write to Parquet 6. Load to PostgreSQL 7. Complete Pipeline Testing Common Patterns Pattern 1: Incremental Data Processing `python Process only new data based on timestamp from datetime import datetime, timedelta lastprocesseddate = datetime(2024, 1, 1) incrementaldf = rawdf.filter( F.col('timestamp') > F.lit(lastprocesseddate) ) Process a
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills aradotso/spark-data-engineering-pipeline