Agent skill · mindrally
pyspark-etl
Best practices for building performant, testable PySpark ETL pipelines with Spark SQL and Apache Iceberg. Use when writing or reviewing PySpark jobs, designing joins and window functions, working with map/array higher-order functions, or building idempotent cumulative/snapshot table merges.
What it needs
About 5k tokens when loaded.
What this skill does
PySpark ETL This skill covers patterns for building production-grade, testable ETL pipelines with PySpark, Spark SQL, and Apache Iceberg, including project structure, join and window-function idioms, and safe cumulative-table merge patterns. Workflow for Building a PySpark ETL Job 1. Scaffold the job class — Create a class that manages the SparkSession lifecycle, accepts an injectable session for testing, and exposes an abstract runjob method. 2. Define config via a factory function — Keep config as a plain dataclass; parse CLI args in a separate factory function so tests can construct configs without touching sys.argv. 3. Read source data with a shared, partition-aware reader — Use a generic reader utility for date filters, hour ranges, and latest-partition lookups; keep business filters in the ETL class. 4. Compose the pipeline with .transform() — Chain named methods (readsource().transform(self.enrich).transform(self.mergewithexisting)) so runjob stays pure orchestration. 5. Apply transformations idiomatically — Use select over withColumn chains, explicit join types, explicit window frames, and native functions instead of UDFs. 6. Write with schema-evolution safety — Use .byName() when writing to Iceberg tables so column order doesn't matter. 7. Validate output — Check primary-key uniqueness and null counts on key columns after every write. 8. Test locally — Unit test transformation methods against a local SparkSession with small, hand-built DataFrames. Project Structure ETL class scaffold Config as a factory function Keep the dataclass as pure data; put CLI parsing in a standalone factory so configs are easy to build in tests. Partition-aware shared reader Build one generic reader for partition mechanics; keep domain-specific filters visible in the ETL, not buried in a one-off reader class. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills mindrally/pyspark-etl