Agent skill · data analytics · aradotso
enterprise-data-engineering-pipeline-ssis-pyspark
End-to-end ELT pipeline using SSIS, SQL Server, and PySpark for enterprise data warehousing and analytics
Why this skill is useful
Adds executable scripts for setting up a complete enterprise data engineering pipeline with SSIS and PySpark that aren't available in public documentation.
What it needs
Requires java, python, sql server, ssis, visual studio installed locally. About 6k tokens when loaded. Last updated 2026-08-05. 5 stars on the source repository.
What this skill does
Enterprise Data Engineering Pipeline (SSIS + PySpark) Skill by ara.so — Data Skills collection. Overview This project provides a complete enterprise data engineering solution that combines: SSIS (SQL Server Integration Services) for ETL orchestration SQL Server with Star Schema data warehouse design (fact and dimension tables) Python (Pandas) for data quality audits and visualization PySpark for big data analytics and aggregation The pipeline ingests raw CSV files (Sales, Products, Customers), transforms them through SSIS, loads into a dimensional model, and performs analytics at scale. Architecture Components 1. Source Layer: Raw CSV files containing transactional and master data 2. ETL Layer: SSIS packages handle extraction, transformation, error handling 3. Storage Layer: SQL Server Data Warehouse with Star Schema 4. Analytics Layer: Python/PySpark scripts for business intelligence Installation & Setup Prerequisites Python Dependencies Database Setup SSIS Package Configuration Creating the SSIS Project 1. Open Visual Studio with SSDT 2. Create new Integration Services Project: EnterpriseETL.sln 3. Add Connection Managers: SourceFlatFile: Points to CSV directory DestinationOLEDB: SQL Server connection string SSIS Package Flow Error Handling in SSIS Python Analytics Data Quality Audit Script Customer Segmentation Analysis PySpark Big Data Processing High-Volume Sales Aggregation Parallel Processing for Large Datasets Common Patterns Incremental ETL (Load Only New Records) Automated Data Refresh Troubleshooting SSIS Connection Issues Unicode/Encoding Errors PySpark JDBC Driver Not Found Performance Optimization Environment Variables Running the Complete Pipeline This enterprise pipeline provides a complete solution for data warehousing, ETL automation, and big data analytics using industry-standard Microsoft and Apache technologies.
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills aradotso/enterprise-data-engineering-pipeline-ssis-pyspark