Agent skill · software engineering · aj-geddes
ML Pipeline Automation
Build end-to-end ML pipelines with automated data processing, training, validation, and deployment using Airflow, Kubeflow, and Jenkins
Why this skill is useful
Adds executable scripts for end-to-end ML pipeline automation using Airflow, Kubeflow, and Jenkins that are not commonly found in public documentation.
What it needs
Requires apache-airflow, jenkins, kubeflow, mlflow installed locally. About 5k tokens when loaded. Last updated 2026-03-04. 307 stars on the source repository.
What this skill does
ML Pipeline Automation ML pipeline automation orchestrates the entire machine learning workflow from data ingestion through model deployment, ensuring reproducibility, scalability, and reliability. Pipeline Components Data Ingestion: Collecting data from multiple sources Data Processing: Cleaning, transformation, feature engineering Model Training: Training and hyperparameter tuning Validation: Cross-validation and testing Deployment: Moving models to production Monitoring: Tracking performance metrics Orchestration Platforms Apache Airflow: Workflow scheduling with DAGs Kubeflow: Kubernetes-native ML workflows Jenkins: CI/CD for ML pipelines Prefect: Modern data flow orchestration Dagster: Asset-driven orchestration Python Implementation Pipeline Best Practices Modularity: Each step should be independent Idempotency: Tasks should be safely repeatable Error Handling: Graceful degradation and alerting Versioning: Track data, code, and model versions Monitoring: Track execution metrics and logs Scheduling Strategies Daily: Standard for daily retraining Weekly: For larger feature engineering On-demand: Triggered by data updates Real-time: For streaming applications Deliverables Automated pipeline DAG Task dependency graph Execution logs and monitoring Performance metrics Rollback procedures Documentation
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills aj-geddes/ml-pipeline-automation