Agent skill · software engineering · secondsky
ml-pipeline-automation
Automate ML workflows with Airflow, Kubeflow, MLflow. Use for reproducible pipelines, retraining schedules, MLOps, or encountering task failures, dependency errors, experiment tracking issues.
Why this skill is useful
Adds executable scripts for automating ML workflows with Airflow and MLflow that aren't commonly found in public documentation.
What it needs
Requires apache-airflow, mlflow installed locally. About 5k tokens when loaded. Last updated 2026-08-06. 204 stars on the source repository.
What this skill does
ML Pipeline Automation Orchestrate end-to-end machine learning workflows from data ingestion to production deployment with production-tested Airflow, Kubeflow, and MLflow patterns. When to Use This Skill Load this skill when: Building ML Pipelines: Orchestrating data → train → deploy workflows Scheduling Retraining: Setting up automated model retraining schedules Experiment Tracking: Tracking experiments, parameters, metrics across runs MLOps Implementation: Building reproducible, monitored ML infrastructure Workflow Orchestration: Managing complex multi-step ML workflows Model Registry: Managing model versions and deployment lifecycle Quick Start: ML Pipeline in 5 Steps Result: Working ML pipeline with experiment tracking in under 5 minutes. Core Concepts Pipeline Stages 1. Data Collection → Fetch raw data from sources 2. Data Validation → Check schema, quality, distributions 3. Feature Engineering → Transform raw data to features 4. Model Training → Train with hyperparameter tuning 5. Model Evaluation → Validate performance on test set 6. Model Deployment → Push to production if metrics pass 7. Monitoring → Track drift, performance in production Orchestration Tools Comparison Tool Best For Strengths ------ ---------- ----------- Airflow General ML workflows Mature, flexible, Python-native Kubeflow Kubernetes-native ML Container-based, scalable MLflow Experiment tracking Model registry, versioning Prefect Modern Python workflows Dynamic DAGs, native caching Dagster Asset-oriented pipelines Data-aware, testable Basic Airflow DAG Known Issues Prevention 1. Task Failures Without Alerts Problem: Pipeline fails silently, no one notices until users complain. Solution: Configure email/Slack alerts on failure: 2. Missing XCom Data Between Tasks Problem: Task expects XCom value from previous task, gets None, crashes. Solution: Always validate XCom pulls: 3. DAG Not Appearing in UI Problem: DAG file exists in dags/ but doesn't show in Airflow UI. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills secondsky/ml-pipeline-automation