Agent skill · software engineering · davila7
ray-train
Distributed training orchestration across clusters. Scales PyTorch/TensorFlow/HuggingFace from laptop to 1000s of nodes. Built-in hyperparameter tuning with Ray Tune, fault tolerance, elastic scaling. Use when training massive models across multiple machines or running distributed hyperparameter sweeps.
Why this skill is useful
Adds executable scripts for distributed training orchestration that significantly enhance AI's capabilities in scaling machine learning models across multiple nodes.
What it needs
Requires ray[train], torch, transformers installed locally. About 4k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
Ray Train - Distributed Training Orchestration Quick start Ray Train scales machine learning training from single GPU to multi-node clusters with minimal code changes. Installation: Basic PyTorch training (single node): That's it! Ray handles: Distributed coordination GPU allocation Fault tolerance Checkpointing Metric aggregation Common workflows Workflow 1: Scale existing PyTorch code Original single-GPU code: Ray Train version (scales to multi-GPU/multi-node): Benefits: Same code runs on 1 GPU or 1000 GPUs Workflow 2: HuggingFace Transformers integration Workflow 3: Hyperparameter tuning with Ray Tune Result: Distributed hyperparameter search across cluster Workflow 4: Checkpointing and fault tolerance Workflow 5: Multi-node training Launch Ray cluster: When to use vs alternatives Use Ray Train when: Training across multiple machines (multi-node) Need hyperparameter tuning at scale Want fault tolerance (auto-restart failed workers) Elastic scaling (add/remove nodes during training) Unified framework (same code for PyTorch/TF/HF) Key advantages: Multi-node orchestration: Easiest multi-node setup Ray Tune integration: Best-in-class hyperparameter tuning Fault tolerance: Automatic recovery from failures Elastic: Add/remove nodes without restarting Framework agnostic: PyTorch, TensorFlow, HuggingFace, XGBoost Use alternatives instead: Accelerate: Single-node multi-GPU, simpler PyTorch Lightning: High-level abstractions, callbacks DeepSpeed: Maximum performance, complex setup Raw DDP: Maximum control, minimal overhead Common issues Issue: Ray cluster not connecting Check ray status: If not connected: Issue: Out of memory Reduce workers or use gradient accumulation: Issue: Slow training Check if data loading is bottleneck: If data loading is slow, increase workers: Advanced topics Multi-node setup: See references/multi-node.md for Ray cluster deployment on AWS, GCP, Kubernetes, and SLURM. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/distributed-training-ray-train