Agent skill · data analytics · davila7
ray-data
Scalable data processing for ML workloads. Streaming execution across CPU/GPU, supports Parquet/CSV/JSON/images. Integrates with Ray Train, PyTorch, TensorFlow. Scales from single machine to 100s of nodes. Use for batch inference, data preprocessing, multi-modal data loading, or distributed ETL pipelines.
Why this skill is useful
Provides domain-specific knowledge for scalable data processing in ML workloads, including integration patterns with Ray, PyTorch, and TensorFlow.
What it needs
Requires pandas, pyarrow, ray[data] installed locally. About 4k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
Ray Data - Scalable ML Data Processing Distributed data processing library for ML and AI workloads. When to use Ray Data Use Ray Data when: Processing large datasets (>100GB) for ML training Need distributed data preprocessing across cluster Building batch inference pipelines Loading multi-modal data (images, audio, video) Scaling data processing from laptop to cluster Key features: Streaming execution: Process data larger than memory GPU support: Accelerate transforms with GPUs Framework integration: PyTorch, TensorFlow, HuggingFace Multi-modal: Images, Parquet, CSV, JSON, audio, video Use alternatives instead: Pandas: Small data (<1GB) on single machine Dask: Tabular data, SQL-like operations Spark: Enterprise ETL, SQL queries Quick start Installation Load and transform data Integration with Ray Train Reading data From cloud storage From Python objects Transformations Map batches (vectorized) Row transformations Filter Group by and aggregate GPU-accelerated transforms Writing data Performance optimization Repartition Batch size tuning Streaming execution Common patterns Batch inference Data preprocessing pipeline Integration with ML frameworks PyTorch TensorFlow Supported data formats Format Read Write Use Case -------- ------ ------- ---------- Parquet ✅ ✅ ML data (recommended) CSV ✅ ✅ Tabular data JSON ✅ ✅ Semi-structured Images ✅ ❌ Computer vision NumPy ✅ ✅ Arrays Pandas ✅ ❌ DataFrames Performance benchmarks Scaling (processing 100GB data): 1 node (16 cores): ~30 minutes 4 nodes (64 cores): ~8 minutes 16 nodes (256 cores): ~2 minutes GPU acceleration (image preprocessing): CPU only: 1,000 images/sec 1 GPU: 5,000 images/sec 4 GPUs: 18,000 images/sec Use cases Production deployments: Pinterest: Last-mile data processing for model training ByteDance: Scaling offline inference with multi-modal LLMs Spotify: ML platform for batch inference References Transformations Guide - Map, filter, groupby operations Integration Guide - Ray Train, PyTorch, TensorFlow Resources …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/data-processing-ray-data