Agent skill · NVIDIA
cupynumeric-parallel-data-load
Load a sharded, on-disk dataset (sharded .npy, Parquet/Arrow, raw binary, sharded HDF5, custom layouts) into a distributed cuPyNumeric ndarray via a manual partition + leaf @task launch with CPU/OMP/GPU variants. Use when no single-call loader fits, including when per-shard row counts differ across files. Prefer cupynumeric.load or legate.io.hdf5.from_file when they apply.
What it needs
About 10k tokens when loaded.
What this skill does
Parallel sharded data -> cupynumeric load Why this skill exists. cupynumeric mirrors NumPy's array API, including cupynumeric.load for a single .npy file. Beyond that, file loading lives in Legate, not cupynumeric: Format Built-in loader --- --- Single .npy cupynumeric.load(path) (NumPy-API parity) HDF5 (single file) legate.io.hdf5.fromfile / fromfilebatched Sharded multi-file (any format), Parquet/Arrow, raw binary, custom layouts No built-in loader — this skill. This skill shows the canonical way to fill the gap in the last row: write a Legate Python task that calls the third-party reader the format needs (h5py, pyarrow, np.memmap, ...) inside the task body, and let Legate distribute the reads across GPUs / nodes. For the formats with a built-in loader, prefer it unless you need a custom in-task body (mmap-based loader, format-specific decoder, sidecar metadata, partial / sharded reads). Canonical pattern: manual partition + manual task launch, sized to the machine, not the files. Only axis 0 is sharded; trailing axes ride along inside each tile. Per-shard row counts may differ across files (only dtype and trailing axes must match); the launch fills every available processor regardless of how many files there are. .npy is the worked example because the header carries shape and dtype on disk, but the skeleton applies to any format with cheap range/slice reads (raw binary, HDF5, Parquet/Arrow — see "Other formats" below). Reference implementation: assets/examples/parallelnpyload.py. Data layout assumption This skill is purely about loading — it assumes the data is already laid out on a shared filesystem in some predictable, indexable way. Producing those files is out of scope (the example ships a write subcommand for convenience, but real users bring their own). The worked example assumes one specific layout: A directory containing files named shard0000.npy, shard0001.npy, ... in a contiguous integer sequence (zero-padded width 4). …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills NVIDIA/cupynumeric-parallel-data-load