Agent skill · software engineering · sickn33
remote-gpu-trainer
Deploy, monitor, and debug long GPU jobs on RENTED/remote instances (AutoDL, RunPod, vast.ai, Lambda, Slurm, K8s): teardown/billing safety, spot resilience, resumable checkpointing, OOM/NaN triage.
Why this skill is useful
Adds executable scripts for managing GPU job orchestration across various rented platforms, including error handling and monitoring patterns not found in public documentation.
What it needs
Requires bash, python installed locally. About 11k tokens when loaded. Last updated 2026-08-07. 44,571 stars on the source repository.
What this skill does
remote-gpu-trainer — Remote GPU Job Orchestration Overview Deploy and babysit long-running GPU jobs on rented boxes you don't own, across any platform, and get the result off the box before the meter or a preemption kills it. The core insight: you are a short-term tenant on someone else's machine — so the job is to detach the work, make the result outlive the instance, and stop the meter safely, not to provision a cluster. This skill is platform-agnostic at the core, platform-specific at the edges: a fixed set of operating principles + a 6-phase lifecycle that hold everywhere, plus one profile per platform (profiles/<platform>.md) that owns every concrete path, proxy, billing verb, and spot semantic. Its defensible value is the union the big orchestrators skip: Chinese cgroup-isolated rentals + bare-SSH cheap boxes + the disk-budget / monitoring / teardown reality that is the job on metered hardware. When to Use This Skill Use whenever the user deploys, trains, monitors, or troubleshoots a long-running GPU job on a RENTED or remote instance they do not own — training, eval, ablation sweeps, batch inference, or large data processing — on AutoDL, RunPod, vast.ai, Lambda, Paperspace, Chinese platforms (恒源云/矩池云/Featurize/ 揽睿星舟), a bare SSH box, Slurm, or Kubernetes; single OR multi-instance. Triggers (multilingual): "远程 GPU 训练", "GPU 租赁", "GPU rental", "租卡", "spot 抢占", "spot preemption", "断点续训", "resumable training", "tmux 训练守护", "防 SSH 断线", "scp/rsync 上传", "多实例 ablation", "远程 GPU 监控", "省钱关机/销毁实例", "stop vs terminate billing", "checkpoint 磁盘满", "CUDA OOM/显存不足", "loss NaN/loss spike", "loss 不下降/不收敛", "overfit 单 batch", "FSDP/DeepSpeed 配置", "多卡训练 hang", "dataloader worker/数据增广 bug". NOT for purely local single-GPU training, in-instance multi-GPU DDP (use torchrun/accelerate), managed multi-cloud price-shopping (use SkyPilot's skill), or zero-ops serverless (use Modal). …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills sickn33/remote-gpu-trainer--9737a9