Agent skill · mindrally
automl-hyperparameter-optimization
Best practices for AutoML and hyperparameter search with Optuna, Ray Tune, and PyCaret, covering search-space design, validation splits, and leakage prevention. Use when tuning model hyperparameters, setting up a pruned or distributed hyperparameter search, designing a nested validation scheme, or evaluating whether an AutoML leaderboard result is production-ready.
What it needs
About 5k tokens when loaded.
What this skill does
AutoML and Hyperparameter Optimization This skill covers designing sound hyperparameter searches and using AutoML tooling (Optuna, Ray Tune, PyCaret, time-series AutoML libraries) without bypassing problem framing, validation design, or explainability. Workflow for Running a Hyperparameter Search 1. Define the target metric and baseline first — Pick the metric before selecting tooling, and train a simple baseline (linear model, random forest, or naive time-series forecast) with a fixed, minimal search. 2. Design the validation scheme — Use nested cross-validation or a final untouched test split for any model-selection claim; use time-aware splits (never shuffled) for time-series problems. 3. Fit preprocessing inside the fold — Fit scalers, encoders, and imputers only on the training portion of each fold to prevent leakage. 4. Define a structured search space — Use log-scale ranges for learning rates, regularization strength, and tree counts; keep ranges domain-informed rather than arbitrarily broad. 5. Choose the right tool — Optuna or Ray Tune for custom training loops with pruning and distributed trials; PyCaret for a quick low-code comparison on a straightforward tabular problem; a time-series-specific library (AutoTS, Merlion, PyAF) when seasonality and horizon handling need first-class support. 6. Run with resource limits and pruning — Set a trial or time budget and use early stopping/pruning so bad trials don't consume the full budget. 7. Track every run — Log datasets, splits, metric definitions, random seeds, library versions, and the search space itself to MLflow, Weights & Biases, TensorBoard, or an equivalent tracker. 8. Report against the baseline — Compare the selected model to the baseline and at least one non-AutoML alternative before calling it production-ready. Experiment Design Define the target metric before choosing tooling — the metric shapes the search space and the pruning strategy, not the other way around. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills mindrally/automl-hyperparameter-optimization