Agent skill · research science · sickn33
run-deep-swe
Run reproducible DeepSWE coding-agent benchmark evaluations through OpenRouter and mini-swe-agent.
Why this skill is useful
Provides specific commands and configurations for running DeepSWE benchmarks that the AI wouldn't reliably generate on its own.
What it needs
Requires docker, git, uv installed locally. Requires openrouter account access. About 3k tokens when loaded. Last updated 2026-08-07. 44,571 stars on the source repository.
What this skill does
Run DeepSWE via OpenRouter When to Use Use when the user wants to benchmark a model on DeepSWE or mini-swe-agent tasks. Use when you need a reproducible coding-agent evaluation plan and output artifacts. DeepSWE (deepswe.datacurve.ai) is a 113-task Harbor-compatible coding-agent benchmark. It runs via Pier (Harbor fork) driving mini-swe-agent (model-agnostic). Any model reachable through OpenRouter can be scored. Prerequisites — state-check first Docker must be running — Pier sandboxes each task in Docker by default (--env modal for cloud instead). OPENROUTERAPIKEY must already be present in the environment. If it is unset, ask the user to configure their preferred secret-management path; do not read shell startup files, print secrets, or invent a key. Setup Run all pier commands from inside deep-swe/, using relative -p tasks/.... OpenRouter wiring (the part the docs don't spell out) mini-swe-agent has a native OpenRouter model class. Both routes below use OPENROUTERAPIKEY and the OpenRouter slug (vendor/model, e.g. minimax/minimax-m3): Route A — native OpenRouter class (preferred, hits openrouter.ai/api/v1 directly): Route B — LiteLLM provider prefix (fallback; same key): Notes: Slug = the exact OpenRouter slug. Verify it at openrouter.ai/models before running. Free/zero-cost models: OpenRouter cost tracking can error. Set export MSWEACOSTTRACKING=ignoreerrors. Flag spelling can vary by version — confirm with pier run --help and mini --help. Smoke test FIRST (1 task — do this before any full run) Always validate end-to-end wiring on a single task before spending tokens on the corpus: Pass criteria: run completes, model returns actions (not auth/format errors), a score/trajectory is emitted. If it 401s → key wrong. If "provider not provided"/"model not mapped" → fix slug or switch route. Subset run (deterministic sample) Full 113-task corpus (costs tokens + time — confirm with user first) Output & leaderboard Trials land in jobs/<run>/<trialid>/. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills sickn33/run-deep-swe--7b04f7