--- name: cdf-anomalygen description: >- Use when running the CDF AnomalyGen pipeline over a defect dataset — mask placement, fine-tuning, synthetic defect-image generation (SDG), evaluation, quality refinement, and pseudo-labeling — even when the user names only one stage. license: Apache-2.0 compatibility: >- Requires a CUDA GPU and the built env (native .venv or the Docker image); Hugging Face auth + network access to download the model checkpoints. metadata: owner: NVIDIA service: anomaly generation version: 1.2.0 author: NVIDIA tags: - physical-ai - anomaly-generation - cosmos - fine-tuning languages: [python] frameworks: [pytorch, cosmos, diffusion] --- # Skill: cdf-anomalygen ## Purpose AnomalyGen fine-tunes the frozen **Cosmos** base into an **anomaly-generation** model: given a clean image + defect mask, it paints a realistic defect into that region (synthetic data for detectors). Pick a mode, collect its params, run the steps — **all commands from the repo root.** ## When to Use This Skill Use for the CDF AnomalyGen pipeline over a defect dataset — any single stage (mask placement, fine-tuning, generation, evaluation, refinement, or pseudo-labeling) or the full run. - "Run the AnomalyGen pipeline on phone_screen" - "Fine-tune an anomaly-generation model for PCB defects" - "Generate 25 synthetic PCB anomaly images" - "Place defect masks", "evaluate quality", or "pseudo-label the generated defects" Do not use this skill for generic image generation, or for synthetic data that isn't defect images built from the `{texture}/{defect}` clean-image + defect-mask contract. ## Prerequisites - A **CUDA GPU** and the project's built env (native `.venv` or the Docker image). - **Hugging Face auth** — run `hf auth login` — plus network access; the downloader fetches the model checkpoints. - A **`dataset_dir`** in the Step 1 contract layout (clean images + `{defect}` masks; plus `defect_spec.jsonl` **only when `mask_backend=amp`** — the default `amg` never reads it, and nothing after Step 2 does either). Preflight the env + checkpoints (GPU, Python, CUDA/torch, deps, HF auth, model checkpoints): ```shell scripts/preflight_env_ckpt.sh ``` **Read which group failed before reacting** — the summary separates them, and only one is fixed by downloading: - *missing checkpoints* → `scripts/download_checkpoints.sh` (run `hf auth login` first). - *env / Python / deps* → rebuild the environment (`bash scripts/env_setup.sh`, then activate it). **Third group — model access for mask placement (checked HERE, never mid-flow).** The default `mask_backend=amg` needs a vision model, and a run that discovers this at Step 2 has already spent the user's time. A **VLM API is a base, a key and a model name** — and all three live in ONE place: **`~/.anomalygen/endpoints.yaml`**, the model registry (nothing is built into the code; the template is `scripts/agentic_mask_generation_serving/endpoints.example.yaml`). It sits under `$HOME`, not in the repo, so it works from any directory, serves every checkout, and is writable inside the read-only product containers. The moment inputs are locked with `mask_backend=amg` (the default), probe: ```shell [ -f ~/.anomalygen/endpoints.yaml ] && echo "amg model: ~/.anomalygen/endpoints.yaml present" \ || echo "amg model: no ~/.anomalygen/endpoints.yaml — NO model can resolve; copy scripts/agentic_mask_generation_serving/endpoints.example.yaml there" ``` **The model resolves in the registry → use it.** That is the whole check: `--gen-model` and `--judge-model` are REQUIRED flags (there is no default model), and every name they carry must have a registry entry (`model` + `base_url` + `key`) — the entry point refuses to start when one does not; there is no environment fallback. A user's own OpenAI-compatible endpoint is the same case: one `endpoints.yaml` entry with their documented base URL, their key, and their model name in `--gen-model`/`--judge-model`. **Anything less → stop and ask the user now**, before Step 1 — unless the user's request already names the model (e.g. "use local Gemma"): that IS the pick, act on it without re-asking. Three options — whichever it is, set `AMG_MODEL` to the chosen name; the Step 2 command passes it to both required flags: - **(a) supply an API** — one `endpoints.yaml` entry: a model name, a base and a key (e.g. `AMG_MODEL=azure/openai/gpt-6-astra`). - **(b) let the skill start the model service** — in a container, as a background process inside it; on a dev host, from `scripts/agentic_mask_generation_serving/`. Needs a GPU; the weights are a default checkpoint — `scripts/download_checkpoints.sh gemma` fetches them and preflight verifies them (the group argument fetches only this model; bare, the script fetches all eleven). `mkdir -p ~/.anomalygen && cp scripts/agentic_mask_generation_serving/endpoints.example.yaml ~/.anomalygen/endpoints.yaml` wires it: the template's gemma entry already routes `AMG_MODEL=nvidia/google/gemma-4-31b-it` to `http://localhost:8902/v1`, and the run carries `--beam-multi 4 --beam-pairwise --exemplar-captions --exemplar-visual-prompt --coverage-note` — the local model always pairs with the in-loop beam and the three information channels. Bootstrap, lifecycle and the why: [`agentic-mask-generation.md`](references/agentic-mask-generation.md). - **(c) set `mask_backend=amp`** for this run. Never fall back silently. Whichever they pick, the run's own capability probe confirms the model can take images and answer in JSON before Step 2 spends anything. ## Inputs Collect the run's inputs from the user before starting. **If the `mode` or any input it requires is missing, ask the user first — never guess a required input or silently fall back to its default.** **Lock the `mode` first** — it fixes which steps run and which inputs are required (ask for any not provided): - **`full`** — train + generate, Steps **1 → 7**. Required: `name`, `dataset_dir`, `num_sdg`. - **`finetune_only`** — train only, Steps **1 → 3**. Required: `name`, `dataset_dir`, `num_sdg`. Here `num_sdg` sizes only the validation set. - **`generation_only`** (a.k.a. `inference_only`) — generate from an existing checkpoint, Steps **2, 4 → 7**. Required: `name`, `checkpoint`, `recipe`, `dataset_dir`, `num_sdg`. The checkpoint's geometry stamp must equal `recipe`'s. A legacy checkpoint (trained before the stamp existed) has none: pass its run-dir recipe copy and it is stamped from that recipe at load (generation only; it cannot be resumed). `name` is required in every mode — an experiment label that names the recipe, run dir, logs and the generated mask-set dirs. It is never a dataset location: where the data lives is `dataset_dir`, always passed separately and never derived from `name`. **Optional in every mode** — all have working defaults, so ask only if the user raises them: `num_gpus` (`1`), `num_search_run` (`3`), `score` (`nn`), `task` (`texture_ft`), `base_seed` (`1`), `mask_backend` (`amg`), `legacy_fixed_size` (off), `per_clean_pairing` (`off`). Anything not listed as **Required** above is optional. `legacy_fixed_size` switches Step 3 to the legacy fixed-input-size training mechanism. It is off by default: every fine-tune uses the current variable-input-size mechanism unless the user **explicitly** asks for the legacy one ("FT using the legacy fixed-size mechanism", "old fixed input size setting", "previous release FT mode", etc.). Never infer it from the dataset, the image sizes, or KPI numbers, and never ask about it when the user has not raised it. When set, the run takes `job_name: anomalygen_texture_ft_legacy` — in the recipe and as `--job_name` to `set_pipeline_vars.sh` — so it never shares a run dir with a default run of the same `name`, and Step 3 appends the legacy block to the recipe. `mask_backend` picks how Step 2 places masks: **`amg`** (default) — the agentic engine proposes, refines and judge-accepts every mask, all defect types included; **`amp`** — the allocate → pair → place pipeline; no VLM beyond `text` ROIs. AMG needs model access (preflight checks before spending); if the user cannot supply it and does not choose, stop and ask — never fall back silently. Parameter tables (each mapped to its CLI arg) and the step map: [`inputs.md`](references/inputs.md). Confirm the inputs, set the shared variables below, then run straight through — in `full`, `checkpoint` is derived after Step 3, so don't pass it. ## Instructions Run the steps your `mode` selects (see **Inputs**), in order — each step's header notes when to skip it. > **A step is done only when its output exists on disk — not when its command is *launched*.** Every > step ends with a named gate (**dataset**, **amp**, **train**, …); run it before moving on. Never > treat starting a long job as a stopping point, nor proceed on a partial artifact. Gate commands and > the fix for each failure: [`references/verification.md`](references/verification.md). Set the shared variables every step below refers to. **Re-source this in each new shell** — variables do not survive one — and read the values it echoes back (`${DATASET_DIR}`, `${VAL_DIR}`, `${GEN_DIR}`, `${OUT}`, `${RUN_DIR}`, `${LOG_DIR}`, …): ```shell source scripts/skill_utility/set_pipeline_vars.sh --name --dataset_dir \ --num_sdg [--num_gpus 1] [--num_search_run 3] [--task texture_ft] \ [--job_name anomalygen_texture_ft_legacy] # legacy_fixed_size only export SCORE="" # repeat the confirmed choice in every shell; use nn when none was requested ``` **`--dataset_dir` is required and is never derived from `--name`.** The name is an experiment label; the dataset is a separate input that comes from the user. They differ in the normal case — a label of `uc1pega-opus5-e2e` over a dataset at `datasets/uc1pega` — so a name-derived path would be wrong far more often than right, and wrong *silently*: the recipe's `dataset_path` is set separately, so the recipe can point at the right dataset while `${DATASET_DIR}` hands Steps 5 and 6 a `--real_root` that scores against nothing. The script does **not** check that the path exists; gate **dataset** does, one step later. **Every step's log goes under `${LOG_DIR}` (`results/`), never the repo root.** Use `${LOG_DIR}/_${NAME}.log`. The same goes for PID files — the launcher writes its own under `results/`; do not add one. The steps below pipe through `tee` to keep the log and the console. **Run `set -o pipefail` first** — without it the pipeline's exit status is `tee`'s, so a step that died reports success: ```shell set -o pipefail ``` `set_pipeline_vars.sh` **rejects** a `name`/`task` outside `[A-Za-z0-9][A-Za-z0-9._-]*` and any non-integer count rather than sanitizing them — both reach command lines below. Apply the same care to user-supplied paths (`dataset_dir`, `recipe`, `checkpoint`): quote every expansion. `texture_ft` is the only `task_type` today; a new task family adds one `${TASK}`→`${SCRIPTS}` line to that script plus a matching `task_type:` in the recipe. Step 3 sets `CKPT` and repoints `RECIPE` at the run-dir copy for Steps 4–7. **`generation_only` skips Step 3**, so set both yourself before Step 4 (use the user's `checkpoint` verbatim if given; `CKPT` and `RECIPE` must come from one run or class ids drift): ```shell RUN_DIR= # only if it is not the ${RUN_DIR} already set above CKPT="${RUN_DIR}/checkpoints/model/$(cat "${RUN_DIR}/checkpoints/best_checkpoint.txt")" RECIPE="${RUN_DIR}/exp_${TASK}_${NAME}.yaml" # run-dir copy — MUST match the checkpoint ``` ### Step 1 — dataset (contract layout) All modes need `dataset_dir` on disk in this layout — training and eval read the `anomaly_image/`+`mask/` pairs; both Step 2 backends read `clean_image/` (AMG also samples those same pairs as exemplars); `defect_spec.jsonl` has exactly one consumer, the AMP backend: ```text {dataset_dir}/{texture}/anomaly_image/{defect}/.png # defect image {dataset_dir}/{texture}/mask/{defect}/_mask.png # binary mask (>=128 = defect); same stem + _mask {dataset_dir}/{texture}/clean_image/.png # clean images (Step 2 inputs — both backends) {dataset_dir}/{texture}/cad_mask/.png # cad/cad_tht only: one per clean image, no {defect} level {dataset_dir}/semantic_segmentation_labels.json # cad/cad_tht only: label map for those CAD masks {dataset_dir}/defect_spec.jsonl # amp only: one row per {texture}+{defect}; sets spatial_dependency (free/text/cad/cad_tht). amg never reads it — absent is normal ``` Ready-made subjects: ```shell python -m anomalygen.scripts.datasets.prepare_phone_screen_defect --output_dir "${DATASET_DIR}" --zip # --zip is required (the images are a manual Roboflow download); --masks_only asks for a dataset with no images. # also: prepare_pcb_defect, prepare_magnetic_tile_defect (see anomalygen/scripts/datasets/README.md) ``` For custom data, emit the same layout and list every `[texture, defect]` pair in the recipe's `anomaly_types`. Confirm the layout before proceeding — gate **dataset**. ### Step 2 — mask placement Two backends produce the same artifact (placed masks + `testcase.jsonl`); everything downstream is backend-blind. Default is **AMG**; set `mask_backend=amp` for the classic pipeline. Model access was verified at **preflight** (third group). If preflight was skipped, run that probe now — it must pass before anything below spends money. One notice: if `defect_spec.jsonl` is present and declares `spatial_dependency`, tell the user — **AMG is the default and does not read that field** (it decides placement by looking at the board); to honour `free`/`text`/`cad` routing, run with `mask_backend=amp`. Do not change the default over it. A dataset with no `defect_spec.jsonl` at all is normal under `amg` — nothing in the pipeline reads it then, and the grep below is simply silent; under `amp` its absence is a **dataset** gate failure. ```shell grep -l spatial_dependency "${DEFECT_SPEC}" >/dev/null 2>&1 && echo "NOTE: spatial_dependency present but mask_backend=amg ignores it; use mask_backend=amp to honour it" ``` #### AMG (default) **Reference** (model resolution, the model-service bootstrap, the capability probe, cost, resume, the run's own deviation ledger): [`agentic-mask-generation.md`](references/agentic-mask-generation.md). One run per output set, each sized by `--num-sdg` — the same request AMP answers, split across the types by the same rule (base each, remainder to the first types). **Both sets use `${NUM_SDG}`**, the same convention as AMP, where `--num_sdg` sizes the validation and the generation set alike. The deliverable is exactly `num_sdg` rows, so gate **amg** checks the number that was asked for: ```shell AMG_MODE=validation; AMG_DIR="${VAL_DIR}"; ALLOC=validation # names the log, the out-dir, the split # then again: AMG_MODE=generation AMG_DIR="${GEN_DIR}" ALLOC=inference python -m anomalygen.scripts.agentic_mask_generation.generate \ --dataset-path "${DATASET_DIR}" --anomaly-types + [...] \ --gen-model "${AMG_MODEL}" --judge-model "${AMG_MODEL}" \ --model-registry "${AMG_REGISTRY:-$HOME/.anomalygen/endpoints.yaml}" \ --num-sdg "${NUM_SDG}" --mode "${ALLOC}" \ --visual-prompt auto --defect-library auto \ --out-dir "${AMG_DIR}" 2>&1 | tee "${LOG_DIR}/amg_${AMG_MODE}_${NAME}.log" ``` `--model-registry` is explicit, not left to the default, because Step 2 may run inside a container whose `$HOME` is not the host's: the flag makes "which registry answered" a fact of the recorded command. Set `AMG_REGISTRY` to wherever the registry actually is (e.g. the path it was mounted at). **`--mode` is why the two sets are not interchangeable.** The generation set is `inference`: an even split, and a type may legitimately get 0 when `num_sdg` is below the type count. The validation set is `validation`: split in proportion to each type's real mask count, so it mirrors what the model trains on — and a share that would round to 0 is **refused**, because validation scores per type and a type with no samples drops silently out of the KPI that picks `best_checkpoint.txt`. Both are AMP's rules, shared through one allocation module rather than reimplemented. **`--num-sdg` is the only way to size a run**, as it is for AMP. It splits the total across the types rather than sizing each alike, so 25 over 3 types is 9/8/8 and lands on 25 — where a per-type count could only reach 27, paying for two masks before anything discards them. For exact per-type targets instead, `--per-defect-counts '{"+": N}'`; omitted types get 0. **`AMG_MODE` is in the log name because there are two runs.** Both would otherwise `tee` to one file and the second would overwrite the first, silently discarding the validation run's deviation ledger — the same reason AMP's log carries `${AMP_MODE}`. **The validation set's size is a training-time cost, not just a one-off API charge.** Every validation pass re-renders the whole `testcase_jsonl` set and runs one SAM2 forward per sample (~0.4 s each — see [`fine-tuning.md`](references/fine-tuning.md)), repeated `max_iter / validation_iter` times. To spend less, lower `num_sdg`; do **not** raise `validation_iter`, which also coarsens the resolution `best_checkpoint.txt` is chosen at. AMG places every type it is given. A stopped or interrupted run **continues from where it was** when re-run with the same command. The run records every deviation as `degraded[KIND]` in its log — read the summary before trusting the output. Verify the deliverable directly against gate **amg** ([`verification.md`](references/verification.md)): `testcase.jsonl` rows = `${NUM_SDG}`. The fill loop delivers in beam-sized batches and then trims to the target, so `trimmed from N` with `N` well above a type's share is normal, not a defect — the surplus is kept under `{texture}/_overflow/{defect}/` + `testcase_overflow.jsonl` and is deliberately **not** counted in `testcase.jsonl`, so counting mask files on disk will exceed the row count with nothing wrong. When the user asks for different backends on different types, each backend places its own subset and this count is over the union of the runs' rows (the AMP gate's `roi_*` checks and remedies are for AMP out-dirs only). **When the model is served here, the service spans Step 2 and nothing else.** Both mask sets are Step 2 products, so serve once and run both against it — then stop it. Serving holds a large share of the GPU, and Step 3 (`full`) and Step 4 (`generation_only`) need that GPU back: ```shell scripts/agentic_mask_generation_serving/serverctl.sh health # idempotent: probe first, start only if down case $? in # 0 up · 1 not running · 2 alive but loading 0) : ;; # ready — run the two generate runs below 1) echo "DOWN — start it (references/agentic-mask-generation.md), then wait for it" ;; 2) echo "LOADING — call: scripts/agentic_mask_generation_serving/serverctl.sh wait (repeat while it exits 2)" ;; esac # ... the two generate runs above: AMG_DIR=${VAL_DIR}, then AMG_DIR=${GEN_DIR} ... scripts/agentic_mask_generation_serving/serverctl.sh stop # part of this step's gate scripts/agentic_mask_generation_serving/serverctl.sh health # ONLY exit 1 means the GPU is free [ $? = 1 ] || echo "STILL HOLDING THE GPU — Step 3 will contend for it" ``` **Test for exit 1, not for failure.** A server that is up (0) and one still loading (2) both hold their `--gpu-memory-utilization 0.9` reservation, so `health && echo ...` would clear the gate on a process that never finished starting and is still sitting on the car