Agent skill · NVIDIA
tao-generate-referring-expressions
Four-step image referring-expression pipeline: turns images plus KITTI bounding-box labels into region
What it needs
About 5k tokens when loaded.
What this skill does
Image Referring Expression Pipeline Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery). Generate referring-expression and grounding annotations from images with KITTI-format bounding box labels. A single VLM (Gemini or any OpenAI-compatible endpoint) runs four steps: per-object region descriptions, holistic image captions, grouped grounding expressions tied to bboxes, and an optional double-check verification pass. Purpose Transform (image, KITTI labels) pairs into a unified annotations.jsonl containing rich, grounded referring expressions. The VLM acts as a "teacher" annotator: Steps 0-1 see the image; Step 2 groups Step 0 outputs into grouping phrases with bbox lists; Step 3 (optional) re-examines those bboxes against the image and corrects mismatches. Pipeline Architecture Step 0 (regionexpr) — VLM emits one short discriminative phrase per KITTI bbox (bbox2d, type, color, description). Step 1 (imagecaption) — VLM emits a holistic, location-agnostic scene caption. Step 2 (groundingexpr) — VLM groups Step 0 objects into grouping phrases and returns one bbox list per group, optionally using Step 1's caption as extra context. Step 3 (doublecheck) — VLM re-checks each Step 2 bbox against the image; bad matches are removed, slightly-off boxes get tightened. Steps 0 and 1 run in parallel within a single thread pool (they only depend on the seed records). Each step writes its own step<N>/annotations.jsonl and skips already-processed images on re-run unless workflow.forcereprocess: true. Instructions Initial setup When a user wants to run this pipeline, walk through these steps: 1. Images: Ask for data.imagedir, the directory containing .jpg, .jpeg, or .png images. 2. KITTI labels: Ask for data.kittilabeldir, the directory containing one .txt label file per image. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills NVIDIA/tao-generate-referring-expressions