Agent skill · NVIDIA
tao-generate-image-grounding
Two-step image grounding pipeline: extracts referring expressions from (image, caption) pairs and grounds them
What it needs
About 4k tokens when loaded.
What this skill does
Image Grounding Pipeline Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery). Turn (image, caption) pairs into per-image grounded annotations: cleaned captions, referring expressions with character spans, and pixel-space bounding boxes for each expression. A single VLM (Gemini or any OpenAI-compatible endpoint) handles both steps. Purpose Generate phrase-grounded training data for referring-expression and grounding models. The VLM acts as a "teacher" annotator: Step 0 extracts referring expressions from the caption while looking at the image; Step 1 returns one bbox set per expression for each image. Pipeline Architecture Steps are individually selectable via workflow.steps. Each step writes a per-sample checkpoint to step<N>/.ckpt/<sampleid>.json and skips already-processed records on re-run. Set workflow.forcereprocess: true to ignore checkpoints and reprocess from scratch. Instructions Initial setup When a user wants to run this pipeline, walk through these steps: 1. Input JSONL: Ask for the JSONL path. Each line must be one object like {"imagepath": "...", "caption": "..."}. imagepath can be absolute or relative. 2. Image root: If any imagepath values are relative, set data.imageroot to the directory they should resolve from. 3. API access: Ask the user which VLM endpoint they want to use. Present these five options and act on the choice: 1. Gemini — set vlm.backend: "gemini"; require GOOGLEAPIKEY (env var or vlm.gemini.apikey). 2. NIM (e.g. https://inference-api.nvidia.com/v1) — set vlm.backend: "openai"; collect baseurl, modelname, and apikey. 3. TAO inference microservice (self-hosted, OpenAI-compatible). Confirm whether the server is already running: Running — collect baseurl, modelname, and (optionally) apikey; set vlm.backend: "openai". …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills NVIDIA/tao-generate-image-grounding