Agent skill · software engineering · nvidia
tao-generate-image-grounding
Two-step image grounding pipeline: extracts referring expressions from (image, caption) pairs and grounds them
Why this skill is useful
Adds a complete image grounding pipeline with executable steps for extracting expressions and generating bounding boxes, which the AI cannot generate on its own.
What it needs
Requires docker, nvidia-container-toolkit installed locally. Requires gemini, openai account access. About 4k tokens when loaded. Last updated 2026-08-07. 2,816 stars on the source repository.
What this skill does
Image Grounding Pipeline Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery). Turn (image, caption) pairs into per-image grounded annotations: cleaned captions, referring expressions with character spans, and pixel-space bounding boxes for each expression. A single VLM (Gemini or any OpenAI-compatible endpoint) handles both steps. Purpose Generate phrase-grounded training data for referring-expression and grounding models. The VLM acts as a "teacher" annotator: Step 0 extracts referring expressions from the caption while looking at the image; Step 1 returns one bbox set per expression for each image. Pipeline Architecture Steps are individually selectable via workflow.steps. Each step writes a per-sample checkpoint to step<N>/.ckpt/<sampleid>.json and skips already-processed records on re-run. Set workflow.forcereprocess: true to ignore checkpoints and reprocess from scratch. Instructions Initial setup When a user wants to run this pipeline, walk through these steps: 1. Input JSONL: Ask for the JSONL path. Each line must be one object like {"imagepath": "...", "caption": "..."}. imagepath can be absolute or relative. 2. Image root: If any imagepath values are relative, set data.imageroot to the directory they should resolve from. 3. API access: Ask the user which VLM endpoint they want to use. Present these five options and act on the choice: 1. Gemini — set vlm.backend: "gemini"; require GOOGLEAPIKEY (env var or vlm.gemini.apikey). 2. NIM (e.g. https://inference-api.nvidia.com/v1) — set vlm.backend: "openai"; collect baseurl, modelname, and apikey. 3. TAO inference microservice (self-hosted, OpenAI-compatible). Confirm whether the server is already running: Running — collect baseurl, modelname, and (optionally) apikey; set vlm.backend: "openai". …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills nvidia/tao-generate-image-grounding