Agent skill · research science · wshobson
vision-sft
Fine-tune vision-language models (VLMs) with supervised learning on image+text data. Use when adapting a VLM to a visual domain or task, configuring frozen-vision-tower LoRA, or debugging a VLM fine-tune that trains without learning.
Why this skill is useful
Adds executable scripts for fine-tuning vision-language models with specific configurations that aren't commonly documented.
What it needs
Requires torch, transformers installed locally. About 5k tokens when loaded. Last updated 2026-08-05. 38,570 stars on the source repository.
What this skill does
Vision-Language SFT This skill assumes finetuning-method-selection already routed here: the data shape is image+text demonstrations, not preference pairs or a verifiable reward signal, and the base is a vision-language model rather than a text-only one. lora-qlora-recipes covers the text-only LoRA/QLoRA recipe this skill specializes for the vision tower and projector; read that skill first if the LoRA fundamentals (rank, alpha, target modules) aren't already familiar. Input: an image+text dataset and a VLM base model already picked from the model catalog. Output format: a validated adapter config — which components are frozen, LoRA target modules, and a minpixels/maxpixels budget — that llm-finetuning-training-engineer consumes directly when it generates a runnable script. Quick Reference Situation Default --- --- Adapting behavior on familiar images Frozen tower+projector, LoRA r=8–16, α=16–32 Visual domain shift Unfreeze last-6 ViT layers, vision LR 5–10x lower Doesn't fit in bf16 at target rank QLoRA — frozen vision tower only fastinference=True finetunevisionlayers=False Loss normal, eval not improving Check the Two Silent Killers below first The Consensus Recipe Freeze the vision tower and the projector. Put LoRA on the LLM only, all-linear (the same attention + MLP target list as text-only SFT — see lora-qlora-recipes), at r=8–16, α=16–32. This is the settled default for adapting a VLM's behavior without disturbing how it sees. The vision tower and projector stay frozen by default. They already encode a general visual representation; retraining them is rarely necessary and adds risk without adding capability for most tasks. LoRA rank runs lower than the text-only general default (r=8–16 here vs r=16–32 for text-only SFT) because the LLM-only adapter is adapting behavior, not injecting new visual knowledge. QLoRA is permitted only with a frozen vision tower. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills wshobson/vision-sft