Agent skill · research science · davila7
blip-2-vision-language
Vision-language pre-training framework bridging frozen image encoders and LLMs. Use when you need image captioning, visual question answering, image-text retrieval, or multimodal chat with state-of-the-art zero-shot performance.
Why this skill is useful
Provides domain-specific knowledge and code examples for implementing vision-language tasks using BLIP-2, which the AI may not generate reliably on its own.
What it needs
Requires pillow, torch, transformers installed locally. About 7k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
BLIP-2: Vision-Language Pre-training Comprehensive guide to using Salesforce's BLIP-2 for vision-language tasks with frozen image encoders and large language models. When to use BLIP-2 Use BLIP-2 when: Need high-quality image captioning with natural descriptions Building visual question answering (VQA) systems Require zero-shot image-text understanding without task-specific training Want to leverage LLM reasoning for visual tasks Building multimodal conversational AI Need image-text retrieval or matching Key features: Q-Former architecture: Lightweight query transformer bridges vision and language Frozen backbone efficiency: No need to fine-tune large vision/language models Multiple LLM backends: OPT (2.7B, 6.7B) and FlanT5 (XL, XXL) Zero-shot capabilities: Strong performance without task-specific training Efficient training: Only trains Q-Former (~188M parameters) State-of-the-art results: Beats larger models on VQA benchmarks Use alternatives instead: LLaVA: For instruction-following multimodal chat InstructBLIP: For improved instruction-following (BLIP-2 successor) GPT-4V/Claude 3: For production multimodal chat (proprietary) CLIP: For simple image-text similarity without generation Flamingo: For few-shot visual learning Quick start Installation Basic image captioning Visual question answering Using LAVIS library Core concepts Architecture overview Model variants Model LLM Backend Size Use Case ------- ------------- ------ ---------- blip2-opt-2.7b OPT-2.7B ~4GB General captioning, VQA blip2-opt-6.7b OPT-6.7B ~8GB Better reasoning blip2-flan-t5-xl FlanT5-XL ~5GB Instruction following blip2-flan-t5-xxl FlanT5-XXL ~13GB Best quality Q-Former components Component Description Parameters ----------- ------------- ------------ Learned queries Fixed set of learnable embeddings 32 × 768 Image transformer Cross-attention to vision features ~108M Text transformer Self-attention for text ~108M Linear projection Maps to LLM dimension Varies Advanced usage Batch processing Co …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/multimodal-blip-2