Agent skill · data analytics · aiskillstore
crawl4ai
This skill should be used when users need to scrape websites, extract structured data, handle JavaScript-heavy pages, crawl multiple URLs, or build automated web data pipelines. Includes optimized extraction patterns with schema generation for efficient, LLM-free extraction.
Why this skill is useful
Provides optimized extraction patterns and schema generation for web data pipelines that the AI wouldn't reliably generate on its own.
What it needs
Requires crawl4ai installed locally. About 4k tokens when loaded. Last updated 2026-08-07. 405 stars on the source repository.
What this skill does
Crawl4AI Overview Crawl4AI provides comprehensive web crawling and data extraction capabilities. This skill supports both CLI (recommended for quick tasks) and Python SDK (for programmatic control). Choose your interface: CLI (crwl) - Quick, scriptable commands: CLI Guide Python SDK - Full programmatic control: SDK Guide --- Quick Start Installation CLI (Recommended) Python SDK For SDK configuration details: SDK Guide - Configuration (lines 61-150) --- Core Concepts Configuration Layers Both CLI and SDK use the same underlying configuration: Concept CLI SDK --------- ----- ----- Browser settings -B browser.yml or -b "param=value" BrowserConfig(...) Crawl settings -C crawler.yml or -c "param=value" CrawlerRunConfig(...) Extraction -e extract.yml -s schema.json extractionstrategy=... Content filter -f filter.yml markdowngenerator=... Key Parameters Browser Configuration: headless: Run with/without GUI viewportwidth/height: Browser dimensions useragent: Custom user agent proxyconfig: Proxy settings Crawler Configuration: pagetimeout: Max page load time (ms) waitfor: CSS selector or JS condition to wait for cachemode: bypass, enabled, disabled jscode: JavaScript to execute cssselector: Focus on specific element For complete parameters: CLI Config SDK Config Output Content Every crawl returns: markdown - Clean, formatted markdown html - Raw HTML links - Internal and external links discovered media - Images, videos, audio found extractedcontent - Structured data (if extraction configured) --- Markdown Generation (Primary Use Case) Crawl4AI excels at generating clean, well-formatted markdown: CLI Filter configuration: Python SDK For content filters: Content Processing (lines 2481-3101) --- Data Extraction 1. Schema-Based CSS Extraction (Most Efficient) No LLM required - fast, deterministic, cost-free. CLI: Schema format: 2. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills aiskillstore/crawl4ai--873592