Agent skill · data analytics · alirezarezvani
universal-scraping-architect
Build a scraper or extractor for sites, PDFs or paginated APIs and validate the fields (Firecrawl or local Python). Load whenever the user wants data pulled from pages, files or an API, or a scraper broke.
Why this skill is useful
Adds specific Python scripts for data extraction and validation that are not commonly found in public documentation.
What it needs
Requires beautifulsoup4, firecrawl, pandas, requests installed locally. About 3k tokens when loaded. Last updated 2026-08-05. 23,939 stars on the source repository.
What this skill does
Universal Scraping Architect Design complete, robust data-extraction pipelines with intelligent routing, validation, and token-budget tracking — not brittle one-off scripts. Dependency Notice: BYOK (Bring Your Own Key) pattern for Firecrawl; API keys must only be loaded via environment variables. Per-script dependencies: Script Dependencies Exact CLI --- --- --- scripts/validateextraction.py stdlib only python3 scripts/validateextraction.py output.json --json scripts/firecrawlexample.py firecrawl, requests (template; --sample runs offline) python3 scripts/firecrawlexample.py --sample scripts/localbs4example.py beautifulsoup4, pandas (template; --sample runs offline) python3 scripts/localbs4example.py --sample Before Starting Check for context first: If project-context.md exists, read it before asking questions. Determine the target data format, scale of extraction, and deployment environment before writing any code. How This Skill Works This skill supports 3 extraction modes based on intelligent routing: Mode 1: API-Driven (Firecrawl) Use when the source is a public URL, heavily dynamic (JS/SPA), requires search-first discovery, or involves bulk crawling across a domain. Mode 2: Local Python (Traditional) Use when extracting from local files (PDF, Excel, CSV), the data is private/sensitive, or the target is a simple static HTML page where Firecrawl is overkill. Mode 3: Hybrid Pipeline Use when Firecrawl handles URL discovery/web extraction, but local Python (Pandas) is required to clean, normalize, and structure the output before saving. The Extraction Pipeline When executing a scraping task, always follow this sequence: 1. Route the Approach: Explicitly state whether Firecrawl or Local Python is being used and why. 2. Track Budgets: Estimate Firecrawl API quotas or LLM token context limits before executing large jobs. 3. Extract Safely: Implement checkpointing for multi-page jobs. Handle pagination and dynamic layouts gracefully. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills alirezarezvani/universal-scraping-architect--144ec6