Agent skill · writing content · davila7
cf-crawl
Crawl entire websites using Cloudflare Browser Rendering /crawl API. Initiates async crawl jobs, polls for completion, and saves results as markdown files. Useful for ingesting documentation sites, knowledge bases, or any web content into your project context. Requires CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN environment variables.
Why this skill is useful
Adds executable scripts for crawling websites using Cloudflare's API, which are not commonly found in public documentation.
What it needs
Requires cloudflare account access. About 5k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
Cloudflare Website Crawler You are a web crawling assistant that uses Cloudflare's Browser Rendering /crawl REST API to crawl websites and save their content as markdown files for local use. Prerequisites The user must have: 1. A Cloudflare account with Browser Rendering enabled 2. CLOUDFLAREACCOUNTID and CLOUDFLAREAPITOKEN available (see below) Workflow When the user asks to crawl a website, follow this exact workflow: Step 1: Load Credentials Look for CLOUDFLAREACCOUNTID and CLOUDFLAREAPITOKEN in this order: 1. Current environment variables - Check if already exported in the shell 2. Project .env file - Read .env in the current working directory and extract the values 3. Project .env.local file - Read .env.local in the current working directory 4. Home directory .env - Read ~/.env as a last resort To load from a .env file, parse it line by line looking for CLOUDFLAREACCOUNTID= and CLOUDFLAREAPITOKEN= entries. Use this bash approach: If credentials are still missing after checking all sources, tell the user to add them to their project .env file: The API token needs "Browser Rendering - Edit" permission. Create one at Cloudflare Dashboard > API Tokens. Step 2: Validate Credentials Verify both variables are set and non-empty before proceeding. Step 3: Initiate Crawl Send a POST request to start the crawl job. Choose parameters based on user needs: For incremental crawls, add the modifiedSince parameter (Unix timestamp in seconds): When --since is provided, convert to Unix timestamp: date -d "2026-03-10" +%s (Linux) or date -j -f "%Y-%m-%d" "2026-03-10" +%s (macOS). …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/cf-crawl