Agent skill · marketing growth · kostja94
robots-txt
When the user wants to configure, audit, or optimize robots.txt. Also use when the user mentions "robots.txt," "crawler rules," "block crawlers," "AI crawlers," "GPTBot," "allow/disallow," "disallow path," "crawl directives," "user-agent," "block Googlebot," "fix robots.txt," "robots.txt blocking," or "search engine crawling." For indexing, use indexing.
Why this skill is useful
Provides specific guidelines and best practices for configuring and auditing robots.txt files that the AI wouldn't reliably generate on its own.
What it needs
About 4k tokens when loaded. Last updated 2026-06-09. 860 stars on the source repository.
What this skill does
SEO Technical: robots.txt Guides configuration and auditing of robots.txt for search engine and AI crawler control. When invoking: On first use, if helpful, open with 1–2 sentences on what this skill covers and why it matters, then provide the main output. On subsequent use or when the user asks to skip, go directly to the main output. Scope (Technical SEO) Robots.txt: Configure Disallow/Allow, Sitemap, Clean-param; audit for accidental blocks Crawler access: Path-level crawl control; AI crawler allow/block strategy Differentiation: robots.txt = crawl control (who accesses what paths); noindex = index control (what gets indexed). See indexing for page-level exclusions. Initial Assessment Check for project context first: If .claude/project-context.md or .cursor/project-context.md exists, read it for site URL and indexing goals. Identify: 1. Site URL: Base domain (e.g., https://example.com) 2. Indexing scope: Full site, partial, or specific paths to exclude 3. AI crawler strategy: Allow search/indexing vs. block training data crawlers Best Practices Purpose and Limitations Point Note ------- ------ Purpose Controls crawler access; does NOT prevent indexing (disallowed URLs may still appear in search without snippet) Advisory Rules are advisory; malicious crawlers may ignore Public robots.txt is publicly readable; use noindex or auth for sensitive content. See indexing Crawl vs Index vs Link Equity (Quick Reference) Tool Controls Prevents indexing? ------ ---------- ------------------- robots.txt Crawl (path-level) No—blocked URLs may still appear in SERP noindex (meta / X-Robots-Tag) Index (page-level) Yes. See indexing nofollow Link equity only No—does not control indexing When to Use robots.txt vs noindex Use Tool Example ----- ------ --------- Path-level (whole directory) robots.txt Disallow: /admin/, Disallow: /api/, Disallow: /staging/ Page-level (specific pages) noindex meta / X-Robots-Tag Login, signup, thank-you, 404, legal. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills kostja94/robots