Agent skill · marketing growth · kostja94
site-crawlability
When the user wants to improve crawlability, fix orphan pages, or optimize site structure for search engines. Also use when the user mentions "crawlability," "crawl budget," "orphan pages," "internal links," "site structure," "site crawlability," "infinite scroll," "pagination," "masonry SEO," "AI crawler optimization," "GPTBot crawlability," "ClaudeBot crawlability," or "content not indexed." For internal links, use internal-links.
Why this skill is useful
Provides specific guidelines and best practices for optimizing site crawlability that the AI wouldn't reliably generate on its own.
What it needs
About 4k tokens when loaded. Last updated 2026-06-09. 860 stars on the source repository.
What this skill does
SEO Technical: Crawlability Guides crawlability improvements: robots, X-Robots-Tag, site structure, and internal linking. When invoking: On first use, if helpful, open with 1–2 sentences on what this skill covers and why it matters, then provide the main output. On subsequent use or when the user asks to skip, go directly to the main output. Scope (Technical SEO) Redirect chains & loops: Fix multi-hop redirects; point directly to final URL Broken links (4xx): Fix broken internal/external links; 301 or remove Site architecture: Logical hierarchy; pages within 3–4 clicks from homepage Orphan pages: Add internal links to pages with no incoming links Pagination: Prefer pagination over infinite scroll for crawlability Crawl budget: Reduce waste on duplicates, redirects, low-value URLs (see below) AI crawler optimization: SSR for critical content; URL management; reduce 404/redirect waste (see below) Initial Assessment Check for project context first: If .claude/project-context.md or .cursor/project-context.md exists, read it for site structure. Identify: 1. Site structure: Flat vs. deep hierarchy 2. Framework: Next.js, static, SPA, etc. 3. Key paths: Sitemap, robots.txt, API, static assets Best Practices Redirect Chains & Loops Fix multi-hop redirects; point directly to final URL Loops: URLs redirecting back to themselves; break the cycle Broken Links (4xx) Fix broken internal/external links; 301 or remove Audit regularly; update or remove broken links Site Architecture Principle Guideline ----------- ----------- Depth Important pages within 3–4 clicks from homepage Orphan pages Add internal links to pages with no incoming links; see internal-links for link strategy Hierarchy Logical structure; hub pages link to content Pagination vs Infinite Scroll Problem: With infinite scroll, crawlers cannot emulate user behavior (scroll, click "Load more"); content loaded after initial page load is not discoverable. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills kostja94/crawlability