How Docs Lens reads your docs.

Coding agents (Claude Code, Cursor, Continue) try the .md variant first to skip Turndown. If you only serve .html they have to convert lossy.

Agent-to-Agent (A2A) discovery lets other agents find your service as a callable agent and learn its capabilities programmatically. Optional unless your product surfaces agent-like functionality.

Modern agent fetchers (Claude Code as of v2.1.105) send Accept: text/markdown. If your server ignores it and returns HTML anyway, you lose the markdown shortcut.

Skills are reusable agent capabilities scoped to your product. The well-known endpoint is how agents find them without manual config.

When auth is non-negotiable, give agents a path: a public mirror, an anonymous-read endpoint, or a documented API key the user can configure.

Auth-gated docs are invisible to every agent. If a portion needs to be gated, expose an auth-free public version or robots-index allowed subset.

Cache validators let agents skip the body when nothing changed (304 Not Modified). Without them, every fetch refetches the full response, which is wasteful for the agent's budget and your bandwidth.

Link headers (RFC 8288) are the cheapest way for agents to discover capabilities, since they ride on every response with no extra fetch needed. Use them to advertise llms.txt, api-catalog, service-doc, and other discovery endpoints.

Agents trust status codes. A page that returns 200 but says 'not found' wastes the agent's budget on dead content.

Even if /llms.txt exists, agents only know to look for it if your homepage points to it (`<link rel="llms.txt" href="/llms.txt">` or an HTTP `Link` header).

Agents parse llms.txt as a structured manifest. Malformed lines cause the entire file to be discarded silently, undoing the discoverability win of having one.

A stale llms.txt that lists removed pages or misses new ones is worse than none, since agents trust the manifest and stop looking elsewhere. Wire generation into your build pipeline so it can never drift.

Each entry in llms.txt is a fetch budget the agent will spend. Broken URLs waste that budget and can cause agents to abandon the manifest entirely.

llms.txt is supposed to point at machine-readable markdown. Linking HTML pages forces agents to do their own conversion, which loses code blocks and tables.

An /llms.txt file (per the llmstxt.org spec) gives coding agents a manifest of your documentation. Without it, agents like Claude Code and Cursor have no shortcut to your structured docs and must crawl the whole site to find anything.

Tighter agent fetchers (MCP defaults to 5 KB, Cursor WebFetch to 28 KB, Claude Code truncates at 100 KB) only see the opening of an oversized llms.txt. Use the progressive-disclosure pattern: a small root file pointing to /docs/section/llms.txt files.

MCP-aware agents check /.well-known/mcp.json for capability metadata. Without it, even users who would benefit from your MCP server never get pointed to it.

OAuth/OIDC discovery metadata lets coding agents programmatically obtain access tokens. Without it, the agent has to be hand-configured with your auth endpoints, and most won't bother.

RFC 9728 metadata describing your protected resource. Coding agents read this to discover which OAuth issuer they need to talk to before calling your API.

JS or meta-refresh redirects break for agents that don't render. Server-side 3xx redirects work for everyone.

Agents that don't run JavaScript (Claude Code WebFetch, Continue, Aider default) only see the initial HTML. CSR-only docs look empty to them.

Without explicit User-agent rules, AI crawlers fall back to platform defaults that vary widely and may not match your intent. Naming each crawler explicitly puts you in control of who can crawl, who can train, and who can answer-engine-cite.

An api-catalog points agents at your API definitions. Without it, agents either guess paths or skip your API entirely.

The HTTP Link header is a lower-effort cousin of well-known endpoints. Even if you skip the file, the header tells agents where to look.

Search-snippet readers (ChatGPT Search, Perplexity, You.com) only see your title and description. Missing metadata means missing citations.

If robots.txt blocks agent crawlers, you're invisible to ChatGPT, Perplexity, and Claude.ai search even if your docs are otherwise perfect.

A sitemap is the canonical index for crawlers and answer engines. Without one, ChatGPT Search, Perplexity, and Google AI Overviews fall back to following links from the homepage and miss pages that aren't navigable from there.

Answer engines like Perplexity heavily favor structured-data-rich pages because the citation surface is more reliable.

Web Bot Auth (RFC 9421) lets origin servers cryptographically verify that an inbound request really comes from a declared bot rather than a spoofed user-agent string. The signing-keys directory is the discovery endpoint.

Code blocks without language tags lose their type when converted to markdown, hurting both rendering and agent parsing of examples.

Many agent extractors heuristically prioritize early content. Pages where the actual answer starts past the halfway mark often get truncated before reaching it.

Agents use heading structure to chunk pages. Skipping levels (H1 → H4) or having multiple H1s makes chunking unreliable.

Headless agent fetchers cap fetch size. A page that's mostly nav and JS exhausts the budget before reaching the content.

Diagram images without alt text are invisible to non-vision agents. Even short alts ('Architecture: API → DB → Cache') help.

Broken internal links waste agent fetch budget and cause incomplete answers when a referenced section can't be reached.

Agents often try to JSON.parse() example payloads. Invalid samples cause silent failures or fallback to text-only answers.

Claude Code's pipeline truncates at 100KB of markdown before any further processing. Long pages get cut mid-content.

JS-only tab widgets are invisible to raw HTTP fetchers. Either render all tab content in the source HTML, or serve a single linear version for non-JS readers.