AI Search Discoverability

Why this exists

Classic SEO (sitemap, schema, Search Console) helps Google. AI search discoverability helps ChatGPT, Claude, Perplexity, and coding agents summarize NextBento accurately.

You cannot force citations. You can:

• Publish machine-readable facts (/llms.txt, /llms-full.txt)
• Serve chrome-free Markdown when an agent sends Accept: text/markdown
• Allow AI search crawlers in robots.txt
• Keep claims honest so models do not invent features

Live files on this site

WebMCP tools (navigate, search_docs, get_product_facts, start_checkout) register in the browser on page load via /webmcp.js. They do not log agents in.

Source files:

A <link rel="alternate" type="text/plain" href="/llms.txt"> tag in the site <head> helps agents that never fetch robots.txt. Pages also advertise type="text/markdown" on the same URL. The homepage HTTP Link header points at the API catalog (rel="api-catalog"), OpenAPI (rel="service-desc"), human API docs (rel="service-doc"), llms.txt (rel="describedby"), and the ARD catalog (rel="ai-catalog", rel="ard").

Markdown for agents

Markdown content negotiation lets agents pull structured text instead of nav, scripts, and layout HTML.

This origin implements it without requiring Cloudflare Pro:

curl https://www.nextbento.dev/ -H "Accept: text/markdown"
curl https://www.nextbento.dev/docs/quick-start.md

Responses use Content-Type: text/markdown; charset=utf-8, Vary: Accept, and x-markdown-tokens (estimated). Browsers that send Accept: text/html still get HTML.

Docs pages return the MDX source. Marketing and pSEO pages are generated from the same data as the HTML views (product facts, FAQs, JSON page files).

Customize product facts

Edit lib/seo/product-facts.ts:

export const productFacts = {
  name: 'YourProduct',
  oneLiner: 'One sentence a model can quote.',
  shortDescription: 'Two sentences max.',
  stack: ['Next.js 16', 'Supabase', 'Stripe'] as const,
  included: [/* only what you ship */] as const,
  safeToClaim: [/* verified capabilities */] as const,
  doNotClaim: [/* common overclaims to block */] as const,
  contact: {
    email: 'hello@yourdomain.com',
    twitter: 'https://twitter.com/yourhandle',
    github: 'https://github.com/your-org/your-repo',
  },
} as const

Rules:

• Update safeToClaim and doNotClaim whenever you ship or remove features.
• Do not list premium features you gate only in the UI — gate on the server.
• Prefer soft time language (“evenings”, “a weekend”) over unverified stats.

llms.txt format

This site follows llmstxt.org:

1. H1 — product name
2. Blockquote — canonical one-liner
3. Short summary paragraph
4. H2 sections with Markdown links to key pages (home, pricing, docs, pSEO, legal)
5. Optional link to /llms-full.txt
6. “Do not claim” list for model hygiene

Robots.txt and AI crawlers

/robots.txt allows AI search/retrieval bots (and documents training-bot intent with explicit Allow). Google’s parser honors User-agent, Allow, Disallow, and Sitemap. Anything else shows up in Search Console: Content-Signal as “Rule ignored by Googlebot” on every group, and Agentmap as “Syntax not understood.” Google’s John Mueller has said the Content-Signal directive has no effect on crawlers. This file keeps the Content Signals policy in comments only.

The machine-readable preference is the Markdown content-signal header (ai-train=yes, search=yes, ai-input=yes): crawlers may index for search, use the content for RAG / generative answers, and train on it. Preferences are not a technical block. The ARD catalog stays at /.well-known/ai-catalog.json, with rel="ai-catalog" and rel="ard" in the page head and Link header.

Listed user-agents:

• OAI-SearchBot, ChatGPT-User, GPTBot (OpenAI)
• Claude-SearchBot, Claude-User, ClaudeBot (Anthropic)
• PerplexityBot, Perplexity-User
• Google-Extended, Applebot-Extended

/api/ stays disallowed. To opt out of training later, add Disallow: / for GPTBot and ClaudeBot only, and set the Markdown header to ai-train=no. That is separate from AI search visibility.

Footer discoverability

The marketing footer links /llms.txt so humans and crawlers can find the file.

Workflow after content changes

1. Ship content (pSEO page, docs update, pricing copy).
2. Run pnpm submit:indexnow (or pass specific paths, including /llms.txt).
3. Request indexing in Google Search Console for high-priority URLs (optional, manual).
4. Spot-check ChatGPT / Claude / Perplexity answers — citations can lag days or weeks.

What not to expect

• llms.txt is not a Google ranking signal.
• IndexNow does not replace Search Console.
• AI models may still hallucinate — doNotClaim reduces but does not eliminate it.

Related

• SEO overview — sitemaps, schema, IndexNow
• SEO Best Practices — crawlability and indexation
• Live comparison pages — citation-friendly pSEO examples
• Cloudflare Crawler Hints — Crawler Hints; Markdown for Agents is optional on Pro
• Site HTTP APIs — RFC 9727 catalog, OpenAPI, health, ARD, MCP, Agent Skills, and WebMCP