What is llms.txt? Definition for AI Crawlers
llms.txt is a proposed convention: a markdown file at /llms.txt that tells language models what a site is and which URLs matter. It is not a Google ranking signal.
Quick Answer
llms.txt is a markdown index at the site root (and optionally /llms-full.txt) so AI agents can read a short, claim-safe summary instead of scraping layout HTML. Crawlers that support the convention fetch it the same way they fetch robots.txt — you do not submit it to Google.
Definition
The llmstxt.org convention (Jeremy Howard / Answer.AI) asks sites to publish /llms.txt as markdown: an H1 name, a one-liner, a short summary, then links to important pages. An optional /llms-full.txt expands stack, inclusions, and things models should not claim.
How it differs from robots.txt and sitemaps
robots.txt tells crawlers what they may fetch. sitemap.xml lists URLs for search engines. llms.txt is a table of contents for models: what the product is, which docs to cite, and which claims to avoid. Google has said extra machine-readable files like this are not a ranking bonus.
What NextBento publishes
www.nextbento.dev/llms.txt is the compact index. /llms-full.txt adds stack, inclusions, and a do-not-claim list from product facts. robots.txt allows OpenAI, Anthropic, and Perplexity search bots. Implementation details live in the AI search discoverability docs, not on this definition page.
Decision Criteria
Publish llms.txt if you care about accurate AI summaries and you can keep the file honest. Skip it if you have nothing stable to say. Do not treat a directory listing of llms.txt files as a substitute for the file on your own origin.
Best For / Not Ideal For
Best for product sites that want agents to quote a canonical price, stack, and feature list. Not ideal as an SEO hack or as a way to force ChatGPT citations.
Evidence and Last Verified
Last verified: 2026-09-02. Spec: llmstxt.org. NextBento files: www.nextbento.dev/llms.txt and /llms-full.txt. Setup: www.nextbento.dev/docs/ai-search-discoverability.
