NextBento logoNextBento
Get NextBento

Documentation

Guides and references

SEO

Cloudflare Crawler Hints Setup

Enable Cloudflare Crawler Hints so search engines learn about content changes faster. Add your site, update nameservers, and turn Crawler Hints on.

Why Use Cloudflare

Cloudflare provides:

  • DNS — Fast, reliable DNS resolution
  • CDN — Caching and edge delivery for better performance
  • SSL/TLS — Free HTTPS and certificate management
  • Bot management — Control crawler and bot traffic
  • Crawler Hints — Tell search engines when content has changed so they crawl more efficiently

Add a Site

  1. Log in to the Cloudflare Dashboard
  2. Click Add a domain (or Add a site)
  3. Enter your domain (e.g. example.com) and click Continue
  4. Choose a plan (Free is sufficient for most sites)
  5. Review the DNS records Cloudflare detected — add any missing records (e.g. apex, www, email)
  6. Click Continue to complete onboarding
Cloudflare — add a domain
Prerequisites

Disable DNSSEC at your domain registrar before changing nameservers to avoid connectivity errors.

Update Nameservers

After adding your domain, Cloudflare assigns two nameservers (e.g. ada.ns.cloudflare.com and bob.ns.cloudflare.com).

  1. Go to your domain registrar (where you bought the domain)
  2. Find the DNS or nameserver settings
  3. Replace the existing nameservers with the two Cloudflare nameservers
  4. Save changes

Propagation can take up to 24–48 hours. Cloudflare will email you when the domain is active.

Enable Crawler Hints

Crawler Hints tells search engines when content has changed so they can crawl more efficiently instead of guessing. When Cloudflare serves a cache MISS (content not in cache or freshly updated), it signals that the URL may have changed.

Steps

  1. In the Cloudflare dashboard, select your domain
  2. Go to Caching → Configuration
  3. Find Crawler Hints and toggle it On
Cloudflare Caching → Crawler Hints
Availability

Crawler Hints is available on all plans (Free, Pro, Business, Enterprise).

How It Works

  • Cache MISS — Cloudflare detects that content was not served from cache (e.g. new or updated page)
  • Hints sent — Cloudflare notifies search engines about the change
  • IndexNow — Crawler Hints integrates with IndexNow, so Bing, Yandex, and other engines can discover updates faster

Crawler Hints + NextBento

NextBento includes IndexNow integration. You can run pnpm submit:sitemap to notify search engines when you publish or update content.

  • IndexNow — Proactive: you explicitly submit URLs when content changes
  • Crawler Hints — Passive: Cloudflare automatically signals cache misses to search engines

Using both gives you the best coverage: manual updates for important changes and automatic hints for cache-related content changes.

Optional: Markdown for Agents (Cloudflare Pro)

This site already serves Markdown at the origin when a client sends Accept: text/markdown (see AI Search Discoverability). If you also enable Markdown for Agents in AI Crawl Control, Cloudflare can convert HTML at the edge for zones on Pro or higher. Origin negotiation still works on Free plans and on Vercel previews.

Optional: Crawler Analytics

To view crawler traffic on your site:

  • Bot Analytics — Security → Bots. Shows traffic by bot type and allows blocking or challenging specific bots
  • AI Crawl Control — For AI crawlers (e.g. AI training bots). Available in the dashboard for domains that proxy traffic through Cloudflare
Note

Crawler analytics are observational; you do not "enable" them. They appear once your site is proxied through Cloudflare.

Best Practices

  • Proxy main site records — Use the orange cloud (Proxied) for A, AAAA, or CNAME records that serve your website so traffic goes through Cloudflare
  • Keep email records DNS-only — Use the grey cloud for MX, TXT, and other email-related records to avoid breaking mail delivery
  • Verify SSL mode — Use Full or Full (strict) when your origin has a valid certificate
  • Do not block good bots — Avoid blocking Googlebot, Bingbot, or other crawlers unless you have a specific reason
  • Turn off Email Address Obfuscation — see below

Email Address Obfuscation

Cloudflare enables this by default. It rewrites mailto: and on-page emails to /cdn-cgi/l/email-protection. Those URLs 404 for crawlers (Ahrefs “links to broken page”, Search Console coverage).

This marketing site already:

  • Wraps HTML in <!--email_off--> so Cloudflare skips the rewrite
  • Disallows /cdn-cgi/ in robots.txt

Still turn the feature Off in the dashboard so the edge does not rewrite emails if a page is served without those comments:

  1. Cloudflare dashboard → your domain → Security → Settings
  2. Filter by Client-side abuse if needed
  3. Email Address Obfuscation → Off

Docs: Email Address Obfuscation

Next Steps

This guide is part of the NextBento SaaS boilerplate. If you are setting up a new product, start with Quick Start. To see how NextBento compares on price and SEO tooling, open pricing or boilerplate comparisons.

References

PreviousGoogle Search ConsoleNextSEO Best Practices