robots.txt · Next.js
Next.js robots.txt
The App Router does not want a file in public/. You export MetadataRoute.Robots from app/robots.ts, set Host and sitemaps, Disallow api, auth, and admin, and Allow GPTBot unless you are opting out of training. Here is the module, the rendered file, and the Cloudflare caveat that silently overrides it.
import type { MetadataRoute } from 'next';
export default function robots(): MetadataRoute.Robots {
return {
rules: [
{
userAgent: '*',
allow: '/',
// MetadataRoute paths are slash-prefixed. Written here without the
// leading slash so crawlers do not treat the examples as URLs on this host.
disallow: ['api/', 'auth', 'admin/'],
},
{ userAgent: 'GPTBot', allow: '/' },
{ userAgent: 'OAI-SearchBot', allow: '/' },
{ userAgent: 'ChatGPT-User', allow: '/' },
{ userAgent: 'Google-Extended', allow: '/' },
{ userAgent: 'Claude-SearchBot', allow: '/' },
],
sitemap: 'https://example.com/sitemap.xml',
host: 'https://example.com',
};
}Host and Sitemap use https://example.com so they are not this site's URLs. Disallow values are written without a leading slash for the same reason — add the slash in your real MetadataRoute object.
What MetadataRoute.Robots actually emits
Next.js turns the object into a robots.txt document. This is the same file a crawler fetches, shown against a dummy host.
# https://example.com/robots.txt
User-agent: *
Allow: /
Disallow: api/
Disallow: auth
Disallow: admin/
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Claude-SearchBot
Allow: /
Host: https://example.com
Sitemap: https://example.com/sitemap.xmlapp/robots.ts → MetadataRoute.RobotsApp Router generates /robots.txt from a TypeScript module, not a static file.
Export a default function that returns MetadataRoute.Robots. Next serializes it on request. This replaces dropping a file in public/ — and if both exist, the public file wins, so delete the static copy once you move to robots.ts.
User-agent: * · Allow: /Opens the site to every crawler that does not have its own group.
A Next.js marketing site should be crawlable by default. Named bot groups below do not inherit these rules — a crawler that matches GPTBot uses only the GPTBot group.
Disallow: api/, auth, admin/Keeps route handlers, the login screen, and the admin shell out of the crawl.
api/ is Route Handlers and server endpoints, not pages. auth is the sign-in UI. admin/ is the authenticated app. None of these should be indexed. Add the leading slash when you paste into MetadataRoute — the examples here omit it so Google does not invent those paths as URLs on this host.
Host: https://example.comTells crawlers which host is canonical for this robots.txt.
Useful when www and apex both serve the app. Host is not in the original robots.txt spec; Google honours it, other crawlers may ignore it. Always write it as an absolute URL on your real domain.
Sitemap: https://example.com/sitemap.xmlPoints crawlers at the XML sitemap the App Router can also generate (app/sitemap.ts).
A sitemap is how Google discovers URL inventory without guessing. Use an absolute URL. Next accepts a string or an array if you also ship an agent or news sitemap.
How to add robots.txt in the App Router
Three rules. Miss any one of them and you will think Next.js ignored your file.
Create app/robots.ts (or robots.js)
In the App Router the file lives next to app/layout.tsx. TypeScript is the usual choice. There is no theme editor and no dashboard field — this is source code, so it ships when you deploy.
Do not also keep public/robots.txt
A file in public/ is served as a static asset and takes precedence over the MetadataRoute module. Pages Router apps often started that way. If you migrate to app/robots.ts, delete the public copy or you will debug the wrong file.
Deploy, then fetch the live URL
robots.ts is compiled into the production server. A local change does not affect production until the deploy finishes. Confirm by requesting https://example.com/robots.txt after release, not by staring at the source tree.
AI crawler rules for Next.js
MetadataRoute.Robots lets you add named user-agent groups. Allow GPTBot, OAI-SearchBot, ChatGPT-User, Google-Extended, and Claude-SearchBot unless you are deliberately blocking training. What each bot does is on our AI crawlers reference →
# Recommended: let AI crawlers in so you can be cited
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Claude-SearchBot
Allow: /Cloudflare managed robots.txt is a separate file that gets prepended in front of origin. If that setting is on, Cloudflare's Disallow groups for GPTBot or Google-Extended win, and the Allow groups in app/robots.ts never apply. Check the live response, not only the module.
The Next.js-specific mistakes
Each of these is something the App Router (or the CDN in front of it) makes easy to get wrong.
Leaving a static public/robots.txt beside app/robots.ts
The static file wins. You will edit robots.ts, deploy, reload /robots.txt, and see the old rules. One file, one source of truth.
Disallowing the Next.js asset prefix
Blocking the _next/static prefix (no leading slash in this sentence on purpose) stops Google from fetching CSS and JS. Pages then render as unstyled HTML. Leave compiled assets crawlable.
Using Disallow to take a page out of Google
Disallow stops crawling, not indexing. A blocked URL with inbound links can still appear as a bare title. To remove a page, keep it crawlable and send noindex — robots.ts cannot do that job.
Assuming origin robots.txt beats Cloudflare managed robots
If Cloudflare managed robots.txt is on, Cloudflare prepends its own User-agent groups before whatever Next.js served. The first matching group wins. An origin Allow: / for GPTBot does nothing while a prepended Disallow: / for that bot is in place. Turn the managed file off, or accept that Cloudflare owns those groups.
Build the file instead of copying one
The robots.txt generator takes your URL and produces a file with the right sitemap directive and crawl rules, as part of a free strategy run.
Open the robots.txt generatorOr let the agent ship it
Rankealo's SEO agent can open a pull request with the robots.txt fix already written, so a crawl-blocking rule gets reviewed and merged instead of sitting in a report.
See how the SEO agent worksrobots.txt on another platform?
Next.js robots.txt questions
How do you set robots.txt in Next.js?
In the App Router, add app/robots.ts and export a default function that returns MetadataRoute.Robots. Next.js serves that module as /robots.txt. You can also put a static robots.txt in public/, but do not use both — the public file overrides robots.ts. There is one canonical slug for this guide (nextjs); next-js and next.js URLs redirect here.
What is next.js robots.ts?
robots.ts is the App Router module that generates robots.txt. You type a MetadataRoute.Robots object: rules (userAgent, allow, disallow), plus optional host and sitemap. Next serializes it. That is the nextjs robots txt file — there is usually nothing on disk at the web root.
What should a Next.js robots.txt disallow?
For most apps: api (Route Handlers), auth (sign-in), and admin (authenticated UI). Do not block compiled assets under the _next prefix. Product and marketing pages stay allowed. Write Disallow paths with a leading slash in the real MetadataRoute object; this page shows them without one so they are not crawled as URLs here.
Should I block GPTBot, OAI-SearchBot, ChatGPT-User, Google-Extended, or Claude-SearchBot?
Allow them unless you have a specific reason to block training. GPTBot and Google-Extended are the training-adjacent tokens; OAI-SearchBot, ChatGPT-User, and Claude-SearchBot are how those products fetch pages for answers and citations. Blocking the search bots removes you from AI answers. If you only want to opt out of training, disallow GPTBot and Google-Extended and keep the search bots allowed. See the AI crawlers reference for what each token does.
Why does Cloudflare serve a different robots.txt than my Next.js app?
Cloudflare managed robots.txt prepends User-agent groups onto the origin file. Those prepended groups are evaluated first, so a Cloudflare Disallow for GPTBot wins over an Allow in app/robots.ts. Check the live /robots.txt response, not only the source module, and disable managed robots if you want origin to be the only author.
Do I need Host and Sitemap in Next.js robots.ts?
Sitemap yes — point it at the absolute URL of app/sitemap.ts (or your sitemap index). Host is optional: set it to your canonical origin when www and apex both resolve. Both fields take absolute URLs, like https://example.com/sitemap.xml.