robots.txt · Webflow
Webflow robots.txt, and what it leaves to you
Webflow's default robots.txt is almost empty — one sitemap line and no rules. That is deliberate, and mostly correct. The interesting parts are where you edit it, why your edit did not go live, and the staging subdomain quietly serving a second copy of your entire site.
Sitemap: https://example.com/sitemap.xmlUser-agent: *
Content-Signal: ai-train=yes, search=yes, ai-input=yes
Disallow: /admin/
Disallow: /template/
Disallow: /design/
Disallow: /dashboard/
Allow: /api/discover/sites/
Allow: /api/v1/marketplace/made-in-webflow/
Disallow: /api/
Sitemap: https://webflow.com/sitemap.xmlReading a Webflow robots.txt line by line
Since the default is nearly empty, the useful lesson is in what a real Webflow file looks like once someone has filled it in — including one directive most people have never seen.
No User-agent group at allA new Webflow site has an empty rule set.
There is nothing to block, so Webflow blocks nothing. Under the robots.txt spec, absence of a rule means allowed — every page is crawlable by default. For most Webflow marketing sites that is the correct starting point, and it means an empty-looking robots.txt is not a bug.
Sitemap: https://example.com/sitemap.xmlWebflow adds a link to your sitemap automatically.
This is the one thing Webflow puts in the file for you. There is a "Remove sitemap.xml from robots.txt" toggle in site settings if you need it gone — you generally do not, since the sitemap directive is how crawlers discover your URL inventory without relying on Search Console.
Allow: before a broader Disallow:Ordering does not decide the winner — specificity does.
In webflow.com's own file, two Allow rules for /api/ paths sit above a blanket Disallow: /api/. That works because Google resolves conflicts by the longest matching rule, not by file order. Useful when you want to open one folder inside an otherwise closed section.
Content-Signal: ai-train=yes, search=yes, ai-input=yesAn emerging, non-standard directive Webflow serves on its own site.
Content Signals is a newer convention for stating how your content may be used — trained on, indexed for search, or used as AI input — separately from whether it can be crawled. It is not part of the original robots.txt standard and crawlers are free to ignore it, so treat it as a declaration of intent rather than an enforcement mechanism.
What Webflow lets you edit
One field, one publish step, and one plan-gated switch. That is the whole surface area.
Site settings → SEO → Indexing
This is the only editor. Webflow has no filesystem you can drop a robots.txt into, so the text field here is the file. Type your User-agent and Disallow lines directly into it.
Changes go live on publish
Saving the field updates your settings, not your site. The file served at /robots.txt only changes when you publish. This is the single most common reason a Webflow robots.txt edit appears to do nothing.
The sitemap toggle
Webflow includes your sitemap link by default and gives you a "Remove sitemap.xml from robots.txt" switch. If you turn it off, make sure something else is telling crawlers where your sitemap lives.
The staging subdomain needs a paid plan
Every Webflow site also lives at a .webflow.io address. Disabling search engine indexing of that subdomain requires a paid Site plan or Workspace — on a free plan you cannot turn it off, which is worth knowing before you publish real content.
AI crawler rules for Webflow
Paste either block straight into the Indexing field in site settings, then publish. What each bot does is covered in our AI crawlers reference →
# Paste into Site settings → SEO → Indexing
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /# Only if you accept losing AI answer visibility
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Google-Extended
Disallow: /Blocking these crawlers does not protect your rankings — it removes your pages from the crawls that feed ChatGPT, Claude and Perplexity, so you lose the citations and referral clicks too. Google-Extended is the narrower case worth considering separately: it governs whether your content is used for Gemini and Vertex AI generative products, and does not affect your inclusion in Google Search.
The Webflow-specific mistakes
Two of these come straight out of how Webflow publishing works, and they account for most of the "my robots.txt is wrong" support threads.
Editing the field and never publishing
The settings field and the served file are two different things. Until you hit publish, /robots.txt still returns whatever the last publish produced. Always re-request the live URL after publishing to confirm.
Leaving the .webflow.io staging copy crawlable
Your whole site exists twice: once on your domain and once on the Webflow subdomain. Indexed, that is a full duplicate competing with your real pages. Turning off indexing for the subdomain requires a paid plan, which is a real constraint on free-plan projects.
Using Disallow when you meant noindex
Disallow prevents crawling, not indexing — a blocked URL with inbound links can still appear as a bare title in results. To keep a Webflow page out of search, use the per-page SEO settings to add a noindex tag and leave the page crawlable so the tag can actually be read.
Pasting a WordPress robots.txt into the field
Rules like Disallow: /wp-admin/ or Disallow: /wp-content/ address paths that do not exist on Webflow. They are harmless noise at best, and they usually arrive as part of a copied block that also strips out your sitemap line.
Blocking a folder to hide a client project
robots.txt is public. Anyone can read yours, and a Disallow line is effectively a signpost pointing at the URL you wanted hidden. Use Webflow's password protection for anything that genuinely should not be seen.
Build the file instead of copying one
Our robots.txt generator takes your URL and produces a file with the right sitemap directive and crawl rules for your site, as part of a free strategy run.
Open the robots.txt generatorOr let the agent ship it
Rankealo's SEO agent can open a pull request with the robots.txt fix already written, so a crawl-blocking rule gets reviewed and merged instead of sitting in a report.
See how the SEO agent worksrobots.txt on another platform?
Webflow robots.txt questions
Where do you edit robots.txt in Webflow?
In Site settings under SEO → Indexing. Webflow gives you a text field that becomes the served robots.txt file — there is no filesystem to upload one to. Type your rules into the field, save, and then publish the site; the live file only updates on publish.
What is in the default Webflow robots.txt?
Almost nothing. A new Webflow site has no crawl rules at all, which under the robots.txt standard means everything is allowed. The one thing Webflow adds for you is a Sitemap directive pointing at your sitemap.xml, and there is a toggle in site settings to remove it if you need to.
Do I need a paid Webflow plan to control indexing?
You can edit robots.txt rules in site settings, but disabling search engine indexing of the .webflow.io staging subdomain requires a paid Site plan or Workspace. That matters because every Webflow project is published to that subdomain as well as your custom domain, so a free-plan site cannot fully prevent the staging copy from being crawled.
Should I block AI crawlers in Webflow?
Only if you are prepared to disappear from AI answers. Blocking GPTBot, ClaudeBot or PerplexityBot removes your pages from the crawls that feed those assistants, so you lose the citations and referral clicks along with the training access. Google-Extended is narrower: it governs use of your content for Gemini and Vertex AI generative products and does not affect your inclusion in Google Search.
