Export
robots.txt
/wiki/robots-txt
The link always opens the latest version of this page.
Ask an AI about it
Open in ChatGPTA new chat that reads this entry from the site Open in ClaudeThe same, in Claude
Sending this to someone?
MarkdownPlain text for notes apps and AI chats
WordA document to edit and annotate
PDFA4, for printing and sharing
ZIPEverything: the documents, the data, and instructions for AI assistants
Share the link instead of a file, and they will always see the latest version.
Files are a snapshot of today. The live page has anything newer.
growing ·
10 feeds
robots.txt
Text file used to communicate crawling preferences to bots
Snapshot
- Domain
- web crawling control 1 source
Intelligence coverage
Text file used to communicate crawling preferences to bots.
Drawn from 1 news source; the facts and relationships below show where each detail comes from.
Facts 1
| Attribute | Value | Confidence | Sources | Cited |
|---|---|---|---|---|
| Domain | web crawling control | low | 1 | 4 |
Frequently mentioned together 16
Sources 13
- I Found My Own Website Blocking ChatGPT. Then I Checked 640 Others. · HackerNoon
- After a Linux Server Data Leak: Containing Public Exposure and Search Indexing · LinuxSecurity: Articles
- Blunt Banhammer Blocks Basic Bots Ballistically · Lobsters
- I've Been Deindexed by Google · Lobsters
- Stay discoverable in search while disallowing AI training · Hacker News (YC)
5 of 13 sources shown · Pro unlocks the full list