OriginalSourceTechRadarUnited KingdomLeanUnratedCredibilityPublished06:50 UTCRatings: AllSides, MBFCSourceTechRadarUnited KingdomLeanUnratedCredibilityPublished06:50 UTCRatings: AllSides, MBFC
“How to actually block AI bots from scraping your website”
OpenAI, Google, Microsoft back opt-out signal for robots.txt
Key points
- A new opt-out signal to block AI training on websites is backed by OpenAI, Google, Microsoft and Anthropic.
- The signal is implemented in a website’s robots.txt file to guide AI crawlers.
- Owners can Disallow: / to block all AI crawlers or target specific agents like ChatGPT-User.
- Alternative methods discussed include user-only viewing, IP limiting, and captchas.
- Authors emphasize both protecting data and potential negative effects on user experience.
How it got here
TechRadar publishes the article.
concise reads it.