Global Giants and European Leaders Blocking OpenAI's Bot

October 29, 2025

OpenAI’s GPTBot, a web crawler essential for training its AI models, is being met with resistance. A growing number of major websites are opting to block the bot, citing concerns over content scraping, data privacy, and the potential for misuse of their information. This move signals a significant pushback from content creators who wish to control how their data is utilized in the age of generative AI. Here are some of the most prominent websites that have put up a digital stop sign to OpenAI’s bot.

1. The New York Times

A stalwart of journalism, The New York Times has blocked GPTBot to protect its premium content from being used to train AI models without compensation.

2. Amazon

The e-commerce giant has blocked the bot, likely to safeguard its vast repository of product descriptions, customer reviews, and proprietary sales data.

3. Pinterest

This image-sharing platform is blocking GPTBot to prevent the scraping of its extensive visual and user-generated content.

4. Indeed

A leading job search engine, Indeed is blocking the bot to protect its listings and user data from being harvested.

5. USA Today

Another major American newspaper, USA Today is blocking the bot to prevent its news articles from being used to train AI without permission.

6. Wired

Known for its in-depth tech journalism, Wired is blocking GPTBot to protect its intellectual property and maintain control over its content.

7. Stack Exchange

A network of Q&A websites, Stack Exchange is blocking the bot to prevent the scraping of its user-generated questions and answers.

8. WebMD

A popular source for medical information, WebMD is blocking GPTBot to protect its copyrighted health content.

9. Quora

This question-and-answer platform has blocked GPTBot to prevent its user-generated content from being used to train AI models.

European Companies Taking a Stand

The trend of blocking GPTBot is not limited to the US. Several major European companies have also implemented measures to prevent OpenAI’s crawler from accessing their content.

The Guardian (UK)

This prominent UK news organization has joined the ranks of media outlets blocking GPTBot, emphasizing the need to control its journalistic content.

MailOnline (UK)

The online platform for the Daily Mail, another major UK news outlet, is blocking GPTBot to protect its content.

BBC (UK)

The British Broadcasting Corporation (BBC) has blocked both of OpenAI’s crawlers, GPTBot and ChatGPT-User, to safeguard its extensive archive of news and media content.

OK Diario (Spain)

This Spanish news publication is one of the few major media outlets in Spain to block GPTBot.

BFM TV (France)

One of France’s most-watched news channels, BFM TV, has also blocked OpenAI’s web crawler.

The Dutch Perspective

While many large companies across Europe are blocking GPTBot, the trend does not appear to have gained the same traction in the Netherlands. At the time of this writing, a review of the most popular Dutch websites, including e-commerce giants like Bol.com and ah.nl, and news outlets like Buienradar, shows that they are not currently blocking OpenAI’s web crawler. This may change as the global conversation around AI and data privacy continues to evolve.

Conclusion

The decision by these and other major websites to block OpenAI’s GPTBot highlights a growing tension between AI developers and content creators. As AI continues to evolve, the debate over data ownership and fair use will undoubtedly intensify. The actions of these digital giants may pave the way for new standards and practices in the ethical sourcing of data for AI training.

Resources

OpenAI’s GPTBot, a web crawler used by OpenAI to collect web content for model development, has been blocked by a number of major websites. Site operators cite concerns such as content scraping, data privacy, and control over how their content is used. The situation is evolving quickly; this post summarises reported blocks and how they were verified. See the “Resources & evidence” section for links to robots.txt excerpts and coverage.

Note: Claims below are based on publicly available sources (robots.txt entries or news coverage). Where possible, verify the linked evidence yourself — robots.txt files can change, and statements may be updated.

1. The New York Times

The New York Times is widely reported to have restricted access to certain crawlers to protect premium content. [Evidence: robots.txt / news coverage — see Resources & evidence]

2. Amazon

Amazon has taken steps to limit automated scraping of product data and reviews; reports indicate they have rules preventing some crawlers. [Evidence: robots.txt / news coverage — see Resources & evidence]

3. Pinterest

Pinterest, which hosts large amounts of user-generated images, has implemented restrictions reported to affect some automated crawlers. [Evidence: robots.txt / news coverage — see Resources & evidence]

4. Indeed

Indeed, a major job search platform, has measures to protect listings and user data from automated harvesting. [Evidence: robots.txt / news coverage — see Resources & evidence]

5. USA Today

USA Today has been reported to take steps to prevent automated copying of its articles. [Evidence: robots.txt / news coverage — see Resources & evidence]

6. Wired

Wired has protections in place to limit automated access to its content, according to public records. [Evidence: robots.txt / news coverage — see Resources & evidence]

7. Stack Exchange

Stack Exchange has previously restricted some crawlers and changed policies around content reuse. Verify current status via their robots.txt or official posts. [Evidence: robots.txt / meta.stackoverflow posts — see Resources & evidence]

8. WebMD

WebMD has protections for its copyrighted health content; reports suggest restrictions on automated scraping. [Evidence: robots.txt / news coverage — see Resources & evidence]

9. Quora

Quora has taken measures to limit automated copying of user-generated content; check their robots.txt and policy pages for details. [Evidence: robots.txt / policy pages — see Resources & evidence]

European outlets reported to restrict GPTBot

  • The Guardian (UK) — reported to restrict some crawlers. [Evidence: robots.txt / coverage]
  • MailOnline (UK) — reported restrictions. [Evidence: robots.txt / coverage]
  • BBC (UK) — reported to have blocked specific OpenAI crawlers in robots.txt. [Evidence: https://www.bbc.co.uk/robots.txt and coverage]
  • OK Diario (Spain) — reported restriction. [Evidence: robots.txt / coverage]
  • BFM TV (France) — reported restriction. [Evidence: robots.txt / coverage]

The Dutch perspective

At the time this article was last checked (2025-10-29), a review of popular Dutch sites such as Bol.com, ah.nl, and Buienradar did not show explicit blocks of OpenAI’s GPTBot in their robots.txt files. These checks are time-sensitive; revisit the robots.txt files for the latest status.

How to verify if a site blocks GPTBot

  • Visit https:///robots.txt and look for lines like:

    User-agent: GPTBot Disallow: /

  • If you find an explicit User-agent: GPTBot entry that disallows access, that indicates the site requests the crawler not access the site. Note that robots.txt is a voluntary standard and not an enforcement mechanism.

Conclusion

A number of high-profile websites have publicly indicated or been reported to restrict automated crawlers, including those associated with some AI developers. Because this is an active and changing situation, add “lastChecked” timestamps and link to primary evidence when making claims.

Resources & evidence (examples to verify)