AI Crawlers and Bad Bots: How Long Can the Open Web Survive?

Opening Discussion

Shiwen: This week, Tech Policy Press reported that AI crawlers are causing significant problems for news websites and blogs. The Congress and New York State are also considering laws requiring all automated web browsers to “propose themselves.” What do you think?

Yongliang: As someone who runs a personal blog and small sites, my first reaction is not “Finally, someone is taking action” but “This regulation seems somewhat arbitrary.” What really hurts is not the technology of crawling itself, but the behavior of crawlers that take content without attribution, no backlinks, or even mirror the entire site. If laws only require crawlers to provide names, it’s like replacing a door lock with a glass one—anyone can push it open. What should be regulated are the actions and consequences of crawling, not forcing crawlers to fill out forms.

Q1: Have AI crawlers really overwhelmed the open internet?

Shiwen: The article says that news organizations train AI with large amounts of scraped content, and AI search and chatbots then take over ads and traffic. As someone who writes online, do you have personal experience?

Yongliang: Yes. I’ve been doing technical content for over a decade. In the past, site traffic mainly came from searches, but now some readers are taken away by AI search summaries. News websites feel this even more strongly because their business model is advertising; when traffic is stolen, ad revenue drops. However, traffic theft is just a symptom; the deeper concern is that content is taken for free. Think about it: hard-written articles are used to train models, which then repeat your ideas elsewhere without attribution or sources. This is completely different from traditional search engines that provide links. Traditional search brings people to your door, while this kind of crawling takes your content and rebuilds a house with it.

Shiwen: What difference does this pressure make for news organizations and individual bloggers?

Yongliang: News organizations face large-scale losses. A report might be seen by hundreds of millions of users through AI chat, but the site’s ad revenue remains zero. Individual bloggers may not suffer such absolute losses, but the psychological impact is strong: my carefully written content is taken away without compensation, even without a thank you. Over time, fewer people will be willing to share their content, reducing the richness of the internet. So, we’re not against AI crawlers, but against those that take content without acknowledgment.

Shiwen: Many publishers now use robots.txt or opt-out mechanisms to prevent AI training of crawlers. Does this solve the problem?

Yongliang: It only partially alleviates it. Robots.txt is a君子 agreement; those who follow the rules will be obeyed, but those who don’t will still be crawled. Moreover, opt-out shifts the burden of proof to content creators, essentially assuming all content can be used unless you declare otherwise. A more reasonable approach would be for AI companies to obtain authorization before using data, rather than having creators declare individually not to allow crawling.

Q2: Is requiring crawlers to disclose identities the wrong direction?

Shiwen: The House of Representatives proposed the Stealth Bot Prohibition Act, and New York State has a similar bill. The core idea is that any automated web access must disclose identity and purpose, otherwise a fine of $53,000 applies. Is this a good solution?

Yongliang: The direction is not right. Knowing who the crawler is and what they do with the data doesn’t mean it’s harmless. Many large AI companies would love to tell you “it was I who collected it,” but traffic and copyright revenue still don’t return to the original authors. What’s truly harmful are those frequent, destructive crawls that don’t follow robots.txt. The laws focus on “anonymity,” which means they also target journalists, security researchers, and archivist workers. These people need anonymity for their work; once forced disclosure occurs, some investigations can be ruined. Even more problematic is that many malicious crawlers can forge identities, and in the end, it’s those who follow rules who suffer.

Shiwen: So according to you, identity disclosure can only solve the problem of “not knowing who it is” but not the problem of “they took my content”?

Yongliang: Exactly. Transparency about identities is more like a business card than a constraint on behavior. Moreover, the phrase “reasonably likely to impair” in these two laws is very vague. There’s no unified standard for what constitutes harm or how to prove it. If law enforcement has too much discretion, small web hosts won’t know how to comply. Small sites can’t afford lawyers, and if they’re wrongly harmed, the safest thing to do is to stop crawling, which makes it harder for content to be discovered. If the laws focused on “whether they follow robots.txt,” “whether they cause server overload,” and “whether they provide reasonable attribution,” they would be more valuable. Otherwise, they’re just punishing a technology form, not the abuse of that technology.

Q3: How important is anonymous crawling to open networks?

Shiwen: The article gave an example: the Wall Street Journal’s Pulitzer Prize-winning investigation used crawlers to collect 31,000 financial disclosures from 12,000 high-ranking officials. Could such reports not have been done without anonymous crawlers?

Yongliang: Yes, anonymous crawling is a fundamental tool for open networks. Academic public opinion analysis, security vulnerability scanning, and non-profit web archiving all rely on this mechanism. CDT’s letter to the New York State Governor said it well: deanonymization threatens the core of open networks but may not solve the real harm caused by AI crawls. If we ban these tools, bad actors remain, while good ones are lost first. Open networks have worked for so many years not because everyone’s identity is transparent, but because of a consensus on robots.txt, cache rules, and rate limits. Now, because of AI crawls, people want to remove the layer of anonymity, which is like taking away knives in the kitchen just because someone used a kitchen knife to hurt someone.

Shiwen: But people might ask, with today’s advanced technology, can journalists not collect data openly?

Yongliang: In reality, many subjects will counterattack. Once your crawler’s identity is revealed, they can immediately block IPs, change page structures, or even track back. Anonymity isn’t a tool for evil; it’s a means to protect information sources and oversight. Internet archives, academic public opinion projects, and vulnerability bounty platforms—which one doesn’t rely on automated crawling? Without this protection, many matters related to public interest won’t be pursued.

Shiwen: Is there a way to protect public interest while limiting the unrestrained crawling by AI companies?

Yongliang: Yes, the key is to distinguish the purposes. For commercial scraping used to train large models, authorization and payment should be required; for scraping for public interest, academic research, or security audits, it should be protected. A one-size-fits-all anonymous ban cannot achieve this distinction. What really needs regulation is behavior: whether you respect robots.txt, cause server downtime, or include citations in your output. Making these clear is much more effective than disclosing identities.

Q4: Are the robots on your site friends or enemies?

Shiwen: You’ve recently implemented GEO and AI search visibility. For small webmasters like you, how do you distinguish between good and bad robot traffic?

Yongliang: I make a simple distinction: AI search citations with traffic are friends, while free scraping is an enemy. For example, some AI searches may include my link in the answers, and readers clicking through that link is a good robot. The role of GEO is to make it easier for these citations to capture my content. But bad robots are different: they operate 24/7, mirror the entire site, and don’t even mention their source. I now use robots.txt to declare rules, limit traffic for abnormal IPs, and sometimes set temporary payment barriers to block them. But if legislation is too rigid, small sites like mine might not even afford the costs of complaints and compliance. Big companies have legal teams, but individual bloggers don’t.

Shiwen: Then how do you determine whether a robot is good or bad? Can you tell by looking at logs?

Yongliang: Most can be identified. Good robots follow robots.txt, have clear User-Agent, reasonable access frequency, and leave traceable citation sources. I’m friendly with clearly labeled crawlers like Googlebot, Bingbot, OpenAI, Perplexity, etc. Bad robots often disguise themselves as regular browsers, with vague User-Agent, abnormally short access intervals, and repeatedly scrape the main text of articles. Further, I look at backlinks and referral traffic: if an AI search brings real traffic to me, I treat it as a good robot; if it only scrapes without any return, it’s free. In terms of tools, I use Nginx’s limit_req for basic traffic limits, combined with CDN’s robot management rules and my own logged-alerts, to cover most common scenarios.

Q5: What can ordinary content creators do now?

Shiwen: For most writers and self-media people, what immediate actions can they take against AI crawlers?

Yongliang: First, write robots.txt properly, marking out directories that you don’t want crawled, which is the most basic declaration. Second, monitor access logs, limit or block abnormal high-frequency IPs; many CDNs and hosting providers offer this feature. Third, set temporary payment barriers or summary modes for important content to prevent valuable content from being easily removed. Fourth, keep your RSS feeds, original timestamps, and sitemaps; in case of infringement, you can at least provide evidence. Fifth, use copyright agreements wisely, letting citation parties know what can be cited and what requires authorization. Finally, don’t open up all your content for free; learn to balance openness and protection. You can also add a simple authorization notice at the bottom of the page, letting compliant crawlers know what to do and making it easier to prove infringement when necessary.

Shiwen: It seems that small website owners can do a lot, but will these protective measures actually affect normal access?

Yongliang: Yes, so the key lies in moderation. Don’t block all crawlers with robots.txt; instead, block only those who don’t follow the rules. Throttling should also be set at reasonable thresholds to avoid harming ordinary readers. My principle is to be friendly to good people and protective against bad ones. If you lock down the entire site just because you fear AI scraping, it goes against the original intention of an open network. The ideal state is where content can be found and cited, but not be scraped at no cost in bulk.

Shiwen: So you don’t oppose AI crawlers themselves, but you oppose scraping without rules?

Yongliang: That’s right. The vitality of an open network lies in its accessibility, but accessibility does not mean it can be plundered. I hope future laws will target destructive behavior, such as frequent, non-compliant scraping without attribution, rather than treating anonymous scraping as a single category. For small website owners and content creators, we welcome AI search that leads to citations, and we also welcome compliant crawlers; but what we need are rules that can distinguish good from bad bots, not simple fixes.

---

Signature: Yongliang