Cloudflare condemns Perplexity's practices, accused of bypassing website protections

To answer its users' questions, the Perplexity service explores the web and retrieves the most relevant information. But on the internet as elsewhere, there are rules: websites are thus free to set limits on data-collecting robots. Since the beginning, editors can allow or disallow these "crawlers" via the robots.txt file or firewall rules (WAF).

Perplexity's tricks

Except that Perplexity does as it pleases. Cloudflare thus claims, with evidence, that the "IA response engine" has adopted a stealthy crawling behavior. In other words, Perplexity accesses content by bypassing blocks: using a fake user-agent that mimics a classic browser and multiple IPs to hide its origin; sometimes the company's robot completely ignores the instructions in robots.txt.

Cloudflare conducted several experiments by creating websites unknown to crawlers, including Perplexity's. The robots.txt file of these sites prohibits all crawling. And despite not being indexed by other search engines, the service managed to provide details about the content of normally blocked pages to the robots!

This does not help in restoring the reputation of AI companies, which are often accused of collecting data from the "open web" without permission or payment to the creators to train their AI models. In contrast, OpenAI's robot respects the blocking rules in robots.txt.

Cloudflare, which protects millions of websites against attacks and abuse, has removed Perplexity from its list of verified bots and blocked stealth activity through new automated rules.

👉🏻 Follow real-time tech news: add 01net to your Google sources, and subscribe to our WhatsApp channel.

Source : Cloudflare