#ThisWeekinNET — Episode 67

In this week’s episode, we go a bit technical in our segment “Let’s //Get\ Technical,” featuring our CTO, John Graham-Cumming. We discuss AI bot and crawlers, elections and more.

Catch more episodes and subscribe to the podcast: https://cloudflare.tv/shows/this-week-in-net

Host João Tomé and John Graham-Cumming begin by explaining how Cloudflare recently launched a new “easy button” to block all AI bots, scrapers, and crawlers to help maintain a safe Internet for content creators. It’s available to all customers, including those on our free tier. We demonstrate how to activate this button on the Cloudflare dashboard. We also explain the purpose of the robots.txt file—it resides at the root of your domain and contains directives for search engine crawlers on how to index pages on a website.

Next, we discuss how we recently replaced links to polyfill.io with Cloudflare’s mirror to ensure a safer Internet—polyfill.io is a popular JavaScript library service that can no longer be trusted and should be removed from websites.

We also analyze recent election trends in the UK and France, as well as the Biden vs. Trump US election debate. We were surprised to find that so many websites related to political parties were the targets of DDoS attacks blocked by Cloudflare around the election (or debate) days.

Also mentioned is Cloudflare’s 1.1.1.1 incident on June 27, 2024. The root cause was a combination of BGP (Border Gateway Protocol) hijacking and a route leak.