Table of Contents
Web crawling (the act of indexing information on the Internet) has been around for decades. It has been primarily used by search engines like Google and non-profit organizations like Internet Archive and common trace catalog the contents of the open Internet and make them searchable. Until recently, the practice of web crawling was rarely considered controversial, as websites relied on this process as a way for people to find their content. But now crawling technology has been subsumed by the big AI of everything, and is being used by companies like Google and Perplexity AI to suck up entire articles that are fed into their summarization machines.
This week on Gadget Lab, WIRED senior writer Kate Knibbs joins the show to talk about web crawling and the Common Crawl controversy. We then spoke to Forbes Chief Content Officer and Publisher Randall Lane about how Perplexity.AI repurposed a Forbes article and presented it as his own story, without first asking permission or properly citing the source.
Show notes
Read Kate’s story about how publishers are seeking AI training data on Common Crawl. Read Randall’s story on how Preplexity.AI copied the work from two Forbes reporters.
recommendations
Randall recommends his new horse racing league, the National Thoroughbred League. Kate recommends the book. Victim By Andrés Boryga. Lauren recommends the show. tricks in max.
Randall Lane can be found on social media @randalllane. Kate Knibbs is @Knibbs. Lauren Goode is @laurengoode. Michael Calore is @snack fight. Bling the main hotline at @gadget lab. The show is produced by Boone Ashworth (@booneashworth). Our musical theme is Solar keys.
How to listen
You can always listen to this week’s podcast through the audio player on this page, but if you want to subscribe for free to get all the episodes, here’s how:
If you’re on an iPhone or iPad, open the app called Podcasts, or just tap this link. You can also download an app like Overcast or Pocket Casts and search for Gadget Lab. If you’re on Android, you can find us on the Google Podcasts app by simply playing here. They were in Spotify also. And in case you really need it, here is the RSS feed.