OpenAI has rolled out its web crawler GPTBot, which can help the company prepare for its next big GPT-5 language model. In other words, the AI company will fetch the data online to develop another big upgrade for ChatGPT. Fortunately, OpenAI has provided websites with a way to prevent the tech company from harvesting their data.
Although less than a year old, this generative AI tool has become a staple for many people around the world. People use it for daily tasks, but some are worried that the program will put their data at risk. Therefore, you should read this article carefully if you are concerned that artificial intelligence is encroaching on your online business or content.
This article explains how to stop GPTBot from using your website data for AI training. Later, I’ll explain why some people think OpenAI will use online content to build a more powerful chatbot.
How to protect your website from GPTBot
OpenAI announced its web crawler GPTBot last week, which means it has started scraping data from the internet. It identifies itself with the following user agent and string:
User agent token: GPTBot
Full user agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)
If you see this on your server, it could mean that OpenAI is fetching data from your site. Fortunately, the company says you can block GPTBot from accessing your website by adding this string to its robots.txt file:
User agent: GPTBot
To forbid : /
Navigate to your website’s robots.txt file by entering your domain name followed by “/robots.txt”. For example, navigate to “www.mywebsite.com/robots.txt” if your website is “www.mywebsite.com”.
You might also like: The Ultimate Guide to ChatGPT
Also, OpenAI provided another text string to customize access to GPTBot. First enter this string, then place the pages you want GPTBot to fetch and ignore:
User agent: GPTBot
Allow: /directory-1/
Disallow: /directory-2/
Enter the URL pages you want the web crawler to check in the “Allow” category. Conversely, enter the ones you want to leave untouched in the “Disallow” section.
Why would OpenAI scrape internet data?
Photo credit: insidetelecom.com
The development of GPT-5 is perhaps the main reason why the creator of ChatGPT needs website data. The AI company did not specify a reason at the time of writing, but it has filed a trademark application for GPT-5.
A trademark prevents others from using that name or its features, which implies that the tech company would release GPT-5. GPT is the large language model used by OpenAI for the world famous ChatGPT program.
“GPT” stands for “pre-trained generative transformer”, which means that it must receive “pre-training”. This training consists of feeding the LLM data to refine its analysis and processing.
ChatGPT could face one of the biggest challenges of modern AI systems: the lack of training data. Nowadays, AI bots lack human-created data for training, so they fetch AI-generated content.
Unfortunately, this can quickly degrade their performance as AI programs repeatedly learn from their models. As a result, they could become unreliable and outdated.
Another reason is that AI companies want their programs to become more useful to attract more users. This can only happen if these chatbots can refer to live online information.
You might also like: Make AI do it all with AutoGPT
Nowadays, OpenAI and other companies have enabled their AI robots to fetch data online. However, they usually warn users that they are not always reliable.
After all, it can be difficult to filter what an AI bot will use as references. The internet is full of misinformation and shoddy content, and programming an AI to check them before showing results is next to impossible.
Still, that’s not stopping OpenAI from trying with its next GPT-5 model since it filed a trademark. GPTBot could be his next step to make the next version of ChatGPT a reality.
Conclusion
OpenAI recently announced that its web crawler GPTBot will scrape data from websites. Fortunately, this has allowed companies to protect their platforms.
Meanwhile, Google proclaimed a similar development but did not provide a way to opt out. It said it would provide this option, but it does not at the time of writing.
You should try the steps above if you have a blog, art gallery, or similar online content. This is especially important if you want to protect your online business. Also check out Inquirer Tech for more digital tips and trends.
Frequently asked questions about GPTBot
What are the risks of AI training?
AI training could jeopardize people’s intellectual property by learning to imitate the works of artists. Soon, artificial intelligence systems could put creatives out of business. In addition, GPTBot could risk trade secrets, jeopardizing the confidentiality of company data. Use the tips above to protect yourself from this web crawler.
Should I prevent GPTBot from scraping my data?
You must prevent GPTBot from accessing your online data for privacy reasons. However, you may want it to access specific pages if you rely on ChatGPT. Consequently, the AI robot could meet your daily needs more effectively. Luckily, you can specify which pages to expose and hide from GPTBot.
Is GPT-5 coming soon?
OpenAI hasn’t specified a release date for GPT-5, and CEO Sam Altman said his company isn’t developing the update. However, OpenAI recently filed a trademark application for GPT-5 and then released a web crawler for AI training. As a result, many sources believe it may launch this new chatbot soon.
Your subscription could not be registered. Try Again.
Your subscription was successful.
Read more
To subscribe to MORE APPLICANT to access The Philippine Daily Inquirer and over 70 titles, share up to 5 gadgets, listen to news, download as early as 4am and share articles on social media. Call 896 6000.
