Support

Admin Tools

#40858 How to Properly Block AI Crawlers on Website without Causing Errors

Posted in ‘Admin Tools for Joomla! 4 & 5’
This is a public ticket

Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.

Environment Information

Joomla! version
5.1.1
PHP version
8.2.18
Admin Tools version
7.5.4

Latest post by nicholas on Monday, 24 June 2024 00:58 CDT

rmesceo

Please look at the bottom of this page (under Support Policy Summary) for our support policy summary, containing important information regarding our working hours and our support policy. Thank you!

 

Dear Support,

I followed the instructions from the website (https://www.cyberciti.biz/web-developer/block-openai-bard-bing-ai-crawler-bots-using-robots-txt-file/) to add the code for blocking AI to the .htaccess Maker in Admintools, but the website was not displayed normally (Error 500), and I had to delete the added code to restore it to normal. Could you please advise how I can add the blocking AI code properly?

 

Added code:

User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Omgilibot
Disallow: /
User-Agent: FacebookBot
Disallow: /
User-Agent: Applebot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: ImagesiftBot
Disallow: /
User-agent: Omgilibot
Disallow: /
User-agent: Omgili
Disallow: /
User-agent: YouBot
Disallow: /

 

Regards,

Johnny

 

nicholas
Akeeba Staff
Manager

You need to read that article again. It tells you to add these lines into your site's robots.txt file, NOT your .htaccess. It even spends the first part of the article explaining what the robots.txt file is.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

rmesceo

That's so funny, it seems I've been too tense, I should take a break and relax now.

 

Thanks

nicholas
Akeeba Staff
Manager

When things stop making sense, turn the computer off and go for a walk. Works every time for me :)

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Support Information

Working hours: We are open Monday to Friday, 9am to 7pm Cyprus timezone (EET / EEST). Support is provided by the same developers writing the software, all of which live in Europe. You can still file tickets outside of our working hours, but we cannot respond to them until we're back at the office.

Support policy: We would like to kindly inform you that when using our support you have already agreed to the Support Policy which is part of our Terms of Service. Thank you for your understanding and for helping us help you!