Support

Admin Tools

#15800 Admin Tools blocks search bots and crawlers

Posted in ‘Admin Tools for Joomla! 4 & 5’
This is a public ticket

Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.

Environment Information

Joomla! version
n/a
PHP version
n/a
Admin Tools version
n/a

Latest post by nicholas on Tuesday, 30 April 2013 10:22 CDT

Thursday, 18 April 2013 10:05 CDT

user68342
Mandatory information about my setup:

Have I read the related troubleshooter articles above before posting (which pages?)? Yes
Have I searched the tickets before posting? No - There are 95 pages of tickets with no search function.
Have I read the documentation before posting (which pages?)? Yes - WAF and .htaccess sections.
Joomla! version: 2.5.9
PHP version: 5.2
MySQL version: (unknown)
Host: bluehost
Admin Tools version: 2.4.4

Description of my issue:
Admin Tools blocks legitimate search engine bots and crawlers, such as Google and MSN bots, from indexing my website's content. I've read the documentation and there is no mention about how to troubleshoot this issue.

I'm sure I'm not the only who has this problem. I tried to search older support tickets but your search box only looks up website articles and not tickets. Can you please help?

Thursday, 18 April 2013 10:23 CDT

nicholas
Every time someone reports a crawler being blocked we start by finding out why it's blocked.

First go to Admin Tools, Web Application Firewall, Configure WAF. Make sure "Log security exceptions" is set to Yes; if it's not, set it to Yes and click on Save. Now try reproducing your issue, e.g. wait until a crawler is blocked. After that, please go to Admin Tools, Web Application Firewall, Security Exceptions Log. Filter by the IP address of the crawler. Please copy the Reason and Target URL here so that I can help you.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Thursday, 18 April 2013 10:34 CDT

user68342
Thanks for the quick reply. Exceptions log has been turned on since installation. Here's an example of a recent Googlebot that's being blocked:

ip: 66.249.74.51
reason: template= in URL

Bots from msn, bing, and other search engines are blocked for pretty much the same reason.

Thursday, 18 April 2013 20:35 CDT

nicholas
OK, that's what I was expecting. It is documented, but it's not obvious. You have to turn on the Allow site templates option in Web Application Firewall's configuration. If you are wondering why, please read the information under "Allow site templates" at https://www.akeebabackup.com/documentation/admin-tools/web-application-firewall.html#waf-configure

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Tuesday, 30 April 2013 10:13 CDT

user68342
That seems to be working so far. I haven't received "IP Blocked" notifications for last 10 days since I made the change.

Thanks for your help!

Tuesday, 30 April 2013 10:22 CDT

nicholas
You're welcome!

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Support Information

Working hours: We are open Monday to Friday, 9am to 7pm Cyprus timezone (EET / EEST). Support is provided by the same developers writing the software, all of which live in Europe. You can still file tickets outside of our working hours, but we cannot respond to them until we're back at the office.

Support policy: We would like to kindly inform you that when using our support you have already agreed to the Support Policy which is part of our Terms of Service. Thank you for your understanding and for helping us help you!