Support

Admin Tools

#32038 Google and BING bots seem to be blocked

Posted in ‘Admin Tools for Joomla! 4 & 5’
This is a public ticket

Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.

Environment Information

Joomla! version
n/a
PHP version
n/a
Admin Tools version
n/a

Latest post by nicholas on Monday, 25 November 2019 01:06 CST

Sunday, 24 November 2019 11:00 CST

baroen
Hi,
It seems like google and bingbots are being blocked by Admin Tools.
As of last month my Adwords adds and Search is dropped massively.
After long investigation I found that there is a 403 response thrown at Bing. (see below file for full response)
Allthough I cannot create such a response for the GoogleBot I guess it is the same problem.
As it says in the response that acces is blocked by Admin Tools: Do you know how to fix this?

Kind regards,
Jeroen
Edited by nicholas on 2019-11-25 01:05 CST

Sunday, 24 November 2019 11:27 CST

nicholas
Without knowing the exact URL being accessed and your Admin Tools configuration or at least having a detailed log file from Admin Tools, no, I cannot tell you with any degree of certainty why these two search bots are blocked.

What people usually do is that they enable Geographic IP blocking. If you block USA or North America then these bots, which access your site from IP addresses in the USA, will also get blocked. I've already documented that geographic IP blocking is a bad idea. That said, geographic IP blocking should not result in the message you are reporting with the default configuration. Please do check if you've enabled geographic IP blocking anyway.

If it's any other reason these two bots in particular will not get blocked by Admin Tools. In the default configuration the Web Application Firewall, Configure WAF, Exceptions tab has the Whitelisted Domains set to
.googlebot.com,.search.msn.com

This explicitly allows Google Bot and Bing Bot to access your site. Have you changed that setting?

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Sunday, 24 November 2019 11:34 CST

baroen
Hi Nicholas,
Thank you for answering so quickly. Super!
Somehow I probably set North-America to be blocked sometime.
Any how, unblocking did the trick.
I am crawlable again. Thanks.
(can you please remove the code from the original ticket, or set it back to private?))

Monday, 25 November 2019 01:06 CST

nicholas
You're welcome! I am glad I could help :)

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Support Information

Working hours: We are open Monday to Friday, 9am to 7pm Cyprus timezone (EET / EEST). Support is provided by the same developers writing the software, all of which live in Europe. You can still file tickets outside of our working hours, but we cannot respond to them until we're back at the office.

Support policy: We would like to kindly inform you that when using our support you have already agreed to the Support Policy which is part of our Terms of Service. Thank you for your understanding and for helping us help you!