Support

Admin Tools

#26687 Issue in indexing in google

Posted in ‘Admin Tools for Joomla! 4 & 5’
This is a public ticket

Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.

Environment Information

Joomla! version
n/a
PHP version
n/a
Admin Tools version
n/a

Latest post by on Saturday, 07 January 2017 17:17 CST

okkhalid
 Hello,
I'm having some issue indexing my site right in google, and I have RSSEO component, they advised me after looking into my site the following:

Looking at your robots.txt, please remove its contents and use only the followings instead:

User-agent: *
Disallow: /administrator/
Disallow: /installation/
Disallow: /logs/
Disallow: /tmp/

Afterwards perform a new fetch for the URLs you've pointed out in your screenshot via Google Search Console. This should be then reflected in a couple of days and we'll be then able to further investigate this matter (let us know when the time comes).

Is that Safe for me just to have the above lines in the robots.txt?

Thanks,

nicholas
Akeeba Staff
Manager
It depends on your site. Let's see what the other core Joomla folders contain:
  • cache This is temporary data which must not be web accessible. Therefore I would add a disallow rule in robots.txt about it.
  • cli Scripts which must only be run through the command line (or a CRON job which is the same thing). Therefore I would add a disallow rule in robots.txt about it.
  • components The private PHP files of components. It MAY also contain CSS and JS for some components. However, since there are no content documents there I'd also add a disallow rule in robots.txt about it UNLESS you WANT the static media files (CSS, JS, images, possibly some user-uploaded content) to be indexed by search engines.
  • images These are all the images you have uploaded. I assume you want them indexed so do NOT create a robots.txt rule about this folder.
  • includes, layouts, libraries Private Joomla! PHP files. MUST all be added in a disallow rule in robots.txt.
  • language Private Joomla .ini translation files. MUST all be added in a disallow rule in robots.txt.
  • media User-uploaded or generated content for etensions. MUST NOT be added in a disallow rule in robots.txt. It's the same logic as the images folder. There are some subtle differences between each folder but the bottom line is that you must not disallow them in robots.txt.
  • modules, plugins, templates The same logic as the components folder. Disallow UNLESS you WANT the static media files of your modules, plugins and templates respectively to be indexed by Google.


As a rule of thumb I recommend disallowing indexing on all folders except images and media (the only directories which should contain user-generated content).

That said, I don't know the context of your support request to the RSSEO guys. Maybe they asked you to create this very basic robots.txt temporarily to troubleshoot something and then restore the file or add more stuff to it. So I won't tell you this is good or this is bad as I lack context. I'd say that the proposed very basic robots.txt is not good as a permanent solution but it's not going to cause you any problems as a short term troubleshooting solution.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

System Task
system
This ticket has been automatically closed. All tickets which have been inactive for a long time are automatically closed. If you believe that this ticket was closed in error, please contact us.

Support Information

Working hours: We are open Monday to Friday, 9am to 7pm Cyprus timezone (EET / EEST). Support is provided by the same developers writing the software, all of which live in Europe. You can still file tickets outside of our working hours, but we cannot respond to them until we're back at the office.

Support policy: We would like to kindly inform you that when using our support you have already agreed to the Support Policy which is part of our Terms of Service. Thank you for your understanding and for helping us help you!