Support

Admin Tools

#21754 htaccess rules blocking jsitemap links

Posted in ‘Admin Tools for Joomla! 4 & 5’
This is a public ticket

Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.

Environment Information

Joomla! version
n/a
PHP version
n/a
Admin Tools version
n/a

Latest post by tampe125 on Monday, 29 December 2014 03:50 CST

quebecguitare
 Hello, I have problem accessing these links for a sitemap component I just bought:

http://www.quebecguitare.ca/index.php?option=com_jmap&view=sitemap&format=mobile&xslt=1
http://www.quebecguitare.ca/index.php?option=com_jmap&view=sitemap&format=gnews&xslt=1
http://www.quebecguitare.ca/index.php?option=com_jmap&view=sitemap&format=images&xslt=1
http://www.quebecguitare.ca/index.php?option=com_jmap&view=sitemap&format=xml&xslt=1

I have tested the same link renaming htaccess to htaccess.txt and it works. I have concluded that the problem occurs as a results of previous entries made using the Admin component htaccess functions.

Could you please advise on a rewrite condition to allow these links? I have no knowledge in that matter.

Here is my .htaccess file:


### ===========================================================================
### Security Enhanced & Highly Optimized .htaccess File for Joomla!
### automatically generated by Admin Tools 3.3.1 on 2014-12-04 14:15:19 GMT
### Auto-detected Apache version: 2.5 (best guess)
### ===========================================================================
###
### The contents of this file are based on the same author's work "Master
### .htaccess", published on http://snipt.net/nikosdion/the-master-htaccess
###
### Admin Tools is Free Software, distributed under the terms of the GNU
### General Public License version 3 or, at your option, any later version
### published by the Free Software Foundation.
###
### !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! IMPORTANT !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
### !!                                                                       !!
### !!  If you get an Internal Server Error 500 or a blank page when trying  !!
### !!  to access your site, remove this file and try tweaking its settings  !!
### !!  in the back-end of the Admin Tools component.                        !!
### !!                                                                       !!
### !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
###

##### Set to PHP 5.4
AddHandler application/x-httpd-php54 .php .php5 .php4 .php3

##### Set server signature OFF

ServerSignature Off
##ServerTokens Prod

##### RewriteEngine enabled - BEGIN
RewriteEngine On
##### RewriteEngine enabled - END

##### RewriteBase set - BEGIN
RewriteBase /
##### RewriteBase set - END

##### File execution order -- BEGIN
DirectoryIndex index.php index.html
##### File execution order -- END

##### No directory listings -- BEGIN
IndexIgnore *
Options -Indexes
##### No directory listings -- END

##### Optimal default expiration time - BEGIN
<IfModule mod_expires.c>
	# Enable expiration control
	ExpiresActive On

	# Default expiration: 1 hour after request
	ExpiresDefault "now plus 1 hour"

	# CSS and JS expiration: 1 week after request
	ExpiresByType text/css "now plus 1 week"
	ExpiresByType application/javascript "now plus 1 week"
	ExpiresByType application/x-javascript "now plus 1 week"

	# Image files expiration: 1 month after request
	ExpiresByType image/bmp "now plus 1 month"
	ExpiresByType image/gif "now plus 1 month"
	ExpiresByType image/jpeg "now plus 1 month"
	ExpiresByType image/jp2 "now plus 1 month"
	ExpiresByType image/pipeg "now plus 1 month"
	ExpiresByType image/png "now plus 1 month"
	ExpiresByType image/svg+xml "now plus 1 month"
	ExpiresByType image/tiff "now plus 1 month"
	ExpiresByType image/vnd.microsoft.icon "now plus 1 month"
	ExpiresByType image/x-icon "now plus 1 month"
	ExpiresByType image/ico "now plus 1 month"
	ExpiresByType image/icon "now plus 1 month"
	ExpiresByType text/ico "now plus 1 month"
	ExpiresByType application/ico "now plus 1 month"
	ExpiresByType image/vnd.wap.wbmp "now plus 1 month"
	ExpiresByType application/vnd.wap.wbxml "now plus 1 month"
	ExpiresByType application/smil "now plus 1 month"

	# Audio files expiration: 1 month after request
	ExpiresByType audio/basic "now plus 1 month"
	ExpiresByType audio/mid "now plus 1 month"
	ExpiresByType audio/midi "now plus 1 month"
	ExpiresByType audio/mpeg "now plus 1 month"
	ExpiresByType audio/x-aiff "now plus 1 month"
	ExpiresByType audio/x-mpegurl "now plus 1 month"
	ExpiresByType audio/x-pn-realaudio "now plus 1 month"
	ExpiresByType audio/x-wav "now plus 1 month"

	# Movie files expiration: 1 month after request
	ExpiresByType application/x-shockwave-flash "now plus 1 month"
	ExpiresByType x-world/x-vrml "now plus 1 month"
	ExpiresByType video/x-msvideo "now plus 1 month"
	ExpiresByType video/mpeg "now plus 1 month"
	ExpiresByType video/mp4 "now plus 1 month"
	ExpiresByType video/quicktime "now plus 1 month"
	ExpiresByType video/x-la-asf "now plus 1 month"
	ExpiresByType video/x-ms-asf "now plus 1 month"
</IfModule>
##### Optimal default expiration time - END

##### Common hacking tools and bandwidth hoggers block -- BEGIN
SetEnvIf user-agent "WebBandit" stayout=1
SetEnvIf user-agent "webbandit" stayout=1
SetEnvIf user-agent "Acunetix" stayout=1
SetEnvIf user-agent "binlar" stayout=1
SetEnvIf user-agent "BlackWidow" stayout=1
SetEnvIf user-agent "Bolt 0" stayout=1
SetEnvIf user-agent "Bot mailto:[email protected]" stayout=1
SetEnvIf user-agent "BOT for JCE" stayout=1
SetEnvIf user-agent "casper" stayout=1
SetEnvIf user-agent "checkprivacy" stayout=1
SetEnvIf user-agent "ChinaClaw" stayout=1
SetEnvIf user-agent "clshttp" stayout=1
SetEnvIf user-agent "cmsworldmap" stayout=1
SetEnvIf user-agent "comodo" stayout=1
SetEnvIf user-agent "Custo" stayout=1
SetEnvIf user-agent "Default Browser 0" stayout=1
SetEnvIf user-agent "diavol" stayout=1
SetEnvIf user-agent "DIIbot" stayout=1
SetEnvIf user-agent "DISCo" stayout=1
SetEnvIf user-agent "dotbot" stayout=1
SetEnvIf user-agent "Download Demon" stayout=1
SetEnvIf user-agent "eCatch" stayout=1
SetEnvIf user-agent "EirGrabber" stayout=1
SetEnvIf user-agent "EmailCollector" stayout=1
SetEnvIf user-agent "EmailSiphon" stayout=1
SetEnvIf user-agent "EmailWolf" stayout=1
SetEnvIf user-agent "Express WebPictures" stayout=1
SetEnvIf user-agent "extract" stayout=1
SetEnvIf user-agent "ExtractorPro" stayout=1
SetEnvIf user-agent "EyeNetIE" stayout=1
SetEnvIf user-agent "feedfinder" stayout=1
SetEnvIf user-agent "FHscan" stayout=1
SetEnvIf user-agent "FlashGet" stayout=1
SetEnvIf user-agent "flicky" stayout=1
SetEnvIf user-agent "GetRight" stayout=1
SetEnvIf user-agent "GetWeb!" stayout=1
SetEnvIf user-agent "Go-Ahead-Got-It" stayout=1
SetEnvIf user-agent "Go!Zilla" stayout=1
SetEnvIf user-agent "grab" stayout=1
SetEnvIf user-agent "GrabNet" stayout=1
SetEnvIf user-agent "Grafula" stayout=1
SetEnvIf user-agent "harvest" stayout=1
SetEnvIf user-agent "HMView" stayout=1
SetEnvIf user-agent "ia_archiver" stayout=1
SetEnvIf user-agent "Image Stripper" stayout=1
SetEnvIf user-agent "Image Sucker" stayout=1
SetEnvIf user-agent "InterGET" stayout=1
SetEnvIf user-agent "Internet Ninja" stayout=1
SetEnvIf user-agent "InternetSeer.com" stayout=1
SetEnvIf user-agent "jakarta" stayout=1
SetEnvIf user-agent "Java" stayout=1
SetEnvIf user-agent "JetCar" stayout=1
SetEnvIf user-agent "JOC Web Spider" stayout=1
SetEnvIf user-agent "kmccrew" stayout=1
SetEnvIf user-agent "larbin" stayout=1
SetEnvIf user-agent "LeechFTP" stayout=1
SetEnvIf user-agent "libwww" stayout=1
SetEnvIf user-agent "Mass Downloader" stayout=1
SetEnvIf user-agent "Maxthon$" stayout=1
SetEnvIf user-agent "microsoft.url" stayout=1
SetEnvIf user-agent "MIDown tool" stayout=1
SetEnvIf user-agent "miner" stayout=1
SetEnvIf user-agent "Mister PiX" stayout=1
SetEnvIf user-agent "NEWT" stayout=1
SetEnvIf user-agent "MSFrontPage" stayout=1
SetEnvIf user-agent "Navroad" stayout=1
SetEnvIf user-agent "NearSite" stayout=1
SetEnvIf user-agent "Net Vampire" stayout=1
SetEnvIf user-agent "NetAnts" stayout=1
SetEnvIf user-agent "NetSpider" stayout=1
SetEnvIf user-agent "NetZIP" stayout=1
SetEnvIf user-agent "nutch" stayout=1
SetEnvIf user-agent "Octopus" stayout=1
SetEnvIf user-agent "Offline Explorer" stayout=1
SetEnvIf user-agent "Offline Navigator" stayout=1
SetEnvIf user-agent "PageGrabber" stayout=1
SetEnvIf user-agent "Papa Foto" stayout=1
SetEnvIf user-agent "pavuk" stayout=1
SetEnvIf user-agent "pcBrowser" stayout=1
SetEnvIf user-agent "PeoplePal" stayout=1
SetEnvIf user-agent "planetwork" stayout=1
SetEnvIf user-agent "psbot" stayout=1
SetEnvIf user-agent "purebot" stayout=1
SetEnvIf user-agent "pycurl" stayout=1
SetEnvIf user-agent "RealDownload" stayout=1
SetEnvIf user-agent "ReGet" stayout=1
SetEnvIf user-agent "Rippers 0" stayout=1
SetEnvIf user-agent "SeaMonkey$" stayout=1
SetEnvIf user-agent "sitecheck.internetseer.com" stayout=1
SetEnvIf user-agent "SiteSnagger" stayout=1
SetEnvIf user-agent "skygrid" stayout=1
SetEnvIf user-agent "SmartDownload" stayout=1
SetEnvIf user-agent "sucker" stayout=1
SetEnvIf user-agent "SuperBot" stayout=1
SetEnvIf user-agent "SuperHTTP" stayout=1
SetEnvIf user-agent "Surfbot" stayout=1
SetEnvIf user-agent "tAkeOut" stayout=1
SetEnvIf user-agent "Teleport Pro" stayout=1
SetEnvIf user-agent "Toata dragostea mea pentru diavola" stayout=1
SetEnvIf user-agent "turnit" stayout=1
SetEnvIf user-agent "vikspider" stayout=1
SetEnvIf user-agent "VoidEYE" stayout=1
SetEnvIf user-agent "Web Image Collector" stayout=1
SetEnvIf user-agent "Web Sucker" stayout=1
SetEnvIf user-agent "WebAuto" stayout=1
SetEnvIf user-agent "WebCopier" stayout=1
SetEnvIf user-agent "WebFetch" stayout=1
SetEnvIf user-agent "WebGo IS" stayout=1
SetEnvIf user-agent "WebLeacher" stayout=1
SetEnvIf user-agent "WebReaper" stayout=1
SetEnvIf user-agent "WebSauger" stayout=1
SetEnvIf user-agent "Website eXtractor" stayout=1
SetEnvIf user-agent "Website Quester" stayout=1
SetEnvIf user-agent "WebStripper" stayout=1
SetEnvIf user-agent "WebWhacker" stayout=1
SetEnvIf user-agent "WebZIP" stayout=1
SetEnvIf user-agent "Wget" stayout=1
SetEnvIf user-agent "Widow" stayout=1
SetEnvIf user-agent "WWW-Mechanize" stayout=1
SetEnvIf user-agent "WWWOFFLE" stayout=1
SetEnvIf user-agent "Xaldon WebSpider" stayout=1
SetEnvIf user-agent "Yandex" stayout=1
SetEnvIf user-agent "Zeus" stayout=1
SetEnvIf user-agent "zmeu" stayout=1
SetEnvIf user-agent "CazoodleBot" stayout=1
SetEnvIf user-agent "discobot" stayout=1
SetEnvIf user-agent "ecxi" stayout=1
SetEnvIf user-agent "GT::WWW" stayout=1
SetEnvIf user-agent "heritrix" stayout=1
SetEnvIf user-agent "HTTP::Lite" stayout=1
SetEnvIf user-agent "HTTrack" stayout=1
SetEnvIf user-agent "ia_archiver" stayout=1
SetEnvIf user-agent "id-search" stayout=1
SetEnvIf user-agent "id-search.org" stayout=1
SetEnvIf user-agent "IDBot" stayout=1
SetEnvIf user-agent "Indy Library" stayout=1
SetEnvIf user-agent "IRLbot" stayout=1
SetEnvIf user-agent "ISC Systems iRc Search 2.1" stayout=1
SetEnvIf user-agent "LinksManager.com_bot" stayout=1
SetEnvIf user-agent "linkwalker" stayout=1
SetEnvIf user-agent "lwp-trivial" stayout=1
SetEnvIf user-agent "MFC_Tear_Sample" stayout=1
SetEnvIf user-agent "Microsoft URL Control" stayout=1
SetEnvIf user-agent "Missigua Locator" stayout=1
SetEnvIf user-agent "panscient.com" stayout=1
SetEnvIf user-agent "PECL::HTTP" stayout=1
SetEnvIf user-agent "PHPCrawl" stayout=1
SetEnvIf user-agent "PleaseCrawl" stayout=1
SetEnvIf user-agent "SBIder" stayout=1
SetEnvIf user-agent "Snoopy" stayout=1
SetEnvIf user-agent "Steeler" stayout=1
SetEnvIf user-agent "URI::Fetch" stayout=1
SetEnvIf user-agent "urllib" stayout=1
SetEnvIf user-agent "Web Sucker" stayout=1
SetEnvIf user-agent "webalta" stayout=1
SetEnvIf user-agent "WebCollage" stayout=1
SetEnvIf user-agent "Wells Search II" stayout=1
SetEnvIf user-agent "WEP Search" stayout=1
SetEnvIf user-agent "zermelo" stayout=1
SetEnvIf user-agent "ZyBorg" stayout=1
SetEnvIf user-agent "Indy Library" stayout=1
SetEnvIf user-agent "libwww-perl" stayout=1
SetEnvIf user-agent "Go!Zilla" stayout=1
SetEnvIf user-agent "TurnitinBot" stayout=1
<IfModule !mod_authz_core.c>
deny from env=stayout
</IfModule>
<IfModule mod_authz_core.c>
  <RequireAll>
    Require all granted
    Require not env stayout
  </RequireAll>
</IfModule>
##### Common hacking tools and bandwidth hoggers block -- END

##### Automatic compression of resources -- BEGIN
<ifmodule mod_deflate.c>
AddOutputFilterByType DEFLATE text/plain text/xml text/css application/xml application/xhtml+xml application/rss+xml application/javascript application/x-javascript
</ifmodule>
<ifModule mod_gzip.c>
mod_gzip_on Yes
mod_gzip_dechunk Yes
mod_gzip_keep_workfiles No
mod_gzip_can_negotiate Yes
mod_gzip_add_header_count Yes
mod_gzip_send_vary Yes
mod_gzip_min_http 1000
mod_gzip_minimum_file_size 300
mod_gzip_maximum_file_size 512000
mod_gzip_maximum_inmem_size 60000
mod_gzip_handle_methods GET
mod_gzip_item_include file \.(html?|txt|css|js|php|pl|xml|rb|py)$
mod_gzip_item_include mime ^text/plain$
mod_gzip_item_include mime ^text/xml$
mod_gzip_item_include mime ^text/css$
mod_gzip_item_include mime ^application/xml$
mod_gzip_item_include mime ^application/xhtml+xml$
mod_gzip_item_include mime ^application/rss+xml$
mod_gzip_item_include mime ^application/javascript$
mod_gzip_item_include mime ^application/x-javascript$
mod_gzip_item_exclude rspheader ^Content-Encoding:.*gzip.*
mod_gzip_item_include handler ^cgi-script$
mod_gzip_item_include handler ^server-status$
mod_gzip_item_include handler ^server-info$
mod_gzip_item_include handler ^application/x-httpd-php
mod_gzip_item_exclude mime ^image/.*
</ifmodule>
##### Automatic compression of resources -- END
##### Redirect index.php to / -- BEGIN
RewriteCond %{THE_REQUEST} !^POST
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /index\.php\ HTTP/
RewriteCond %{SERVER_PORT}>s ^(443>(s)|[0-9]+>s)$
RewriteRule ^index\.php$ http%2://www.quebecguitare.ca/ [R=301,L]
##### Redirect index.php to / -- END
##### Redirect non-www to www -- BEGIN
RewriteCond %{HTTP_HOST} !^www\. [NC]
RewriteRule ^(.*)$ http://www.%{HTTP_HOST}/$1 [R=301,L]
##### Redirect non-www to www -- END

##### Rewrite rules to block out some common exploits -- BEGIN
RewriteCond %{QUERY_STRING} proc/self/environ [OR]
RewriteCond %{QUERY_STRING} mosConfig_[a-zA-Z_]{1,21}(=|\%3D) [OR]
RewriteCond %{QUERY_STRING} base64_(en|de)code\(.*\) [OR]
RewriteCond %{QUERY_STRING} (<|%3C).*script.*(>|%3E) [NC,OR]
RewriteCond %{QUERY_STRING} GLOBALS(=|\[|\[0-9A-Z]{0,2}) [OR]
RewriteCond %{QUERY_STRING} _REQUEST(=|\[|\[0-9A-Z]{0,2})
RewriteRule .* index.php [F]
##### Rewrite rules to block out some common exploits -- END
##### File injection protection -- BEGIN
RewriteCond %{REQUEST_METHOD} GET
RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=http:// [OR]
RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=(\.\.//?)+ [OR]
RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=/([a-z0-9_.]//?)+ [NC]
RewriteRule .* - [F]
##### File injection protection -- END

##### Advanced server protection rules exceptions -- BEGIN
RewriteRule ^administrator\/components\/com_akeeba\/restore\.php$ - [L]
RewriteRule ^administrator\/components\/com_admintools\/restore\.php$ - [L]
RewriteRule ^administrator\/components\/com_joomlaupdate\/restore\.php$ - [L]
RewriteRule ^templates\/gk_john_s/ - [L]
##### Advanced server protection rules exceptions -- END

##### Advanced server protection -- BEGIN

RewriteCond %{QUERY_STRING} \=PHP[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12} [NC]
RewriteRule .* - [F]
## Back-end protection
RewriteRule ^administrator/?$ - [L]
RewriteRule ^administrator/index\.(php|html?)$ - [L]
RewriteRule ^administrator/index[23]\.php$ - [L]
RewriteRule ^administrator/(components|modules|templates|images|plugins)/.*\.(jpe|jpg|jpeg|jp2|jpe2|png|gif|bmp|css|js|swf|html|mpg|mp3|mpeg|mp4|avi|wav|ogg|ogv|xls|xlsx|doc|docx|ppt|pptx|zip|rar|pdf|xps|txt|7z|svg|odt|ods|odp|flv|mov|htm|ttf|woff|eot|JPG|JPEG|PNG|GIF|CSS|JS|TTF|WOFF|EOT)$ - [L]
RewriteRule ^administrator/ - [F]
## Allow limited access for certain Joomla! system directories with client-accessible content
RewriteRule ^(components|modules|templates|images|plugins|media|libraries|media/jui/fonts)/.*\.(jpe|jpg|jpeg|jp2|jpe2|png|gif|bmp|css|js|swf|html|mpg|mp3|mpeg|mp4|avi|wav|ogg|ogv|xls|xlsx|doc|docx|ppt|pptx|zip|rar|pdf|xps|txt|7z|svg|odt|ods|odp|flv|mov|ico|htm|ttf|woff|eot|JPG|JPEG|PNG|GIF|CSS|JS|TTF|WOFF|EOT)$ - [L]
RewriteRule ^(components|modules|templates|images|plugins|media|libraries|media/jui/fonts)/ - [F]
## Disallow front-end access for certain Joomla! system directories (unless access to their files is allowed above)
RewriteRule ^includes/js/ - [L]
RewriteRule ^(cache|includes|language|logs|log|tmp)/ - [F]
RewriteRule ^(configuration\.php|CONTRIBUTING\.md|htaccess\.txt|joomla\.xml|LICENSE\.txt|phpunit\.xml|README\.txt|web\.config\.txt) - [F]

## Disallow access to rogue PHP files throughout the site, unless they are explicitly allowed
RewriteCond %{REQUEST_FILENAME} (\.php)$
RewriteCond %{REQUEST_FILENAME} !(/index[23]?\.php)$
RewriteCond %{REQUEST_FILENAME} -f
RewriteRule (.*\.php)$ - [F]
## Disallow access to htaccess.txt, php.ini and configuration.php-dist
RewriteRule ^(htaccess\.txt|configuration\.php-dist|php\.ini)$ - [F]
##### Advanced server protection -- END

##### Joomla! core SEF Section -- BEGIN
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
RewriteCond %{REQUEST_URI} !^/index\.php
RewriteCond %{REQUEST_URI} /component/|(/[^.]*|\.(php|html?|feed|pdf|raw|ini|zip|json|file|vcf))$ [NC]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule .* index.php [L]
##### Joomla! core SEF Section -- END





Thank you,
Regards

quebecguitare
There is this link too that needs to be added:

http://www.quebecguitare.ca/index.php?option=com_jmap&view=sitemap&format=rss&xslt=1

tampe125
Akeeba Staff
Hello Alexandre,

I can access those links just fine, this means that the section Common hacking tools and bandwidth hoggers block is blocking your search engine.
You have to disable that feature or remove the user agents you don't want to block.

Davide Tampellini

Developer and Support Staff

🇮🇹Italian: native 🇬🇧English: good • 🕐 My time zone is Europe / Rome (UTC +1)
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

quebecguitare
Thank you Davide.

It seems to have worked. I can access the links now.
What do you mean by "is blocking your search engine"? Is this a typo?

Does this mean I have an exploit agent on my computer or this is due to my internet provider?
Do you have any suggestions on which one to remove or troubleshoot this instead of removing them all like I just did?

Regards

Section removed from .htaccess


##### Common hacking tools and bandwidth hoggers block -- BEGIN
SetEnvIf user-agent "WebBandit" stayout=1
SetEnvIf user-agent "webbandit" stayout=1
SetEnvIf user-agent "Acunetix" stayout=1
SetEnvIf user-agent "binlar" stayout=1
SetEnvIf user-agent "BlackWidow" stayout=1
SetEnvIf user-agent "Bolt 0" stayout=1
SetEnvIf user-agent "Bot mailto:[email protected]" stayout=1
SetEnvIf user-agent "BOT for JCE" stayout=1
SetEnvIf user-agent "casper" stayout=1
SetEnvIf user-agent "checkprivacy" stayout=1
SetEnvIf user-agent "ChinaClaw" stayout=1
SetEnvIf user-agent "clshttp" stayout=1
SetEnvIf user-agent "cmsworldmap" stayout=1
SetEnvIf user-agent "comodo" stayout=1
SetEnvIf user-agent "Custo" stayout=1
SetEnvIf user-agent "Default Browser 0" stayout=1
SetEnvIf user-agent "diavol" stayout=1
SetEnvIf user-agent "DIIbot" stayout=1
SetEnvIf user-agent "DISCo" stayout=1
SetEnvIf user-agent "dotbot" stayout=1
SetEnvIf user-agent "Download Demon" stayout=1
SetEnvIf user-agent "eCatch" stayout=1
SetEnvIf user-agent "EirGrabber" stayout=1
SetEnvIf user-agent "EmailCollector" stayout=1
SetEnvIf user-agent "EmailSiphon" stayout=1
SetEnvIf user-agent "EmailWolf" stayout=1
SetEnvIf user-agent "Express WebPictures" stayout=1
SetEnvIf user-agent "extract" stayout=1
SetEnvIf user-agent "ExtractorPro" stayout=1
SetEnvIf user-agent "EyeNetIE" stayout=1
SetEnvIf user-agent "feedfinder" stayout=1
SetEnvIf user-agent "FHscan" stayout=1
SetEnvIf user-agent "FlashGet" stayout=1
SetEnvIf user-agent "flicky" stayout=1
SetEnvIf user-agent "GetRight" stayout=1
SetEnvIf user-agent "GetWeb!" stayout=1
SetEnvIf user-agent "Go-Ahead-Got-It" stayout=1
SetEnvIf user-agent "Go!Zilla" stayout=1
SetEnvIf user-agent "grab" stayout=1
SetEnvIf user-agent "GrabNet" stayout=1
SetEnvIf user-agent "Grafula" stayout=1
SetEnvIf user-agent "harvest" stayout=1
SetEnvIf user-agent "HMView" stayout=1
SetEnvIf user-agent "ia_archiver" stayout=1
SetEnvIf user-agent "Image Stripper" stayout=1
SetEnvIf user-agent "Image Sucker" stayout=1
SetEnvIf user-agent "InterGET" stayout=1
SetEnvIf user-agent "Internet Ninja" stayout=1
SetEnvIf user-agent "InternetSeer.com" stayout=1
SetEnvIf user-agent "jakarta" stayout=1
SetEnvIf user-agent "Java" stayout=1
SetEnvIf user-agent "JetCar" stayout=1
SetEnvIf user-agent "JOC Web Spider" stayout=1
SetEnvIf user-agent "kmccrew" stayout=1
SetEnvIf user-agent "larbin" stayout=1
SetEnvIf user-agent "LeechFTP" stayout=1
SetEnvIf user-agent "libwww" stayout=1
SetEnvIf user-agent "Mass Downloader" stayout=1
SetEnvIf user-agent "Maxthon$" stayout=1
SetEnvIf user-agent "microsoft.url" stayout=1
SetEnvIf user-agent "MIDown tool" stayout=1
SetEnvIf user-agent "miner" stayout=1
SetEnvIf user-agent "Mister PiX" stayout=1
SetEnvIf user-agent "NEWT" stayout=1
SetEnvIf user-agent "MSFrontPage" stayout=1
SetEnvIf user-agent "Navroad" stayout=1
SetEnvIf user-agent "NearSite" stayout=1
SetEnvIf user-agent "Net Vampire" stayout=1
SetEnvIf user-agent "NetAnts" stayout=1
SetEnvIf user-agent "NetSpider" stayout=1
SetEnvIf user-agent "NetZIP" stayout=1
SetEnvIf user-agent "nutch" stayout=1
SetEnvIf user-agent "Octopus" stayout=1
SetEnvIf user-agent "Offline Explorer" stayout=1
SetEnvIf user-agent "Offline Navigator" stayout=1
SetEnvIf user-agent "PageGrabber" stayout=1
SetEnvIf user-agent "Papa Foto" stayout=1
SetEnvIf user-agent "pavuk" stayout=1
SetEnvIf user-agent "pcBrowser" stayout=1
SetEnvIf user-agent "PeoplePal" stayout=1
SetEnvIf user-agent "planetwork" stayout=1
SetEnvIf user-agent "psbot" stayout=1
SetEnvIf user-agent "purebot" stayout=1
SetEnvIf user-agent "pycurl" stayout=1
SetEnvIf user-agent "RealDownload" stayout=1
SetEnvIf user-agent "ReGet" stayout=1
SetEnvIf user-agent "Rippers 0" stayout=1
SetEnvIf user-agent "SeaMonkey$" stayout=1
SetEnvIf user-agent "sitecheck.internetseer.com" stayout=1
SetEnvIf user-agent "SiteSnagger" stayout=1
SetEnvIf user-agent "skygrid" stayout=1
SetEnvIf user-agent "SmartDownload" stayout=1
SetEnvIf user-agent "sucker" stayout=1
SetEnvIf user-agent "SuperBot" stayout=1
SetEnvIf user-agent "SuperHTTP" stayout=1
SetEnvIf user-agent "Surfbot" stayout=1
SetEnvIf user-agent "tAkeOut" stayout=1
SetEnvIf user-agent "Teleport Pro" stayout=1
SetEnvIf user-agent "Toata dragostea mea pentru diavola" stayout=1
SetEnvIf user-agent "turnit" stayout=1
SetEnvIf user-agent "vikspider" stayout=1
SetEnvIf user-agent "VoidEYE" stayout=1
SetEnvIf user-agent "Web Image Collector" stayout=1
SetEnvIf user-agent "Web Sucker" stayout=1
SetEnvIf user-agent "WebAuto" stayout=1
SetEnvIf user-agent "WebCopier" stayout=1
SetEnvIf user-agent "WebFetch" stayout=1
SetEnvIf user-agent "WebGo IS" stayout=1
SetEnvIf user-agent "WebLeacher" stayout=1
SetEnvIf user-agent "WebReaper" stayout=1
SetEnvIf user-agent "WebSauger" stayout=1
SetEnvIf user-agent "Website eXtractor" stayout=1
SetEnvIf user-agent "Website Quester" stayout=1
SetEnvIf user-agent "WebStripper" stayout=1
SetEnvIf user-agent "WebWhacker" stayout=1
SetEnvIf user-agent "WebZIP" stayout=1
SetEnvIf user-agent "Wget" stayout=1
SetEnvIf user-agent "Widow" stayout=1
SetEnvIf user-agent "WWW-Mechanize" stayout=1
SetEnvIf user-agent "WWWOFFLE" stayout=1
SetEnvIf user-agent "Xaldon WebSpider" stayout=1
SetEnvIf user-agent "Yandex" stayout=1
SetEnvIf user-agent "Zeus" stayout=1
SetEnvIf user-agent "zmeu" stayout=1
SetEnvIf user-agent "CazoodleBot" stayout=1
SetEnvIf user-agent "discobot" stayout=1
SetEnvIf user-agent "ecxi" stayout=1
SetEnvIf user-agent "GT::WWW" stayout=1
SetEnvIf user-agent "heritrix" stayout=1
SetEnvIf user-agent "HTTP::Lite" stayout=1
SetEnvIf user-agent "HTTrack" stayout=1
SetEnvIf user-agent "ia_archiver" stayout=1
SetEnvIf user-agent "id-search" stayout=1
SetEnvIf user-agent "id-search.org" stayout=1
SetEnvIf user-agent "IDBot" stayout=1
SetEnvIf user-agent "Indy Library" stayout=1
SetEnvIf user-agent "IRLbot" stayout=1
SetEnvIf user-agent "ISC Systems iRc Search 2.1" stayout=1
SetEnvIf user-agent "LinksManager.com_bot" stayout=1
SetEnvIf user-agent "linkwalker" stayout=1
SetEnvIf user-agent "lwp-trivial" stayout=1
SetEnvIf user-agent "MFC_Tear_Sample" stayout=1
SetEnvIf user-agent "Microsoft URL Control" stayout=1
SetEnvIf user-agent "Missigua Locator" stayout=1
SetEnvIf user-agent "panscient.com" stayout=1
SetEnvIf user-agent "PECL::HTTP" stayout=1
SetEnvIf user-agent "PHPCrawl" stayout=1
SetEnvIf user-agent "PleaseCrawl" stayout=1
SetEnvIf user-agent "SBIder" stayout=1
SetEnvIf user-agent "Snoopy" stayout=1
SetEnvIf user-agent "Steeler" stayout=1
SetEnvIf user-agent "URI::Fetch" stayout=1
SetEnvIf user-agent "urllib" stayout=1
SetEnvIf user-agent "Web Sucker" stayout=1
SetEnvIf user-agent "webalta" stayout=1
SetEnvIf user-agent "WebCollage" stayout=1
SetEnvIf user-agent "Wells Search II" stayout=1
SetEnvIf user-agent "WEP Search" stayout=1
SetEnvIf user-agent "zermelo" stayout=1
SetEnvIf user-agent "ZyBorg" stayout=1
SetEnvIf user-agent "Indy Library" stayout=1
SetEnvIf user-agent "libwww-perl" stayout=1
SetEnvIf user-agent "Go!Zilla" stayout=1
SetEnvIf user-agent "TurnitinBot" stayout=1
<IfModule !mod_authz_core.c>
deny from env=stayout
</IfModule>
<IfModule mod_authz_core.c>
  <RequireAll>
    Require all granted
    Require not env stayout
  </RequireAll>
</IfModule>
##### Common hacking tools and bandwidth hoggers block -- END

tampe125
Akeeba Staff
I meant "the search engine you are interested into". We currently have no issues with Bing and Google, what's the search engine you're having issues?
The best way is to remove all the lines and then re-add them one by one. If you can't fetch the sitemap, the last line you introduced is the blocking one.

Davide Tampellini

Developer and Support Staff

🇮🇹Italian: native 🇬🇧English: good • 🕐 My time zone is Europe / Rome (UTC +1)
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

quebecguitare
I am using google chrome. So these links I tested them using the right-click "go to address..." function.

So there is another problem, because according to what I understand, I would be using google if any at all, since I am pasting directly the URL. Am I even using a search engine when accessing an URL that way?

Thank you

nicholas
Akeeba Staff
Manager
It's complicated to explain, but site maps are not accessed directly by your browser. They go through a third party server first. Since you're using Google Chrome and you're signed in as a Google user they go through Google's servers. These servers use wither wget or cURL (or libcurl, the library version of the cURL utility). Since you were blocking access to your site by wget and curl they could not read the links, hence the problem.

For your information, when your ticket came in I did try accessing the same URL with Safari and it was loaded properly. That's why I instructed Davide to have you disable the bandwidth hoggers feature. I know it's all a bit over your head. You will just have to trust us, we know what we're doing :)

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

quebecguitare
I understand thank you for you answer and yes it's a bit over my head.

However now I tested the link again, with the same .htaccess file that I edited less than an hour ago and I cannot access these links again. However you are telling me that google or other user will be able to crawl these maps so I shouldn't be worried and put back all these agents blocking htaccess lines?

Thank you
Regards

quebecguitare
Shortly after removing these rules from the .htaccess file, I noticed that my error_log file in joomla root has grown exponentially to 27 meg.

Is this of any concerns? PHP injection attempts?

Regards


I currently have the following .htaccess setup:


### ===========================================================================
### Security Enhanced & Highly Optimized .htaccess File for Joomla!
### automatically generated by Admin Tools 3.3.1 on 2014-12-04 14:15:19 GMT
### Auto-detected Apache version: 2.5 (best guess)
### ===========================================================================
###
### The contents of this file are based on the same author's work "Master
### .htaccess", published on http://snipt.net/nikosdion/the-master-htaccess
###
### Admin Tools is Free Software, distributed under the terms of the GNU
### General Public License version 3 or, at your option, any later version
### published by the Free Software Foundation.
###
### !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! IMPORTANT !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
### !!                                                                       !!
### !!  If you get an Internal Server Error 500 or a blank page when trying  !!
### !!  to access your site, remove this file and try tweaking its settings  !!
### !!  in the back-end of the Admin Tools component.                        !!
### !!                                                                       !!
### !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
###

##### Set to PHP 5.4
AddHandler application/x-httpd-php54 .php .php5 .php4 .php3

##### Set server signature OFF

ServerSignature Off
##ServerTokens Prod

##### RewriteEngine enabled - BEGIN
RewriteEngine On
##### RewriteEngine enabled - END

##### RewriteBase set - BEGIN
RewriteBase /
##### RewriteBase set - END

##### File execution order -- BEGIN
DirectoryIndex index.php index.html
##### File execution order -- END

##### No directory listings -- BEGIN
IndexIgnore *
Options -Indexes
##### No directory listings -- END

##### Optimal default expiration time - BEGIN
<IfModule mod_expires.c>
	# Enable expiration control
	ExpiresActive On

	# Default expiration: 1 hour after request
	ExpiresDefault "now plus 1 hour"

	# CSS and JS expiration: 1 week after request
	ExpiresByType text/css "now plus 1 week"
	ExpiresByType application/javascript "now plus 1 week"
	ExpiresByType application/x-javascript "now plus 1 week"

	# Image files expiration: 1 month after request
	ExpiresByType image/bmp "now plus 1 month"
	ExpiresByType image/gif "now plus 1 month"
	ExpiresByType image/jpeg "now plus 1 month"
	ExpiresByType image/jp2 "now plus 1 month"
	ExpiresByType image/pipeg "now plus 1 month"
	ExpiresByType image/png "now plus 1 month"
	ExpiresByType image/svg+xml "now plus 1 month"
	ExpiresByType image/tiff "now plus 1 month"
	ExpiresByType image/vnd.microsoft.icon "now plus 1 month"
	ExpiresByType image/x-icon "now plus 1 month"
	ExpiresByType image/ico "now plus 1 month"
	ExpiresByType image/icon "now plus 1 month"
	ExpiresByType text/ico "now plus 1 month"
	ExpiresByType application/ico "now plus 1 month"
	ExpiresByType image/vnd.wap.wbmp "now plus 1 month"
	ExpiresByType application/vnd.wap.wbxml "now plus 1 month"
	ExpiresByType application/smil "now plus 1 month"

	# Audio files expiration: 1 month after request
	ExpiresByType audio/basic "now plus 1 month"
	ExpiresByType audio/mid "now plus 1 month"
	ExpiresByType audio/midi "now plus 1 month"
	ExpiresByType audio/mpeg "now plus 1 month"
	ExpiresByType audio/x-aiff "now plus 1 month"
	ExpiresByType audio/x-mpegurl "now plus 1 month"
	ExpiresByType audio/x-pn-realaudio "now plus 1 month"
	ExpiresByType audio/x-wav "now plus 1 month"

	# Movie files expiration: 1 month after request
	ExpiresByType application/x-shockwave-flash "now plus 1 month"
	ExpiresByType x-world/x-vrml "now plus 1 month"
	ExpiresByType video/x-msvideo "now plus 1 month"
	ExpiresByType video/mpeg "now plus 1 month"
	ExpiresByType video/mp4 "now plus 1 month"
	ExpiresByType video/quicktime "now plus 1 month"
	ExpiresByType video/x-la-asf "now plus 1 month"
	ExpiresByType video/x-ms-asf "now plus 1 month"
</IfModule>
##### Optimal default expiration time - END

##### Common hacking tools and bandwidth hoggers block -- BEGIN
SetEnvIf user-agent "WebBandit" stayout=1
SetEnvIf user-agent "webbandit" stayout=1
SetEnvIf user-agent "Acunetix" stayout=1
SetEnvIf user-agent "binlar" stayout=1
SetEnvIf user-agent "BlackWidow" stayout=1
SetEnvIf user-agent "Bolt 0" stayout=1
SetEnvIf user-agent "Bot mailto:[email protected]" stayout=1
SetEnvIf user-agent "BOT for JCE" stayout=1
SetEnvIf user-agent "casper" stayout=1
SetEnvIf user-agent "checkprivacy" stayout=1
SetEnvIf user-agent "ChinaClaw" stayout=1
SetEnvIf user-agent "clshttp" stayout=1
SetEnvIf user-agent "cmsworldmap" stayout=1
SetEnvIf user-agent "comodo" stayout=1
SetEnvIf user-agent "Custo" stayout=1
SetEnvIf user-agent "Default Browser 0" stayout=1
SetEnvIf user-agent "diavol" stayout=1
SetEnvIf user-agent "DIIbot" stayout=1
SetEnvIf user-agent "DISCo" stayout=1
SetEnvIf user-agent "dotbot" stayout=1
SetEnvIf user-agent "Download Demon" stayout=1
SetEnvIf user-agent "eCatch" stayout=1
SetEnvIf user-agent "EirGrabber" stayout=1
SetEnvIf user-agent "EmailCollector" stayout=1
SetEnvIf user-agent "EmailSiphon" stayout=1
SetEnvIf user-agent "EmailWolf" stayout=1
SetEnvIf user-agent "Express WebPictures" stayout=1
SetEnvIf user-agent "extract" stayout=1
SetEnvIf user-agent "ExtractorPro" stayout=1
SetEnvIf user-agent "EyeNetIE" stayout=1
SetEnvIf user-agent "feedfinder" stayout=1
SetEnvIf user-agent "FHscan" stayout=1
SetEnvIf user-agent "FlashGet" stayout=1
SetEnvIf user-agent "flicky" stayout=1
SetEnvIf user-agent "GetRight" stayout=1
SetEnvIf user-agent "GetWeb!" stayout=1
SetEnvIf user-agent "Go-Ahead-Got-It" stayout=1
SetEnvIf user-agent "Go!Zilla" stayout=1
SetEnvIf user-agent "grab" stayout=1
SetEnvIf user-agent "GrabNet" stayout=1
SetEnvIf user-agent "Grafula" stayout=1
SetEnvIf user-agent "harvest" stayout=1
SetEnvIf user-agent "HMView" stayout=1
SetEnvIf user-agent "ia_archiver" stayout=1
SetEnvIf user-agent "Image Stripper" stayout=1
SetEnvIf user-agent "Image Sucker" stayout=1
SetEnvIf user-agent "InterGET" stayout=1
SetEnvIf user-agent "Internet Ninja" stayout=1
SetEnvIf user-agent "InternetSeer.com" stayout=1
SetEnvIf user-agent "jakarta" stayout=1
SetEnvIf user-agent "Java" stayout=1
SetEnvIf user-agent "JetCar" stayout=1
SetEnvIf user-agent "JOC Web Spider" stayout=1
SetEnvIf user-agent "kmccrew" stayout=1
SetEnvIf user-agent "larbin" stayout=1
SetEnvIf user-agent "LeechFTP" stayout=1
SetEnvIf user-agent "libwww" stayout=1
SetEnvIf user-agent "Mass Downloader" stayout=1
SetEnvIf user-agent "Maxthon$" stayout=1
SetEnvIf user-agent "microsoft.url" stayout=1
SetEnvIf user-agent "MIDown tool" stayout=1
SetEnvIf user-agent "miner" stayout=1
SetEnvIf user-agent "Mister PiX" stayout=1
SetEnvIf user-agent "NEWT" stayout=1
SetEnvIf user-agent "MSFrontPage" stayout=1
SetEnvIf user-agent "Navroad" stayout=1
SetEnvIf user-agent "NearSite" stayout=1
SetEnvIf user-agent "Net Vampire" stayout=1
SetEnvIf user-agent "NetAnts" stayout=1
SetEnvIf user-agent "NetSpider" stayout=1
SetEnvIf user-agent "NetZIP" stayout=1
SetEnvIf user-agent "nutch" stayout=1
SetEnvIf user-agent "Octopus" stayout=1
SetEnvIf user-agent "Offline Explorer" stayout=1
SetEnvIf user-agent "Offline Navigator" stayout=1
SetEnvIf user-agent "PageGrabber" stayout=1
SetEnvIf user-agent "Papa Foto" stayout=1
SetEnvIf user-agent "pavuk" stayout=1
SetEnvIf user-agent "pcBrowser" stayout=1
SetEnvIf user-agent "PeoplePal" stayout=1
SetEnvIf user-agent "planetwork" stayout=1
SetEnvIf user-agent "psbot" stayout=1
SetEnvIf user-agent "purebot" stayout=1
SetEnvIf user-agent "pycurl" stayout=1
SetEnvIf user-agent "RealDownload" stayout=1
SetEnvIf user-agent "ReGet" stayout=1
SetEnvIf user-agent "Rippers 0" stayout=1
SetEnvIf user-agent "SeaMonkey$" stayout=1
SetEnvIf user-agent "sitecheck.internetseer.com" stayout=1
SetEnvIf user-agent "SiteSnagger" stayout=1
SetEnvIf user-agent "skygrid" stayout=1
SetEnvIf user-agent "SmartDownload" stayout=1
SetEnvIf user-agent "sucker" stayout=1
SetEnvIf user-agent "SuperBot" stayout=1
SetEnvIf user-agent "SuperHTTP" stayout=1
SetEnvIf user-agent "Surfbot" stayout=1
SetEnvIf user-agent "tAkeOut" stayout=1
SetEnvIf user-agent "Teleport Pro" stayout=1
SetEnvIf user-agent "Toata dragostea mea pentru diavola" stayout=1
SetEnvIf user-agent "turnit" stayout=1
SetEnvIf user-agent "vikspider" stayout=1
SetEnvIf user-agent "VoidEYE" stayout=1
SetEnvIf user-agent "Web Image Collector" stayout=1
SetEnvIf user-agent "Web Sucker" stayout=1
SetEnvIf user-agent "WebAuto" stayout=1
SetEnvIf user-agent "WebCopier" stayout=1
SetEnvIf user-agent "WebFetch" stayout=1
SetEnvIf user-agent "WebGo IS" stayout=1
SetEnvIf user-agent "WebLeacher" stayout=1
SetEnvIf user-agent "WebReaper" stayout=1
SetEnvIf user-agent "WebSauger" stayout=1
SetEnvIf user-agent "Website eXtractor" stayout=1
SetEnvIf user-agent "Website Quester" stayout=1
SetEnvIf user-agent "WebStripper" stayout=1
SetEnvIf user-agent "WebWhacker" stayout=1
SetEnvIf user-agent "WebZIP" stayout=1
SetEnvIf user-agent "Wget" stayout=1
SetEnvIf user-agent "Widow" stayout=1
SetEnvIf user-agent "WWW-Mechanize" stayout=1
SetEnvIf user-agent "WWWOFFLE" stayout=1
SetEnvIf user-agent "Xaldon WebSpider" stayout=1
SetEnvIf user-agent "Yandex" stayout=1
SetEnvIf user-agent "Zeus" stayout=1
SetEnvIf user-agent "zmeu" stayout=1
SetEnvIf user-agent "CazoodleBot" stayout=1
SetEnvIf user-agent "discobot" stayout=1
SetEnvIf user-agent "ecxi" stayout=1
SetEnvIf user-agent "GT::WWW" stayout=1
SetEnvIf user-agent "heritrix" stayout=1
SetEnvIf user-agent "HTTP::Lite" stayout=1
SetEnvIf user-agent "HTTrack" stayout=1
SetEnvIf user-agent "ia_archiver" stayout=1
SetEnvIf user-agent "id-search" stayout=1
SetEnvIf user-agent "id-search.org" stayout=1
SetEnvIf user-agent "IDBot" stayout=1
SetEnvIf user-agent "Indy Library" stayout=1
SetEnvIf user-agent "IRLbot" stayout=1
SetEnvIf user-agent "ISC Systems iRc Search 2.1" stayout=1
SetEnvIf user-agent "LinksManager.com_bot" stayout=1
SetEnvIf user-agent "linkwalker" stayout=1
SetEnvIf user-agent "lwp-trivial" stayout=1
SetEnvIf user-agent "MFC_Tear_Sample" stayout=1
SetEnvIf user-agent "Microsoft URL Control" stayout=1
SetEnvIf user-agent "Missigua Locator" stayout=1
SetEnvIf user-agent "panscient.com" stayout=1
SetEnvIf user-agent "PECL::HTTP" stayout=1
SetEnvIf user-agent "PHPCrawl" stayout=1
SetEnvIf user-agent "PleaseCrawl" stayout=1
SetEnvIf user-agent "SBIder" stayout=1
SetEnvIf user-agent "Snoopy" stayout=1
SetEnvIf user-agent "Steeler" stayout=1
SetEnvIf user-agent "URI::Fetch" stayout=1
SetEnvIf user-agent "urllib" stayout=1
SetEnvIf user-agent "Web Sucker" stayout=1
SetEnvIf user-agent "webalta" stayout=1
SetEnvIf user-agent "WebCollage" stayout=1
SetEnvIf user-agent "Wells Search II" stayout=1
SetEnvIf user-agent "WEP Search" stayout=1
SetEnvIf user-agent "zermelo" stayout=1
SetEnvIf user-agent "ZyBorg" stayout=1
SetEnvIf user-agent "Indy Library" stayout=1
SetEnvIf user-agent "libwww-perl" stayout=1
SetEnvIf user-agent "Go!Zilla" stayout=1
SetEnvIf user-agent "TurnitinBot" stayout=1
<IfModule !mod_authz_core.c>
deny from env=stayout
</IfModule>
<IfModule mod_authz_core.c>
  <RequireAll>
    Require all granted
    Require not env stayout
  </RequireAll>
</IfModule>
##### Common hacking tools and bandwidth hoggers block -- END

##### Automatic compression of resources -- BEGIN
<ifmodule mod_deflate.c>
AddOutputFilterByType DEFLATE text/plain text/xml text/css application/xml application/xhtml+xml application/rss+xml application/javascript application/x-javascript
</ifmodule>
<ifModule mod_gzip.c>
mod_gzip_on Yes
mod_gzip_dechunk Yes
mod_gzip_keep_workfiles No
mod_gzip_can_negotiate Yes
mod_gzip_add_header_count Yes
mod_gzip_send_vary Yes
mod_gzip_min_http 1000
mod_gzip_minimum_file_size 300
mod_gzip_maximum_file_size 512000
mod_gzip_maximum_inmem_size 60000
mod_gzip_handle_methods GET
mod_gzip_item_include file \.(html?|txt|css|js|php|pl|xml|rb|py)$
mod_gzip_item_include mime ^text/plain$
mod_gzip_item_include mime ^text/xml$
mod_gzip_item_include mime ^text/css$
mod_gzip_item_include mime ^application/xml$
mod_gzip_item_include mime ^application/xhtml+xml$
mod_gzip_item_include mime ^application/rss+xml$
mod_gzip_item_include mime ^application/javascript$
mod_gzip_item_include mime ^application/x-javascript$
mod_gzip_item_exclude rspheader ^Content-Encoding:.*gzip.*
mod_gzip_item_include handler ^cgi-script$
mod_gzip_item_include handler ^server-status$
mod_gzip_item_include handler ^server-info$
mod_gzip_item_include handler ^application/x-httpd-php
mod_gzip_item_exclude mime ^image/.*
</ifmodule>
##### Automatic compression of resources -- END
##### Redirect index.php to / -- BEGIN
RewriteCond %{THE_REQUEST} !^POST
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /index\.php\ HTTP/
RewriteCond %{SERVER_PORT}>s ^(443>(s)|[0-9]+>s)$
RewriteRule ^index\.php$ http%2://www.quebecguitare.ca/ [R=301,L]
##### Redirect index.php to / -- END
##### Redirect non-www to www -- BEGIN
RewriteCond %{HTTP_HOST} !^www\. [NC]
RewriteRule ^(.*)$ http://www.%{HTTP_HOST}/$1 [R=301,L]
##### Redirect non-www to www -- END

##### Rewrite rules to block out some common exploits -- BEGIN
RewriteCond %{QUERY_STRING} proc/self/environ [OR]
RewriteCond %{QUERY_STRING} mosConfig_[a-zA-Z_]{1,21}(=|\%3D) [OR]
RewriteCond %{QUERY_STRING} base64_(en|de)code\(.*\) [OR]
RewriteCond %{QUERY_STRING} (<|%3C).*script.*(>|%3E) [NC,OR]
RewriteCond %{QUERY_STRING} GLOBALS(=|\[|\[0-9A-Z]{0,2}) [OR]
RewriteCond %{QUERY_STRING} _REQUEST(=|\[|\[0-9A-Z]{0,2})
RewriteRule .* index.php [F]
##### Rewrite rules to block out some common exploits -- END
##### File injection protection -- BEGIN
RewriteCond %{REQUEST_METHOD} GET
RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=http:// [OR]
RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=(\.\.//?)+ [OR]
RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=/([a-z0-9_.]//?)+ [NC]
RewriteRule .* - [F]
##### File injection protection -- END

##### Advanced server protection rules exceptions -- BEGIN
RewriteRule ^administrator\/components\/com_akeeba\/restore\.php$ - [L]
RewriteRule ^administrator\/components\/com_admintools\/restore\.php$ - [L]
RewriteRule ^administrator\/components\/com_joomlaupdate\/restore\.php$ - [L]
RewriteRule ^templates\/gk_john_s/ - [L]
##### Advanced server protection rules exceptions -- END

##### Advanced server protection -- BEGIN

RewriteCond %{QUERY_STRING} \=PHP[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12} [NC]
RewriteRule .* - [F]
## Back-end protection
RewriteRule ^administrator/?$ - [L]
RewriteRule ^administrator/index\.(php|html?)$ - [L]
RewriteRule ^administrator/index[23]\.php$ - [L]
RewriteRule ^administrator/(components|modules|templates|images|plugins)/.*\.(jpe|jpg|jpeg|jp2|jpe2|png|gif|bmp|css|js|swf|html|mpg|mp3|mpeg|mp4|avi|wav|ogg|ogv|xls|xlsx|doc|docx|ppt|pptx|zip|rar|pdf|xps|txt|7z|svg|odt|ods|odp|flv|mov|htm|ttf|woff|eot|JPG|JPEG|PNG|GIF|CSS|JS|TTF|WOFF|EOT)$ - [L]
RewriteRule ^administrator/ - [F]
## Allow limited access for certain Joomla! system directories with client-accessible content
RewriteRule ^(components|modules|templates|images|plugins|media|libraries|media/jui/fonts)/.*\.(jpe|jpg|jpeg|jp2|jpe2|png|gif|bmp|css|js|swf|html|mpg|mp3|mpeg|mp4|avi|wav|ogg|ogv|xls|xlsx|doc|docx|ppt|pptx|zip|rar|pdf|xps|txt|7z|svg|odt|ods|odp|flv|mov|ico|htm|ttf|woff|eot|JPG|JPEG|PNG|GIF|CSS|JS|TTF|WOFF|EOT)$ - [L]
RewriteRule ^(components|modules|templates|images|plugins|media|libraries|media/jui/fonts)/ - [F]
## Disallow front-end access for certain Joomla! system directories (unless access to their files is allowed above)
RewriteRule ^includes/js/ - [L]
RewriteRule ^(cache|includes|language|logs|log|tmp)/ - [F]
RewriteRule ^(configuration\.php|CONTRIBUTING\.md|htaccess\.txt|joomla\.xml|LICENSE\.txt|phpunit\.xml|README\.txt|web\.config\.txt) - [F]

## Disallow access to rogue PHP files throughout the site, unless they are explicitly allowed
RewriteCond %{REQUEST_FILENAME} (\.php)$
RewriteCond %{REQUEST_FILENAME} !(/index[23]?\.php)$
RewriteCond %{REQUEST_FILENAME} -f
RewriteRule (.*\.php)$ - [F]
## Disallow access to htaccess.txt, php.ini and configuration.php-dist
RewriteRule ^(htaccess\.txt|configuration\.php-dist|php\.ini)$ - [F]
##### Advanced server protection -- END

##### Joomla! core SEF Section -- BEGIN
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
RewriteCond %{REQUEST_URI} !^/index\.php
RewriteCond %{REQUEST_URI} /component/|(/[^.]*|\.(php|html?|feed|pdf|raw|ini|zip|json|file|vcf))$ [NC]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule .* index.php [L]
##### Joomla! core SEF Section -- END







Here is the complete error_log file from joomla (the log is 27 meg for this attack alone so...):

These 4 entries for about 80000 lines of error log.

[30-Nov-2014 23:35:27 America/Chicago] PHP Notice:  Use of undefined constant JAUTHENTICATE_STATUS_FAILURE - assumed 'JAUTHENTICATE_STATUS_FAILURE' in /home2/guitar88/public_html/_quebecguitareca/plugins/authentication/facebook/facebook.php on line 100
[01-Dec-2014 00:49:07 America/Chicago] PHP Notice:  Undefined property: stdClass::$author in /home2/guitar88/public_html/_quebecguitareca/templates/gk_john_s/html/com_k2/templates/suivi_k2/category_item.php on line 18

[01-Dec-2014 22:54:52 America/Chicago] PHP Notice:  Undefined offset: 1 in /home2/guitar88/public_html/_quebecguitareca/templates/gk_john_s/html/pagination.php on line 34 

[01-Dec-2014 00:59:00 America/Chicago] PHP Warning:  Division by zero in /home2/guitar88/public_html/_quebecguitareca/components/com_contushdvideoshare/views/player/tmpl/default_player.php on line 236 

[01-Dec-2014 07:03:14 America/Chicago] PHP Warning:  Division by zero in /home2/guitar88/public_html/_quebecguitareca/templates/videoplus/html/com_contushdvideoshare/player/default_player.php on line 144 

tampe125
Akeeba Staff
The errors you are seeing are not related to our software: if you take a look at the paths, they are caused by:
  • Facebook authentication plugin
  • Your template
  • Component com_contushdvideoshare
  • Videoplus templates

Most likely a search engine is indexing all the pages ofyour site, raising all those errors.

Regarding the crawling, you can easily try with another browser, for example Firefox. I just used it and those sitemap links are working just fine.

Finally, please avoid pasting the whole contents of the .htaccess, it turns ticket very long and hard to read ;)

Davide Tampellini

Developer and Support Staff

🇮🇹Italian: native 🇬🇧English: good • 🕐 My time zone is Europe / Rome (UTC +1)
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Support Information

Working hours: We are open Monday to Friday, 9am to 7pm Cyprus timezone (EET / EEST). Support is provided by the same developers writing the software, all of which live in Europe. You can still file tickets outside of our working hours, but we cannot respond to them until we're back at the office.

Support policy: We would like to kindly inform you that when using our support you have already agreed to the Support Policy which is part of our Terms of Service. Thank you for your understanding and for helping us help you!