Robots.txt optimisation
When we first wrote this post back in 2011 we outlined best practice concerning the robots.txt file, blocking un-relavant parts of file structures to prevent search engines from indexing certain parts of a site, this was considered best practice until late 2014 when Google released the Fetch and Render Tool, from this point blocking parts of your site such as the theme and plugin folders can have negative effects on your site. Google now give guidelines to allow bots to crawl parts of the site which contain CSS, Scripting and anything else which can alter the appearance of a website to a user.
As a result of these we generally leave the robots.txt to bare basics and adjust it per situation, but for reference here is our base robots.txt which blocks some resource hogs, scanners and potentially bad bots.
User-agent: * Disallow: /cgi-bin/ Sitemap: https://www.YOURDOMAIN.co.uk/sitemap_index.xml User-agent: MJ12bot Disallow: / User-agent: Yandex Disallow: / User-agent: moget User-agent: ichiro Disallow: / User-agent: NaverBot User-agent: Yeti Disallow: / User-agent: Baiduspider User-agent: Baiduspider-video User-agent: Baiduspider-image Disallow: / User-agent: sogou spider Disallow: / User-agent: YoudaoBot Disallow: / user-agent: AhrefsBot disallow: /
Redirect non Www to Www. :
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_HOST} ^yourdomain.com
RewriteRule (.*) http://www.yourdomain.com/$1 [R=301,L]
Redirect Www to non Www :
RewriteEngine On RewriteBase / RewriteCond % ^www.yourdomain.com [NC] RewriteRule ^(.*)$ http://yourdomain.com/$1 [L,R=301]
3. Protecting your .htaccess file
You really dont want people looking at your .htaccess file so use this to block them.
order allow,deny deny from all
Redirect Dedicated IP to domain ( duplicate content fix )
# IP TO DOMAIN REDIRECT
RewriteCond %{HTTP_HOST} ^211\.122\.10\.10$
RewriteRule ^(.*)$ https://www.yourdomain.co.uk/$1 [L,R=301]
Redirect http to https/SSL
RewriteEngine On
RewriteCond %{HTTPS} !=on
RewriteRule ^(.*) https://%{SERVER_NAME}/$1 [R,L]
Single Page 301 htaccess redirect
Redirect 301 /oldfileorurl.php https://www.domain.co.uk/new-page
Redirect entire directory and child URLs to one page or domain
redirectMatch 301 ^/old-directory/ https://www.newdomain.co.uk/new-directory
Enable Gzip Compression
AddOutputFilterByType DEFLATE text/plain AddOutputFilterByType DEFLATE text/html AddOutputFilterByType DEFLATE text/xml AddOutputFilterByType DEFLATE text/css AddOutputFilterByType DEFLATE application/xml AddOutputFilterByType DEFLATE application/xhtml+xml AddOutputFilterByType DEFLATE application/rss+xml AddOutputFilterByType DEFLATE application/javascript AddOutputFilterByType DEFLATE application/x-javascript AddOutputFilterByType DEFLATE application/x-httpd-php AddOutputFilterByType DEFLATE application/x-httpd-fastphp AddOutputFilterByType DEFLATE image/svg+xml SetOutputFilter DEFLATE
Expires Header caching (Leverage Browser Caching)
ExpiresActive On ExpiresByType image/jpg "access 2 week" ExpiresByType image/jpeg "access 2 week" ExpiresByType image/gif "access 2 week" ExpiresByType image/png "access 2 week" ExpiresByType text/css "access 2 week" ExpiresByType application/pdf "access 2 week" ExpiresByType text/x-javascript "access 2 week" ExpiresByType application/x-shockwave-flash "access 2 week" ExpiresByType image/x-icon "access 2 week" ExpiresDefault "access 2 week"
Prevent directory listing / Browsing
IndexIgnore *
WordPress Hardening
Below are a couple of htaccess additions which will help harden a WordPress based website.
Protect wp-config.php
order allow,deny deny from all
Secure / Harden wordpress includes folder
RewriteEngine On RewriteBase / RewriteRule ^wp-admin/includes/ - [F,L] RewriteRule !^wp-includes/ - [S=3] RewriteRule ^wp-includes/[^/]+\.php$ - [F,L] RewriteRule ^wp-includes/js/tinymce/langs/.+\.php - [F,L] RewriteRule ^wp-includes/theme-compat/ - [F,L]
