Robots.txt optimisation
When we first wrote this post back in 2011 we outlined best practice concerning the robots.txt file, blocking un-relavant parts of file structures to prevent search engines from indexing certain parts of a site, this was considered best practice until late 2014 when Google released the Fetch and Render Tool, from this point blocking parts of your site such as the theme and plugin folders can have negative effects on your site. Google now give guidelines to allow bots to crawl parts of the site which contain CSS, Scripting and anything else which can alter the appearance of a website to a user.
As a result of these we generally leave the robots.txt to bare basics and adjust it per situation, but for reference here is our base robots.txt which blocks some resource hogs, scanners and potentially bad bots.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | User-agent: * Disallow: /cgi-bin/ Sitemap: https://www.YOURDOMAIN.co.uk/sitemap_index.xml User-agent: MJ12bot Disallow: / User-agent: Yandex Disallow: / User-agent: moget User-agent: ichiro Disallow: / User-agent: NaverBot User-agent: Yeti Disallow: / User-agent: Baiduspider User-agent: Baiduspider-video User-agent: Baiduspider-image Disallow: / User-agent: sogou spider Disallow: / User-agent: YoudaoBot Disallow: / user-agent: AhrefsBot disallow: / |
Redirect non Www to Www. :
1 2 3 4 5 | RewriteEngine On RewriteBase / RewriteCond %{HTTP_HOST} ^yourdomain.com RewriteRule (.*) http://www.yourdomain.com/$1 [R=301,L] |
Redirect Www to non Www :
1 2 3 4 5 | RewriteEngine On RewriteBase / RewriteCond % ^www.yourdomain.com [NC] RewriteRule ^(.*)$ http://yourdomain.com/$1 [L,R=301] |
3. Protecting your .htaccess file
You really dont want people looking at your .htaccess file so use this to block them.
1 2 3 | <Files .htaccess> order allow,deny deny from all </Files> |
Redirect Dedicated IP to domain ( duplicate content fix )
1 2 3 | # IP TO DOMAIN REDIRECT RewriteCond %{HTTP_HOST} ^211\.122\.10\.10$ RewriteRule ^(.*)$ https://www.yourdomain.co.uk/$1 [L,R=301] |
Redirect http to https/SSL
1 2 3 | RewriteEngine On RewriteCond %{HTTPS} !=on RewriteRule ^(.*) https://%{SERVER_NAME}/$1 [R,L] |
Single Page 301 htaccess redirect
1 | Redirect 301 /oldfileorurl.php https://www.domain.co.uk/new-page |
Redirect entire directory and child URLs to one page or domain
1 | redirectMatch 301 ^/old-directory/ https://www.newdomain.co.uk/new-directory |
Enable Gzip Compression
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | <IfModule mod_deflate.c> AddOutputFilterByType DEFLATE text/plain AddOutputFilterByType DEFLATE text/html AddOutputFilterByType DEFLATE text/xml AddOutputFilterByType DEFLATE text/css AddOutputFilterByType DEFLATE application/xml AddOutputFilterByType DEFLATE application/xhtml+xml AddOutputFilterByType DEFLATE application/rss+xml AddOutputFilterByType DEFLATE application/javascript AddOutputFilterByType DEFLATE application/x-javascript AddOutputFilterByType DEFLATE application/x-httpd-php AddOutputFilterByType DEFLATE application/x-httpd-fastphp AddOutputFilterByType DEFLATE image/svg+xml SetOutputFilter DEFLATE </IfModule> |
Expires Header caching (Leverage Browser Caching)
1 2 3 4 5 6 7 8 9 10 11 12 13 | <IfModule mod_expires.c> ExpiresActive On ExpiresByType image/jpg "access 2 week" ExpiresByType image/jpeg "access 2 week" ExpiresByType image/gif "access 2 week" ExpiresByType image/png "access 2 week" ExpiresByType text/css "access 2 week" ExpiresByType application/pdf "access 2 week" ExpiresByType text/x-javascript "access 2 week" ExpiresByType application/x-shockwave-flash "access 2 week" ExpiresByType image/x-icon "access 2 week" ExpiresDefault "access 2 week" </IfModule> |
Prevent directory listing / Browsing
1 | IndexIgnore * |
WordPress Hardening
Below are a couple of htaccess additions which will help harden a WordPress based website.
Protect wp-config.php
1 2 3 4 | <files wp-config.php> order allow,deny deny from all </files> |
Secure / Harden wordpress includes folder
1 2 3 4 5 6 7 8 9 | <IfModule mod_rewrite.c> RewriteEngine On RewriteBase / RewriteRule ^wp-admin/includes/ - [F,L] RewriteRule !^wp-includes/ - [S=3] RewriteRule ^wp-includes/[^/]+\.php$ - [F,L] RewriteRule ^wp-includes/js/tinymce/langs/.+\.php - [F,L] RewriteRule ^wp-includes/theme-compat/ - [F,L] </IfModule> |
