<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>RivMedia &#187; Search Engine Optimisation</title>
	<atom:link href="http://www.rivmedia.co.uk/category/seo/feed" rel="self" type="application/rss+xml" />
	<link>http://www.rivmedia.co.uk</link>
	<description>Digital Marketing Agency</description>
	<lastBuildDate>Sun, 25 Mar 2012 00:15:46 +0000</lastBuildDate>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<generator>http://wordpress.org/?v=3.3.2</generator>
<xhtml:meta xmlns:xhtml="http://www.w3.org/1999/xhtml" name="robots" content="noindex" />
		<item>
		<title>Top tips for .htaccess and Robots.txt Part 1</title>
		<link>http://www.rivmedia.co.uk/top-tips-for-htaccess-and-robots-txt-part-1/627</link>
		<comments>http://www.rivmedia.co.uk/top-tips-for-htaccess-and-robots-txt-part-1/627#comments</comments>
		<pubDate>Sun, 16 Jan 2011 20:00:11 +0000</pubDate>
		<dc:creator>admin</dc:creator>
				<category><![CDATA[Search Engine Optimisation]]></category>
		<category><![CDATA[Web Spam]]></category>
		<category><![CDATA[Bandwidth]]></category>
		<category><![CDATA[Blackwidow]]></category>
		<category><![CDATA[Bots]]></category>
		<category><![CDATA[Canonicalization]]></category>
		<category><![CDATA[Choices]]></category>
		<category><![CDATA[Custo]]></category>
		<category><![CDATA[Email]]></category>
		<category><![CDATA[Email Addresses]]></category>
		<category><![CDATA[File Extensions]]></category>
		<category><![CDATA[Forum Script]]></category>
		<category><![CDATA[Htaccess File]]></category>
		<category><![CDATA[Htaccess Files]]></category>
		<category><![CDATA[Matt Cutts]]></category>
		<category><![CDATA[Mp3]]></category>
		<category><![CDATA[Rewr]]></category>
		<category><![CDATA[Rewritecond]]></category>
		<category><![CDATA[Robots]]></category>
		<category><![CDATA[Robots Txt]]></category>
		<category><![CDATA[Unnecessary Bandwidth]]></category>
		<category><![CDATA[Vbulletin]]></category>
		<category><![CDATA[Visa]]></category>
		<category><![CDATA[Web Crawlers]]></category>
		<category><![CDATA[Wordpress]]></category>
		<category><![CDATA[Yahoo]]></category>
		<category><![CDATA[Yourdomain]]></category>

		<guid isPermaLink="false">http://www.rivmedia.co.uk/?p=627</guid>
		<description><![CDATA[<p><p><a href="http://www.rivmedia.co.uk">RivMedia - Digital Marketing Agency</a></p><p>We often find with clients that the .htaccess and robots.txt file is most of the time left as standard or not used at all, This can not only cause your site to be using unnecessary bandwidth but can also expose it to scrapers, crawlers and &#8220;bad bots&#8221;. In this post we will go through a [...]</p></p><p><a href="http://www.rivmedia.co.uk/top-tips-for-htaccess-and-robots-txt-part-1/627">Top tips for .htaccess and Robots.txt Part 1</a> </p>]]></description>
			<content:encoded><![CDATA[<p><a href="http://www.rivmedia.co.uk">RivMedia - Digital Marketing Agency</a></p><p><a href="http://www.rivmedia.co.uk/top-tips-for-htaccess-and-robots-txt-part-1/627/badrobot" rel="attachment wp-att-639"><img src="http://www.rivmedia.co.uk/wp-content/uploads/2011/01/badrobot.jpg" alt="BadBots and the .htaccess" title="BadBots and the .htaccess" width="300" height="190" class="alignleft size-full wp-image-639" /></a>We often find with clients that the .htaccess and robots.txt file is most of the time left as standard or not used at all, This can not only cause your site to be using unnecessary bandwidth but can also expose it to scrapers, crawlers and &#8220;bad bots&#8221;. In this post we will go through a couple of useful hints and tips surrounding our favourite Blogging platform &#8220;wordpress&#8221; and our favorite Forum script &#8220;vbulletin&#8221; , that is not to say however that some of these things will not work on an ordinary site.I have commented below if a certain part is specifically for wordpress or vbulletin , the rest can be used across the board. This information can be found around the web</p>
<p>Lets start things off with the .htaccess file :</p>
<h2>1. Redirecting www. to a non www or visa versa</h2>
<p> : This is to ensure your site is not seen the same in two locations and is more for SEO or more specifically URL canonicalization.</p>
<p>What is URL canonicalization ? i hear you asking , URL canonicalization is what has been described by Matt Cutts Webspam team as <em>&#8220;the process of picking the best url when there are several choices</em>&#8221;</p>
<h2>Redirect non Www to Www. : </h2>
<div class='code_child'>RewriteEngine On</p>
<p>RewriteBase /</p>
<p>RewriteCond %{HTTP_HOST} ^yourdomain.com</p>
<p>RewriteRule (.*) http://www.yourdomain.com/$1 [R=301,L]</p></div>
<p></p>
<h2>Redirect Www to non Www : </h2>
<div class='code_child'>RewriteEngine On</p>
<p>RewriteBase /</p>
<p>RewriteCond % ^www.yourdomain.com [NC]</p>
<p>RewriteRule ^(.*)$ http://yourdomain.com/$1 [L,R=301]</p></div>
<p></p>
<h2>2. Filtering Out Badbots</h2>
<p> Bad bots are generally Web Crawlers which cause you and your site problems, From spamming , scaping ( stealing your content ) finding certain file extensions ( like Mp3&#8242;s ) and harvesting email addresses. This bad bot list should help fight at least some of them . Feel free to add to this list in the comment sections <img src='http://www.rivmedia.co.uk/wp-includes/images/smilies/icon_smile.gif' alt=':)' class='wp-smiley' /> </p>
<div class='code_child'>RewriteEngine On</p>
<p>RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:craftbot@yahoo.com [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Custo [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^DISCo [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^GetWeb! [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^HTTrack [NC,OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Image\ Stripper [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Image\ Sucker [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Indy\ Library [NC,OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^JOC\ Web\ Spider [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^larbin [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^NetZIP [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Navigator [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^pavuk [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^RealDownload [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^WebGo\ IS [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Website\ eXtractor [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Website\ Quester [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Widow [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^Zeus [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*WebZIP.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Iria.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Stripper.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Offline.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Copier.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Crawler.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Snagger.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Teleport.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Reaper.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Wget.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Grabber.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Sucker.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Downloader.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Siphon.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Collector.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Mag-Net.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Widow.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Pockey.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*DA.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Snake.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*BackWeb.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*gotit.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Vacuum.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*SmartDownload.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Pump.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*HMView.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Ninja.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*HTTrack.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*JOC.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*likse.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Memo.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*pcBrowser.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*SuperBot.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*leech.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Mirror.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Recorder.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*GrabNet.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Likse.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Navroad.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*attach.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Magnet.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Surfbot.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Bandit.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Ants.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Buddy.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Whacker.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*DISCo\Pump.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Drip.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*EirGrabber.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*ExtractorPro.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*EyeNetIE.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*FlashGet.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*GetRight.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Gets.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Go!Zilla.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Go-Ahead-Got-It.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Grafula.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*IBrowse.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*InterGET.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Internet\Ninja.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*JetCar.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*JustView.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*MIDown\tool.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Mister\PiX.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*NearSite.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*NetSpider.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Offline\Explorer.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*PageGrabber.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Papa\Foto.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Pockey.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*ReGet.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Slurp.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*SpaceBison.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*SuperHTTP.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Teleport.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*WebAuto.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Webcam\Watcher.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*WebCopier.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*WebFetch.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*WebReaper.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*FreeLoader.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Clint&#8217;s\Webcam.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*WebCam\Spy.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*CamEVU.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*iCamMaster.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Cam\Chaser\Pro.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*FlashIT.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*WebSauger.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*WebStripper.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*WebWhacker.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*WebZIP.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Web\Image\Collector.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Web\Sucker.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Webster.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*Wget.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*eCatch.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*ia_archiver.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*lftp.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*tAkeOut.*$ [OR]<br />
RewriteCond %{HTTP_USER_AGENT} ^.*FileHound.*$<br />
RewriteRule ^.* &#8211; [F,L]</p></div>
<p></p>
<h2>3. Blocking access to your .htaccess file</h2>
<p>. You really dont want people looking at your .htaccess file so use this to block them.</p>
<div class='code_child'>&lt;Files .htaccess&gt;</p>
<p>order allow,deny</p>
<p>deny from all  &lt;/Files&gt;</p></div>
<p></p>
<p>Now for the Robots.txt , The robots.txt is known as <em>The Robots Exclusion  Protocol</em>. Basically meaning it can be used to tell web robots not to crawl a certain part of the sites structure, Most importantly places like admin folders , not only will this save you bandwidth but will also save the bots from indexing potencially harmful folders.</p>
<h2>1. The best Robots.txt Document to use for wordpress blogs :</h2>
<p> Note &#8211; Should only be used with WordPress installs</p>
<div class='code_child'>User-agent: *</p>
<p>Disallow: /wp-content/</p>
<p>Disallow: /wp-icludes/</p>
<p>Disallow: /trackback/</p>
<p>Disallow: /wp-admin/</p>
<p>Disallow: /archives/</p>
<p>Disallow: /category/</p>
<p>Disallow: /tag/*</p>
<p>Disallow: /tag/</p>
<p>Disallow: /wp-*</p>
<p>Disallow: /login/</p>
<p>Disallow: /*.js$</p>
<p>Disallow: /*.inc$</p>
<p>Disallow: /*.css$</p>
<p>Disallow: /*.php$</p>
<p>User-agent: All</p>
<p>Allow: /</p>
<p>User-agent: Googlebot-Image</p>
<p>Disallow: /</p>
<p>User-agent: ia_archiver</p>
<p>Disallow: /</p>
<p>User-agent: duggmirror</p>
<p>Disallow: /</p></div>
<p></p>
<h2>2. The best Robots.txt documents to use with vbulletin Forums</h1>
<p> : Note &#8211; Should only be used with vbulletin installs</p>
<div class='code_child'>User-agent: *</p>
<p>Disallow: /cgi-bin/</p>
<p>Disallow: /admincp/</p>
<p>Disallow: /announcement.php</p>
<p>Disallow: /calendar.php</p>
<p>Disallow: /cron.php</p>
<p>Disallow: /editpost.php</p>
<p>Disallow: /faq.php</p>
<p>Disallow: /joinrequests.php</p>
<p>Disallow: /login.php</p>
<p>Disallow: /member.php</p>
<p>Disallow: /misc.php</p>
<p>Disallow: /modcp/</p>
<p>Disallow: /moderator.php</p>
<p>Disallow: /newreply.php</p>
<p>Disallow: /newthread.php</p>
<p>Disallow: /online.php</p>
<p>Disallow: /printthread.php</p>
<p>Disallow: /private.php</p>
<p>Disallow: /profile.php</p>
<p>Disallow: /register.php</p>
<p>Disallow: /search.php</p>
<p>Disallow: /sendmessage.php</p>
<p>Disallow: /showgroups.php</p>
<p>Disallow: /showpost.php</p>
<p>Disallow: /subscription.php</p>
<p>Disallow: /subscriptions.php</p>
<p>Disallow: /threadrate.php</p>
<p>Disallow: /usercp.php</p></div>
<p></p>
<p>Obviously you could expand on these no end , however we feel these are at very least the basic functions that every wordpress and vbulletin install should ship with.</p>
<p>We&#8217;d love to hear comments on other methods or code you guys use to protect and opimise your sites, this is the end of part 1 . However Stay subscribed for Part 2 to be posted in the near future with more hints and tips including hotlink protection and how to locate the badbots useragents from your sites log files.</p>
<p><a href="http://www.rivmedia.co.uk/top-tips-for-htaccess-and-robots-txt-part-1/627">Top tips for .htaccess and Robots.txt Part 1</a> </p>]]></content:encoded>
			<wfw:commentRss>http://www.rivmedia.co.uk/top-tips-for-htaccess-and-robots-txt-part-1/627/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Web CEO Review and Purchase</title>
		<link>http://www.rivmedia.co.uk/comparing-internet-business-promoter-vs-seo-powersuite-vs-web-ceo-reviews/309</link>
		<comments>http://www.rivmedia.co.uk/comparing-internet-business-promoter-vs-seo-powersuite-vs-web-ceo-reviews/309#comments</comments>
		<pubDate>Wed, 16 Dec 2009 14:09:48 +0000</pubDate>
		<dc:creator>admin</dc:creator>
				<category><![CDATA[Search Engine Optimisation]]></category>
		<category><![CDATA[buy]]></category>
		<category><![CDATA[Ceo Software]]></category>
		<category><![CDATA[Five Points]]></category>
		<category><![CDATA[Free Version]]></category>
		<category><![CDATA[Ftp Uploader]]></category>
		<category><![CDATA[Link Partner]]></category>
		<category><![CDATA[Link Popularity]]></category>
		<category><![CDATA[Market Leader]]></category>
		<category><![CDATA[Partner Finder]]></category>
		<category><![CDATA[Professional Edition]]></category>
		<category><![CDATA[purchase]]></category>
		<category><![CDATA[Purchase Web]]></category>
		<category><![CDATA[Quality Audit]]></category>
		<category><![CDATA[Research Search Engine]]></category>
		<category><![CDATA[review]]></category>
		<category><![CDATA[Search Engine Optimization]]></category>
		<category><![CDATA[Search Engine Submission]]></category>
		<category><![CDATA[Seo Software]]></category>
		<category><![CDATA[Software Package]]></category>
		<category><![CDATA[Submission Link]]></category>
		<category><![CDATA[Web Analytics]]></category>
		<category><![CDATA[Web Ceo]]></category>
		<category><![CDATA[Web Page Editor]]></category>
		<category><![CDATA[webceo]]></category>

		<guid isPermaLink="false">http://www.rivmedia.co.uk/?p=306</guid>
		<description><![CDATA[<p><p><a href="http://www.rivmedia.co.uk">RivMedia - Digital Marketing Agency</a></p><p>Web CEO Review If you are interested in what Web CEO has to offer, you do not have to look much further than the company’s homepage. It states, “Web CEO is the most complete SEO software package on the planet; plus, our SEO software offers more for free than any other software package or suite [...]</p></p><p><a href="http://www.rivmedia.co.uk/comparing-internet-business-promoter-vs-seo-powersuite-vs-web-ceo-reviews/309">Web CEO Review and Purchase</a> </p>]]></description>
			<content:encoded><![CDATA[<p><a href="http://www.rivmedia.co.uk">RivMedia - Digital Marketing Agency</a></p><h1>Web CEO Review</h1>
<p><a href="http://www.triphp.com/products/webceo.php"><br />
<img src="http://webceo.com/sp/images/key_125x125.gif" alt="" align="left" border="0" hspace="5" /></a>If you are interested in what <a href="http://www.triphp.com/products/webceo.php">Web CEO has to offer</a>, you do not have to look much further than the company’s homepage. It states, “Web CEO is the most complete SEO software package on the planet; plus, our SEO software offers more for free than any other software package or suite for SEO.”</p>
<p>As you can already tell, <a href="http://www.triphp.com/products/webceo.php">Web CEO</a> has positioned itself as a market leader. One of the ways they did this was by offering more for free than its competition. Obviously, people like anything that they can get free of charge. Although there is a paid version of Web CEO, with the free software you receive quite a few features that are sure to help you take your site to the next level.</p>
<p>There are five points that make Web CEO one of the best going today. They include:</p>
<p>1. Find your niche. This is done through keyword research, search engine optimization, and a web page editor. You never have to worry about finding a website niche when you rely on Web CEO.</p>
<p>2. <a href="http://www.triphp.com/products/webceo.php">Promote your site</a>. Don’t know anything about internet marketing? Again, Web CEO has you covered. It gives you tools for search engine submission, link partner finder, and PPC campaigns manager.</p>
<p>3. Analyze your site. You always need to be watching for changes to your site, both good and bad. The site analyzer offers a ranking checker, link popularity analyzer, and web analytics.</p>
<p>4. Maintain your site. This portion of the <a href="http://www.triphp.com/products/webceo.php">Web CEO</a> includes an FTP uploader, website quality audit, and website monitoring.</p>
<p>5. Professionalize. Web CEO software offers many features that will make your entire business run more smoothly. With a scheduler, dashboard, journal, and report sharing system you get everything you could possibly need.</p>
<p>As noted above, there is a full functional free version of Web CEO that is a great place to start. But when you are ready to upgrade and take advantage of all the top features, <a href="http://www.triphp.com/products/webceo.php">you will be asked to pay $199 for the SmallBiz Edition or $389 for the Professional Edition.</a></p>
<p>You should be able to see by now that <a href="http://www.triphp.com/products/webceo.php">Web CEO is a market leader in the SEO tool department. </a></p>
<p>Take a look at these three reviews to make up your mind :</p>
<ul>
<li><a href="http://www.rivmedia.co.uk/ibp-10-review-ibusiness-promoter-purchase/299">IBP (ibusiness Promoter) Review</a></li>
<li><a href="http://www.rivmedia.co.uk/seo-powersuite-review-link-assistant-purchase/303">SEO Powersuite Review</a></li>
<li><a href="http://www.rivmedia.co.uk/web-ceo-review-and-purchase/306">WebCEO Review </a></li>
<li><a href="http://www.rivmedia.co.uk/comparing-internet-business-promoter-vs-seo-powersuite-vs-web-ceo-reviews/309">Review of all 3 SEO Software</a></li>
</ul>
<p><a href="http://www.rivmedia.co.uk/comparing-internet-business-promoter-vs-seo-powersuite-vs-web-ceo-reviews/309">Web CEO Review and Purchase</a> </p>]]></content:encoded>
			<wfw:commentRss>http://www.rivmedia.co.uk/comparing-internet-business-promoter-vs-seo-powersuite-vs-web-ceo-reviews/309/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>SEO Powersuite Review &#8211; Link Assistant Purchase</title>
		<link>http://www.bestseosoftware.co</link>
		<comments>http://www.bestseosoftware.co#comments</comments>
		<pubDate>Wed, 16 Dec 2009 13:56:53 +0000</pubDate>
		<dc:creator>admin</dc:creator>
				<category><![CDATA[Search Engine Optimisation]]></category>
		<category><![CDATA[buy]]></category>
		<category><![CDATA[Fortune 500 Companies]]></category>
		<category><![CDATA[Free Updates]]></category>
		<category><![CDATA[Free Version]]></category>
		<category><![CDATA[Google]]></category>
		<category><![CDATA[Google Yahoo]]></category>
		<category><![CDATA[Guidance]]></category>
		<category><![CDATA[Next Level]]></category>
		<category><![CDATA[Popular Search Engines]]></category>
		<category><![CDATA[power]]></category>
		<category><![CDATA[Professional Support]]></category>
		<category><![CDATA[promoter]]></category>
		<category><![CDATA[Proper Steps]]></category>
		<category><![CDATA[purchase]]></category>
		<category><![CDATA[review]]></category>
		<category><![CDATA[Search Engine]]></category>
		<category><![CDATA[Site Rank]]></category>
		<category><![CDATA[suite]]></category>
		<category><![CDATA[Tons Of Money]]></category>
		<category><![CDATA[Yahoo]]></category>

		<guid isPermaLink="false">http://www.rivmedia.co.uk/?p=303</guid>
		<description><![CDATA[<p><p><a href="http://www.rivmedia.co.uk">RivMedia - Digital Marketing Agency</a></p><p>SEO Powersuite Review Becoming a search engine leader is not as difficult as you may think. In fact, you can take your site to the top of the most popular search engines in no time at all. Of course, if you want this to happen you need to take the proper steps. One of the [...]</p></p><p><a href="http://www.bestseosoftware.co">SEO Powersuite Review &#8211; Link Assistant Purchase</a> </p>]]></description>
			<content:encoded><![CDATA[<p><a href="http://www.rivmedia.co.uk">RivMedia - Digital Marketing Agency</a></p><h1>SEO Powersuite Review</h1>
<p><a href="http://www.triphp.com/prducts/linkass.php"><img class="alignleft" src="http://www.link-assistant.com/images/boxes/wa-box.png" alt="" width="126" height="134" /></a></p>
<p>Becoming a search engine leader is not as  difficult as you may think. In fact, you can take your site to the top of the  most popular search engines in no time at all. Of course, if you want this to  happen you need to take the proper steps. One of the best things you can do is  get involved with <a href="http://www.bestseosoftware.co/">SEO Powersuite</a>. This software can go a long way in  helping your site rank higher, and in turn make you more money. And that is  what you want, right?</p>
<p>Here are a few of the features that have  made Link-Assistant and its <a href="http://www.bestseosoftware.co/">SEO PowerSuite</a> software a major player in the  industry:</p>
<p>1. You get it all. When you rely on this  software you will never again have to look anywhere else for SEO assistance.  Simply put, you get everything you want plus much more. Not only does this save  you time, but it saves you tons of money in the long run as well.</p>
<p>2. Reliable. <a href="http://www.bestseosoftware.co/">SEO PowerSuite</a> was tested time  and time again to ensure that it could stand up to anything that is thrown its  way. Everybody from individual webmasters to Fortune 500 companies rely on this  software. Surely that is good enough for you.</p>
<p>3. Fast. Do you really want to wait month  after month for your site to reach the top of the search engines? Of course you  don’t. With <a href="http://www.bestseosoftware.co/">SEO PowerSuite</a> you have the guidance needed to grab the number one  spot on all the search engines including Google, Yahoo, and Bing.</p>
<p>If you are interested in getting started  with SEO PowerSuite there is a free version that offers many of its top  features. Those who want to take things to the next level can purchase a <a href="http://www.triphp.com/products/linkass.php">full  copy of SEO PowerSuite for $249</a>. This gives you more features, professional  support, and free updates for life.</p>
<p>Those who want more from their website  should head to <a href="http://www.triphp.com/products/linkass.php">here</a> and check out <a href="http://www.triphp.com/products/linkass.php">SEO PowerSuite and all its  benefits</a>. You will surely like what you see!</p>
<p>Take a look at these three reviews to make up your mind :</p>
<ul>
<li><a href="http://www.rivmedia.co.uk/ibp-10-review-ibusiness-promoter-purchase/299">IBP (ibusiness Promoter) Review</a></li>
<li><a href="http://www.bestseosoftware.co/">SEO Powersuite Review</a></li>
<li><a href="http://www.rivmedia.co.uk/web-ceo-review-and-purchase/306">WebCEO Review </a></li>
<li><a href="http://www.rivmedia.co.uk/comparing-internet-business-promoter-vs-seo-powersuite-vs-web-ceo-reviews/309">Review of all 3 SEO Software</a></li>
</ul>
<p><a href="http://www.bestseosoftware.co">SEO Powersuite Review &#8211; Link Assistant Purchase</a> </p>]]></content:encoded>
			<wfw:commentRss>http://www.bestseosoftware.co/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>IBP 10 Review &#8211; IBusiness Promoter Purchase</title>
		<link>http://www.rivmedia.co.uk/comparing-internet-business-promoter-vs-seo-powersuite-vs-web-ceo-reviews/309</link>
		<comments>http://www.rivmedia.co.uk/comparing-internet-business-promoter-vs-seo-powersuite-vs-web-ceo-reviews/309#comments</comments>
		<pubDate>Wed, 16 Dec 2009 13:48:17 +0000</pubDate>
		<dc:creator>admin</dc:creator>
				<category><![CDATA[Search Engine Optimisation]]></category>
		<category><![CDATA[Business]]></category>
		<category><![CDATA[Business Focus]]></category>
		<category><![CDATA[buy]]></category>
		<category><![CDATA[Free Demo]]></category>
		<category><![CDATA[Google]]></category>
		<category><![CDATA[Guarantee]]></category>
		<category><![CDATA[IBP]]></category>
		<category><![CDATA[Internet Business Promoter]]></category>
		<category><![CDATA[Internet Business Promoter Ibp]]></category>
		<category><![CDATA[Little Bit]]></category>
		<category><![CDATA[Many Other Services]]></category>
		<category><![CDATA[Money]]></category>
		<category><![CDATA[Other Search Engines]]></category>
		<category><![CDATA[Popular Search Engines]]></category>
		<category><![CDATA[promoter]]></category>
		<category><![CDATA[Promotion Software]]></category>
		<category><![CDATA[purchase]]></category>
		<category><![CDATA[review]]></category>
		<category><![CDATA[Search Engine Rankings]]></category>
		<category><![CDATA[Software Tool]]></category>
		<category><![CDATA[testimonial]]></category>
		<category><![CDATA[Top 10 Rankings]]></category>
		<category><![CDATA[Traffic]]></category>
		<category><![CDATA[Yahoo]]></category>

		<guid isPermaLink="false">http://www.rivmedia.co.uk/?p=299</guid>
		<description><![CDATA[<p><p><a href="http://www.rivmedia.co.uk">RivMedia - Digital Marketing Agency</a></p><p>IPB 10 Review Are you looking for a way to take your website to the top 10 rankings on Google and Yahoo? If so, you may need a little bit of help. Fortunately, this help is available: it is known as Internet Business Promoter. From the company’s homepage, they describe their service by saying, “Internet [...]</p></p><p><a href="http://www.rivmedia.co.uk/comparing-internet-business-promoter-vs-seo-powersuite-vs-web-ceo-reviews/309">IBP 10 Review &#8211; IBusiness Promoter Purchase</a> </p>]]></description>
			<content:encoded><![CDATA[<p><a href="http://www.rivmedia.co.uk">RivMedia - Digital Marketing Agency</a></p><h1>IPB 10 Review</h1>
<p><a href="http://www.triphp.com/products/ibp.php"><br />
<img src="http://www.axandra.com/affiliates/ibp/english/180x150-rectangle.gif" alt="Ibp review" align="left" border="0" hspace="10" /></a><br />
Are you looking for a way to take your website to the top 10 rankings on Google and Yahoo? If so, you may need a little bit of help. Fortunately, this help is available: it is known as Internet Business Promoter. From the company’s homepage, they describe their service by saying, “Internet Business Promoter (IBP) is a multi-award winning website promotion software tool. <a href="http://www.triphp.com/products/ibp.php">IBP</a> helps you get high search engine rankings.” Does it get any easier than that?</p>
<p>The first thing you need to know is that <a href="http://www.triphp.com/products/ibp.php">Internet Business Promoter</a> does not focus solely on Google, unlike some of the competition. When you get involved with this software you will have the chance to rank high in Google, as well as other popular search engines such as Yahoo and Bing. Although Google has claimed the majority of the market, this does not mean you should avoid the other search engines – they still offer a high level of traffic to websites that rank on the first page.</p>
<p>How many services like this offer a guarantee? With Internet Business Promoter you receive the following: we guarantee that your website will be found in the regular (unpaid) top 10 results on Google for the keywords of your choice. No catch, no strings attached. With a guarantee backing the offer there is no reason to pass over this opportunity for another day. If you do not receive the results you were expecting you get all your money back. There are not many other services, online or off, that have this type of guarantee.</p>
<p><a href="http://www.triphp.com/products/ibp.php">Internet Business Promoter</a> makes things simple. All you have to do is follow their directions in three areas: optimize, promote, and track. If you do what you are asked in these areas there is no stopping you.</p>
<p>Interested in getting started? If so, download the free demo. This gives you the chance to try out Internet Business Promoter without spending a dime. From there, if you like what you see you can <a href="http://www.triphp.com/products/ibp.php">buy the paid version for the low price of $249.95.</a></p>
<p>Take a look at these three reviews to make up your mind :</p>
<ul>
<li><a href="http://www.rivmedia.co.uk/ibp-10-review-ibusiness-promoter-purchase/299">IBP (ibusiness Promoter) Review</a></li>
<li><a href="http://www.rivmedia.co.uk/seo-powersuite-review-link-assistant-purchase/303">SEO Powersuite Review</a></li>
<li><a href="http://www.rivmedia.co.uk/web-ceo-review-and-purchase/306">WebCEO Review </a></li>
<li><a href="http://www.rivmedia.co.uk/comparing-internet-business-promoter-vs-seo-powersuite-vs-web-ceo-reviews/309">Review of all 3 SEO Software</a></li>
</ul>
<p><a href="http://www.rivmedia.co.uk/comparing-internet-business-promoter-vs-seo-powersuite-vs-web-ceo-reviews/309">IBP 10 Review &#8211; IBusiness Promoter Purchase</a> </p>]]></content:encoded>
			<wfw:commentRss>http://www.rivmedia.co.uk/comparing-internet-business-promoter-vs-seo-powersuite-vs-web-ceo-reviews/309/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
	</channel>
</rss>

