When it comes to SEO, most people understand that a Web site must have content, \u201csearch engine friendly\u201d site architecture\/HTML, and meta data such as title tags, graphic alt tag tags and so on.<\/p>\n
However, some web sites totally disregarded the robots.txt file. When optimizing a Web site: don\u2019t disregard the power of this little text file.<\/p>\n
What is a Robots.txt File?<\/strong><\/p>\n Simply put, if you go to www.domain.com\/robots.txt, you should see a list of directories of the Web site that the site owner is asking the search engines to \u201cskip\u201d (or \u201cdisallow\u201d). However, if you\u2019re not careful when editing a robots.txt file, you could be putting information in your robots.txt file that could really hurt your business.<\/p>\n There\u2019s tons of information about the robots.txt file available at the Web Robots Pages, including the proper usage of the disallow feature, and blocking \u201cbad bots\u201d from indexing your Web site.<\/p>\n The general rule of thumb is to make sure a robots.txt file exists at the root of your domain (e.g., www.domain.com\/robots.txt). To exclude all robots from indexing part of your Web site, your robots.txt file would look something like this:<\/p>\n User-agent: The above syntax would tell all robots not to index the \/cgi-bin\/, the \/tmp\/, and the \/junk\/ directories on your Web site.<\/p>\n There are situations where you might use the Robots.txt file to cause issues with your site optimisation.\u00a0 For instance if you include a * Disallow: \u201c\/\u201d in your Robots.txt file it will be telling the search engines not to crawl any part of the web site giving you no web presence \u2013 not what you want.<\/p>\n Another point to watch out for is if you modify your Robots.txt file to dissallow old legacy pages and directories \u2013 you should really do a 301 permanent redirect to pass the value from the old Web pages to the new web pages.<\/p>\n Robots.txt Dos and Don\u2019ts<\/strong><\/p>\n There are many good reasons to stop the search engines from indexing certain directories on a Web site and allowing others for SEO purposes.<\/p>\n Here\u2019s what you should do with robots.txt:<\/p>\n * Take a look at all of the directories in your Web site. Most likely, there are directories that you\u2019d want to disallow the search engines from indexing, including directories like \/cgi-bin\/,\u00a0 \/wp-amin\/,\u00a0 \/cart\/,\u00a0 \/scripts\/,\u00a0 and others that might include sensitive data. Here\u2019s what you should not do with robots.txt:<\/strong><\/p>\n * Don\u2019t use comments in your robots.txt file. By taking a good look at your Web site\u2019s robots.txt file and making sure that the syntax is set up correctly, you\u2019ll avoid search engine ranking problems.\u00a0 By disallowing the search engines to index duplicate content on your Web site, you can potentially overcome duplicate content issues that might hurt your search engine rankings.<\/p>\n Test a robots.txt file<\/strong><\/p>\n Google provides a facility as part of there Webmaster Tools system to enable you to test a robots.txt file.<\/p>\n Test a site’s robots.txt file:<\/p>\n On the Webmaster Tools Home page, click the site you want. Any changes you make in this tool will not be saved. To save any changes, you’ll need to copy the contents and paste them into your robots.txt file.<\/p>\n","protected":false},"excerpt":{"rendered":" When it comes to SEO, most people understand that a Web site must have content, \u201csearch engine friendly\u201d site architecture\/HTML, and meta data such as title tags, graphic alt tag tags and so on. However, some web sites totally disregarded…<\/span><\/p>\n
\n* Disallow: \/cgi-bin\/
\nDisallow: \/tmp\/
\nDisallow: \/junk\/<\/p>\n
\n* Stop the search engines from indexing certain directories of your site that might include duplicate content. For example, some Web sites have \u201cprint versions\u201d of Web pages and articles that allow visitors to print them easily. You should only allow the search engines to index one version of your content.
\n* Make sure that nothing stops the search engines from indexing the main content of your Web site.
\n* Look for certain files on your site that you might want to disallow the search engines from indexing, such as certain scripts, or files that might contain e-mail addresses, phone numbers, or other sensitive data.<\/p>\n
\n* Don\u2019t list all your files in the robots.txt file. Listing the files allows people to find files that you don\u2019t want them to find.
\n* There\u2019s no \u201c\/allow\u201d command in the robots.txt file, so there\u2019s no need to add it to the robots.txt file.<\/p>\n
\nUnder Health, click Blocked URLs.
\nIf it’s not already selected, click the Test robots.txt tab.
\nCopy the content of your robots.txt file, and paste it into the first box.
\nIn the URLs box, list the site to test against.
\nIn the User-agents list, select the user-agents you want.<\/p>\n