Create robots.txt file for better SEO
It has been a half month since ThemeLib.com was created, not a long time, huh ?. Today, I decided to test how is my site in two popular Search Engines : Google and Yahoo.
Here is the result :
- Google: get 48 results returned with keywords http://themelib.com
- Yahoo: get 35 results returned with keywords http://themelib.com
All of the results in Google is fine. They indexed my posts, my tags, my keywords, … But there are some problems with my site in Yahoo. Yahoo indexed my wp-login page and download links !!! How did it happen, huh ? If you have knowledge on SEO (Search Engine Optimization), these results are not good, really. After examining ThemeLib.com a few minutes, I realized that I have not created a Robots.txt file yet !!! Oh man, how can I forget it? 
Introduction
The robots.txt file is used to instruct search engine robots about what pages on your website should be crawled and consequently indexed. Most websites have files and folders that are not relevant for search engines (like images, download links, admin files, …) therefore creating a robots.txt file can actually improve your website indexation.
How to Create a Robots.txt file
A robots.txt is just a simple text file that can be created with any text editor such as Notepad. If you are using WordPress, a sample robots.txt file would be :
User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /download/
“User-agent: *” means that all the search engine (Google, Yahoo, MSN and so on) should use those instructions to crawl your website.
“Disallow: /wp-” will make sure that the search engines will not crawl the WordPress files. This line will exclude all files and foldes starting with “wp-” from the indexation, avoiding duplicated content and admin files. Similar to Disallow: /feed/, Disallow: /trackback/, …
After you created the robots.txt file just upload it to your root directory and DONE!
Further reading
Here are some good links that may useful for you if you want to know more about SEO and Robots.txt
- SEO Services
- Search Engine Optimisation
- Search engine optimisation in the UK
- Search engine optimization services program
Conclusion
You should always verify your site in Search Engine like Google, Yahoo to see if there are any problems with the indexing. Hope you do not make a mistake like me. If you need professional help, you can hire a SEO company
Twitter
Great post. Thanks for the info. It’s a tough job learning all this SEO stuff.
Seo is really very good and difficult (to start) to learn, to optimize the site, I understand, is need many studies related to that!
Now I ask you, the template I am using and you convert is optimized with Seo? If not, some assistance x)
thanks!
MOST of SEO techniques depend on you, not the theme
You can search on internet about SEO, lots of Blogger have written about it.
Thats The real Problem .. Just becoz everybody Writes we don’t know who to follow.. i just moved from blogger to wordpress.. lets See How it goes..
Whats the most important with SEO?The title? I really need work on SEO lol.
Hello John,
Title is important on SEO, but not the most important. In proper SEO, the key things are:
* Title
* Keyword Density (Percentage of the keyword in the total post)
* Image parameters like Image title, Alternate text etc
* Non indexing of duplicate contetn on tag pages etc.
* Proper links (nofollow for non relevant links)
* First 160 characters must describe the post. plus proper description tag etc.
Use the All in one SEo pack, its comprehensive + Google Sitemap Generator plugin
That should do. My friend, Mani is a professional SEO expert. His blog dailyseoblog.com has lots of tips.
Hope you find the info useful.
Arun.
There is another easy way for those who dont like to edit code directly, Use the Google-Sitemap Generator plugin. There is an option to create Robots.txt file. The file redirects the google bot to the sitemap and sitemap will have only the relevant links.
with this plugin, my contents are indexed within 36-48 hours.
Hope that helps someone.
Arun
This is a great source of info. I am hoping to improve my seo on my sites too. I will be hanging out in your site alot the coming days.
No body wants our wp-admin login page to get index. It will just make our blog look more visible for the attackers. I think sitemap file should be installed when a new blog installation period and it should come together with wordpress not as another plugin. I use google sitemap and so far so good.
I am also sufering from the same problem for my site http://www.icbse.com
When i search for the keywords for my site in google then the login and the other non-needed pages appears.
But after reading this article i will soon add the robots.txt in my site.
Thanks for sharing the best article.
I really don’t mess with robots.txt. I use an online robot.txt maker. It helps alot. Also there are many wordpress robot.txt plugins
Nice idea, then this will add up my knowledge about SEO optimization again. Thank bro, just keep it up and enjoy!
I never knew that robots.txt file is such a useful thing for SEO. But its a nice idea and I think it will definitely help in our SEO works.
I have blogger account. And its not possible to edit the robots.txt file in blogger account. Is it possible to create a seperate robots.txt file for a blog?