Add custom robots.txt file to blogger for better crawling and indexing by Google search engine.You can instruct search engine to disallow and allow to crawl and index particular pages in blogger.Make sure that,this file is very crucial for every blog,incorrect use of this robot.s txt file may ignore by Google search engine.There are several blogger robots.txt generator tools available,but always edit this file as per your requirements.
How to Add robots.txt file to blogger:
1. Login to your blogger blog.
2.Go to Settings > Search Preferences > Crawlers and indexing > Custom robots.txt > Edit > Yes
3.Now paste the following custom robots.txt file in your blogger
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://yoursitename.com/feeds/posts/default?orderby=UPDATED
4.Finally click on save changes,that's it.(change yoursitename.com to your blog address)
5.Now check out your blogger robots.txt file by entering www.yourblogname.com/robots.txt
Understanding the custom robots.txt file :
understand the each and every terms/code used in blogger robots.txt file.
Media partners- Google :
Media partners Google is the user agent for Google adsense,which helps to display relevance or related adds based on content of your blog.
Disallow:
Instructing to Google robots not to index and crawl particular pages,posts,affiliate links and other required posts.Normally in Blogger, many use this feature to no index labels (as WordPress tags and categories) in your blog.
User-agent: *
This is for all robots marked with asterisk (*). In default settings our blog’s labels links are restricted to indexed by search crawlers that means the web crawlers will not index our labels page links because of below code.
Disallow: /search
This refers not to index labels of your blog.
And if you remove Disallow: /search from the above code then Google search engine will crawl and index all pages,labels and posts in your blog
Allow: /
This refers to allow Google search engine to crawl and index blog home page
Sitemap :
Finally sitemap refers to crawl and index blog as per the custom robots.txt file each time.
Also see: How to add Sitemap to Bloggers blog.