Genesis Themes

  • Home
  • Genesis Tutorials
  • WP Tutorial
  • Twenty Fifteen Theme
  • Social Media
  • WP Plugins

How to Add Custom Robots.txt to Blogger

Add custom robots.txt file to blogger for better crawling and indexing by Google search engine.You can instruct search engine to disallow and allow to crawl and index particular pages in blogger.Make sure that,this file is very crucial for every blog,incorrect use of this robot.s txt file may ignore by Google search engine.There are several blogger robots.txt generator tools available,but always edit  this file as per your requirements.

How to Add robots.txt file to blogger:

1. Login to your blogger blog.

2.Go to  Settings > Search Preferences > Crawlers and indexing > Custom robots.txt > Edit > Yes

3.Now paste the following custom robots.txt file in your blogger

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Allow: /

Sitemap: http://yoursitename.com/feeds/posts/default?orderby=UPDATED

custom robots txt

4.Finally click on save changes,that's it.(change yoursitename.com to your blog address)

5.Now check out your blogger robots.txt file by entering www.yourblogname.com/robots.txt

custom robots txt

Understanding the custom robots.txt file :

understand the each and every terms/code used in blogger robots.txt file.

Media partners- Google :

Media partners Google is the user agent for Google adsense,which helps to display relevance or related adds based on content of your blog.

Disallow:

Instructing to Google robots not to index and crawl particular pages,posts,affiliate links and other required posts.Normally in Blogger, many use this feature to no index labels (as WordPress tags and categories) in your blog.

User-agent: *

This is for all robots marked with asterisk (*). In default settings our blog’s labels links are restricted to indexed by search crawlers that means the web crawlers will not index our labels page links because of below code.

Disallow: /search

This refers not to index labels of your blog.

And if you remove Disallow: /search from the above code then Google search engine will crawl and index all pages,labels and posts in your blog

Allow: /

This refers to allow Google search engine to crawl and index blog home page

Sitemap :

Finally sitemap refers to crawl and index blog as per the custom robots.txt file each time.

Also see: How to add Sitemap to Bloggers blog.

 

Related Posts

  • How to Submit Blogger Sitemap to Google WebMaster Tools
  • How to add Google Adsense Below Post Title in Blogger
  • How to Add Google Adsense Below Navigation Menu in Blogger

Bluehost and Siteground Coupons for January 2016.

Best Premium WordPress Themes from TemplateMonster
download wordpress themes from MonsterOne

Categories

  • Blog
  • Blogger
  • Genesis Tutorials
  • Social Media
  • Top Websites
  • Twenty Fifteen WordPress Theme
  • WordPress Plugins
  • WordPress Tutorial