Google Webmaster Tools - Crawler Access

See below for screenshot and explanation of Crawler Access.

Here are the Google webmaster tools categories as they look when you are logged in (left).
Below is the same structure with clickable links which go to a Hard Web webpage with a screenshot of what is on that webmaster tools category page (in my own webmaster tools account).

• Site Configuration
  • Sitemaps
  • Crawler access - YOU ARE HERE!
  • Sitelinks
  • Change of address
  • Settings
  • URL parameters
• Your site on the web
  • Search Queries
  • Links to your site
  • Keywords
  • Internal links
  • Subscriber stats
• +1 Metrics
  • Search impact
  • Activity
  • Audience
• Diagnostics
  • Malware
  • Crawl errors
  • Crawl stats
  • Fetch as Googlebot
  • HTML suggestions
• Labs
  • Custom Search
  • Instant Previews
  • Site performance
  • Video Sitemaps

Webmaster tools - Site Configuration - Crawler Access

The Crawler Access tool tells you whether you have a robots.txt file submitted or not.
This category should probably have been called Robots.txt files in my opinion.
The page also has a tool to generate a robots.txt file and to ask for removal of a specific URL from the Google index.
I used these tools in tandem recently (Robots.txt and URL removal) and began to feel there was a conflict whereby if I had a 301 permanent redirect on a URL (or folder) AND I had asked for it to be removed AND I had excluded the page (or "directory" aka "folder") via the robots.txt file...
Was I sending the robots on a mission only for them to hit a redirect? I thought that might hurt their feelings (and my rankings).

Help for the Site Configuration master category within Webmaster Tools (new window).
See the Webmaster Tools help page on "How to Block or remove pages using a robots.txt file" (new window).

Here is a screenshot of the category " Crawler Access" from within Webmaster tools (below).

Screenshots on this page are copyright © 2011 Google Inc.
If this page is not helping then you must go to the fount...Google Webmaster Tools Help (new window).
Also click to signup to get your own Webmaster tools account (new window).

Contact Hardweb - making websites work!

Web Design SEO SEM

07 5526 2939
0401 352 269

Home/Office Address
25 Savoy Drive
Broadbeach Waters
Gold Coast Queensland 4218