.Google.com's John Mueller responded to an inquiry regarding why Google.com indexes webpages that are actually prohibited coming from crawling by robots.txt and why the it's risk-free to overlook the similar Browse Console records regarding those crawls.Robot Web Traffic To Query Guideline URLs.The individual talking to the concern recorded that crawlers were making web links to non-existent inquiry criterion URLs (? q= xyz) to pages along with noindex meta tags that are also shut out in robots.txt. What triggered the concern is actually that Google is crawling the web links to those web pages, receiving shut out through robots.txt (without seeing a noindex robots meta tag) then receiving shown up in Google Browse Console as "Indexed, though blocked out by robots.txt.".The person talked to the complying with inquiry:." But listed here's the large inquiry: why would Google.com mark webpages when they can not also find the content? What's the benefit during that?".Google.com's John Mueller validated that if they can not creep the web page they can't see the noindex meta tag. He also creates an intriguing mention of the web site: hunt driver, recommending to ignore the end results given that the "common" users will not find those outcomes.He wrote:." Yes, you're appropriate: if we can't creep the web page, our team can not observe the noindex. That mentioned, if our team can't crawl the pages, at that point there is actually not a lot for our company to index. Thus while you could find a number of those pages along with a targeted web site:- concern, the typical consumer will not observe all of them, so I wouldn't fuss over it. Noindex is additionally great (without robots.txt disallow), it only indicates the Links will end up being actually crawled (and also find yourself in the Browse Console document for crawled/not listed-- neither of these conditions trigger concerns to the rest of the site). The vital part is that you do not create them crawlable + indexable.".Takeaways:.1. Mueller's response confirms the restrictions being used the Internet site: search progressed hunt driver for diagnostic explanations. Among those reasons is actually given that it's not attached to the normal search mark, it's a distinct thing altogether.Google's John Mueller discussed the site hunt driver in 2021:." The brief solution is actually that a website: question is actually certainly not implied to be total, nor utilized for diagnostics functions.A site query is actually a details sort of search that confines the outcomes to a certain website. It's basically just the word web site, a colon, and afterwards the web site's domain name.This concern confines the end results to a particular web site. It's certainly not meant to be a complete compilation of all the webpages coming from that internet site.".2. Noindex tag without utilizing a robots.txt is great for these kinds of situations where a robot is actually connecting to non-existent webpages that are actually obtaining found by Googlebot.3. URLs with the noindex tag will certainly create a "crawled/not listed" item in Search Console which those won't have an unfavorable result on the rest of the web site.Read through the question as well as respond to on LinkedIn:.Why will Google index web pages when they can't even observe the material?Included Photo by Shutterstock/Krakenimages. com.