.Google's John Mueller answered an inquiry about why Google marks pages that are actually forbidden coming from creeping by robots.txt and also why the it's safe to neglect the associated Search Console files concerning those creeps.Bot Visitor Traffic To Concern Guideline URLs.The individual asking the concern recorded that bots were actually making hyperlinks to non-existent inquiry specification URLs (? q= xyz) to pages with noindex meta tags that are also shut out in robots.txt. What prompted the concern is that Google.com is actually crawling the links to those web pages, acquiring blocked out by robots.txt (without noticing a noindex robots meta tag) at that point obtaining reported in Google.com Search Console as "Indexed, though blocked by robots.txt.".The person talked to the complying with inquiry:." Yet right here's the large concern: why would Google.com mark pages when they can not even observe the content? What is actually the conveniences in that?".Google.com's John Mueller validated that if they can not creep the page they can't observe the noindex meta tag. He also helps make an appealing mention of the web site: hunt driver, urging to dismiss the outcomes since the "normal" users will not observe those outcomes.He wrote:." Yes, you are actually correct: if our team can not crawl the web page, our company can not see the noindex. That mentioned, if our experts can not crawl the pages, then there's not a great deal for our team to mark. Therefore while you may find a few of those webpages with a targeted web site:- inquiry, the normal individual won't observe all of them, so I definitely would not bother it. Noindex is actually additionally great (without robots.txt disallow), it merely suggests the Links will wind up being actually crept (and also wind up in the Search Console document for crawled/not indexed-- neither of these conditions result in concerns to the remainder of the internet site). The important part is that you do not produce them crawlable + indexable.".Takeaways:.1. Mueller's response confirms the limits being used the Site: hunt evolved hunt operator for diagnostic reasons. Among those main reasons is given that it's certainly not attached to the routine hunt index, it is actually a different thing entirely.Google's John Mueller discussed the internet site search operator in 2021:." The brief solution is actually that a website: inquiry is not indicated to be comprehensive, neither utilized for diagnostics functions.A website concern is actually a details kind of hunt that limits the end results to a certain website. It is actually generally merely words website, a bowel, and after that the web site's domain name.This question confines the results to a specific internet site. It's certainly not implied to become a complete collection of all the pages from that web site.".2. Noindex tag without making use of a robots.txt is actually fine for these type of situations where a bot is actually connecting to non-existent pages that are obtaining found by Googlebot.3. URLs with the noindex tag will definitely produce a "crawled/not recorded" entry in Explore Console which those won't possess an unfavorable impact on the rest of the site.Go through the concern and also address on LinkedIn:.Why will Google mark web pages when they can not also find the content?Featured Photo through Shutterstock/Krakenimages. com.