.Google.com's John Mueller responded to a concern about why Google indexes webpages that are refused from creeping by robots.txt and also why the it's risk-free to dismiss the relevant Explore Console documents concerning those creeps.Bot Web Traffic To Concern Specification URLs.The individual talking to the question recorded that robots were actually creating web links to non-existent concern specification Links (? q= xyz) to pages along with noindex meta tags that are actually additionally blocked in robots.txt. What cued the concern is actually that Google.com is actually crawling the links to those webpages, receiving blocked by robots.txt (without seeing a noindex robotics meta tag) then obtaining shown up in Google Browse Console as "Indexed, though blocked by robots.txt.".The person talked to the observing concern:." But below is actually the major question: why would certainly Google.com mark webpages when they can't even observe the web content? What is actually the benefit in that?".Google.com's John Mueller confirmed that if they can't crawl the page they can not observe the noindex meta tag. He likewise helps make an appealing reference of the web site: search driver, encouraging to neglect the results considering that the "average" consumers won't find those results.He wrote:." Yes, you're proper: if we can't creep the page, we can't find the noindex. That mentioned, if our team can't crawl the web pages, then there's certainly not a lot for our company to index. Therefore while you may view a number of those pages with a targeted site:- question, the ordinary user won't view all of them, so I definitely would not bother it. Noindex is likewise fine (without robots.txt disallow), it just means the URLs will find yourself being actually crept (and end up in the Look Console report for crawled/not listed-- neither of these statuses lead to concerns to the remainder of the website). The vital part is that you do not produce all of them crawlable + indexable.".Takeaways:.1. Mueller's solution verifies the restrictions being used the Web site: hunt advanced hunt operator for analysis factors. One of those main reasons is given that it's certainly not linked to the normal hunt mark, it's a separate point completely.Google's John Mueller commented on the web site search operator in 2021:." The brief answer is that a website: concern is actually not suggested to be comprehensive, neither made use of for diagnostics reasons.A site query is a details kind of search that confines the outcomes to a specific internet site. It is actually generally merely the word internet site, a digestive tract, and after that the site's domain.This inquiry limits the results to a details internet site. It's not meant to become an extensive selection of all the web pages coming from that web site.".2. Noindex tag without making use of a robots.txt is actually alright for these kinds of situations where a crawler is linking to non-existent web pages that are actually getting discovered by Googlebot.3. URLs with the noindex tag will certainly generate a "crawled/not catalogued" entry in Look Console and that those will not possess a negative effect on the rest of the web site.Read the inquiry and also respond to on LinkedIn:.Why would Google.com mark web pages when they can't also see the content?Featured Photo through Shutterstock/Krakenimages. com.