Robots and Indexability Checker

Yes. Robots.txt stops crawling, not indexing. Google says a disallowed URL can still be indexed if other sites link to it. It usually appears with no snippet because Google could not read the content. If you want a page out of results, allow crawling and add a noindex tag instead. The checker flags this exact situation as a Disallow plus noindex conflict.

Noindex keeps a page out of search results, while robots.txt keeps crawlers from fetching it. They do different jobs. Use noindex for thank you pages, internal search results or thin tag pages. Use robots.txt to save crawl requests on things like endless filter URLs. Using both on the same URL breaks the noindex, because Google can never read it.

It is an HTTP header that carries the same rules as the meta robots tag. Servers send it before the page loads, so you will not find it in the HTML source. It is the usual way to noindex PDFs and images. This checker reads response headers for every URL, so a hidden X-Robots-Tag: noindex shows up right next to the meta tag result.

Usually, but not always. Google treats rel canonical as a strong signal, not a command. If your page points its canonical at another URL, Google will most likely index that other URL and drop yours. Sometimes it ignores the tag when the pages are clearly different. If you want the page indexed, make its canonical point to itself.

No. It checks indexability, meaning whether anything technical blocks indexing. It does not query Google, so it cannot confirm what is in the index. For that, use the URL Inspection tool in Search Console, which shows Google's own view. Run this checker first so you know the blockers are gone before you ask Google to look.

Up to 100 URLs per run. You can paste them one per line or enter a sitemap URL, and the tool reads the URLs from it. For sitemaps larger than 100 URLs, the checker takes the first 100. Split a big site into smaller sitemaps, such as one for posts and one for pages, and check each separately.

It means Google fetched the page and decided not to index it for now. It is a quality or priority choice, not a technical block. This checker will often show such pages as fine. Improve the content, cut duplicates and add internal links from strong pages. Then request indexing again and give it a few weeks.

It follows the matching rules Google documents. The most specific rule wins, Allow beats Disallow on a tie, and * and $ wildcards work. It tests against the Googlebot user agent group, or the * group if none exists. It cannot account for server rules that block bots by IP, so a page can pass here and still fail in the real world.