How do I test whether my robots.txt blocks a URL?

Enter your domain in the robots.txt Tester. It fetches your robots.txt, parses the allow and disallow rules, and gives a clear allowed or blocked verdict for any URL you test.

How does the tester decide if a URL is blocked?

It parses the robots.txt into user-agent groups, picks the most specific matching group for your chosen user-agent, and applies the longest matching allow or disallow rule, exactly how crawlers do it. Allow wins ties.

What does an asterisk in a rule mean?

The asterisk is a wildcard matching any sequence of characters, and a dollar sign at the end of a rule anchors it to the end of the URL. Both are part of the Google robots.txt specification.

Which user-agent should I test with?

Test with the crawler you care about. The tool defaults to the wildcard group that applies to all crawlers, but you can pick Googlebot, Bingbot or any custom user-agent string.

What if the site has no robots.txt?

A missing robots.txt (404) means everything is crawlable by default. The tool reports this explicitly instead of failing.

Yes. Fetches use SSRF protection, a 10 second timeout and a capped body size. Only the robots.txt content is read and nothing is stored.

SEO TOOLS · FREE ONLINE TOOL

robots.txt Tester

Test any URL against a site robots.txt rulesDo not trust your robots.txt. Test it. Rules interact in subtle ways: the most specific match wins, Allow can override Disallow, and a single typo can invert a whole rule block. See exactly what a crawler will do before your visitors do.

✓ Free to use✓ No sign-up✓ Ready in your browser

// ROBOTS.TXT TESTER · GET STARTED

Use robots.txt Tester

Test any URL against a site robots.txt rules

// ROBOTS.TXT TESTER · HOW IT WORKS

How to use the robots.txt Tester

Enter site and URL

Enter the site URL and the page URL you want to test.

Pick user-agent

Pick a user-agent (or leave the default).

See the verdict

Click "Test" to see whether the URL is allowed or blocked, and why.

// ROBOTS.TXT TESTER · ABOUT THIS TOOL

About the robots.txt Tester

A single wrong disallow line can hide your whole site from Google, and syntax errors in robots.txt are invisible until you test them. This tool fetches the live robots.txt, parses every user-agent group with its allow, disallow and sitemap lines, and evaluates your URL with the same matching logic crawlers use. Use it before site launches, after edits, and during SEO audits.

Where it helps most

Moments when seeing a rule's real behavior beats reading the file and guessing.

Verify before deploying

Developers verify every robots.txt change here before deploying, instead of discovering a painful site-wide block in Search Console days later.

Debug blocked warnings

SEOs debug blocked-by-robots.txt warnings by finding the exact offending rule named in the verdict, instead of guessing through the whole file.

Protect admin areas

Confirm admin areas and staging paths stay hidden, while the JavaScript and CSS resources Google needs to render pages stay reachable.

Audit client sites

Auditors sweep client sites for CSS, JS or pages accidentally hidden from Google during redesigns, migrations and bulk content updates.

Testing it well

How to get verdicts you can actually trust.

Test the paths that matter

Test the paths that matter most: admin areas that must stay hidden, staging paths, and the JavaScript and CSS Google needs to render pages.

Switch user agents

Test with the exact user-agent strings that matter to you, since a rule that behaves for one bot can surprise you with another.

Draft before going live

Paste your draft rules directly into the tester with the paths you care about, and only upload the file after every verdict looks right.

robots.txt Tester: built for trust

Test draft rules before anything goes live. The tester evaluates them exactly as a crawler would and names the rule behind each verdict.

// ROBOTS.TXT TESTER · COMMON QUESTIONS

robots.txt Tester FAQs

How is the tester different from the generator?

The generator builds a new robots.txt file from a form; the tester checks what an existing set of rules actually does to specific URLs. Build with the generator, verify with the tester, and you cover both directions.

Which rule wins when Allow and Disallow overlap?

Why would Google need access to my CSS and JS files?

Can I test rules before uploading the file?

tools-guide

DNS Lookup: Free Online DNS Record Checker for Any Domain

Free online DNS lookup tool. Check A, AAAA, MX, TXT, CNAME, NS and SOA records for any domain instantly. Fast, private, accurate, no sign-up required.

Read article →