AI visibility

Are you blocking the crawlers meant to quote you?

Before a model can cite your site, it has to be allowed to read it. Access is decided in one file most teams last touched when they set up the project — and never checked again.

Check it yourself See Trust

Who is asking

The crawlers this is about.

Each is a separate user agent with its own rule in robots.txt. Allowing one says nothing about the others.

User agent	Operated by	Used for
GPTBot	OpenAI	Training and answers in ChatGPT
ClaudeBot	Anthropic	Answers and browsing in Claude
PerplexityBot	Perplexity	Search results in Perplexity
Google-Extended	Google	Grounding for Google AI answers
CCBot	Common Crawl	Open dataset used by many models

The list changes. New agents appear, existing ones split into separate names — which is why a rule set that is never updated silently drifts out of date.

Five minutes, no account

What you can check yourself right now.

1

Can they in?

Open your robots.txt and look for the agents above. No entry means no decision was made.
2

Is there a summary?

Request /llms.txt. A 404 means a model has to infer your site from your navigation.
3

Does the page say what it is?

View source on your pricing page and find the title. If it matches your home page, the two are indistinguishable.
4

Is the content in the HTML?

Disable JavaScript and reload. Whatever disappears is content some crawlers will never see.

Why sites stay invisible

It is rarely one big mistake.

✗A staging rule that shipped to production — Disallow: / and nobody noticed.
✗A CMS that puts noindex on pages that should be found.
✗Content rendered only in the browser, invisible to crawlers that do not run JavaScript.
✗One title and one description shared across the whole site.
✗A sitemap listing pages that no longer exist, while the new ones are missing.

And then again tomorrow

A check is a snapshot. A deploy changes everything.

The tests above tell you where you stand today. The Web Ready report runs them with every scan, keeps the history and mails you when something flips — because the change that costs you visibility is almost never the one you made on purpose.

What Web Ready checks

Questions

Should I allow every AI crawler?

That is your call, and both answers are defensible. What is not defensible is not knowing which ones you currently admit.

Does blocking them protect my content?

It stops well-behaved crawlers, which is most of the named ones. It does not stop anyone who ignores robots.txt — that file is a request, not a lock.

Will I show up in ChatGPT if I allow everything?

Not automatically. Access is the precondition, not the result. Being readable and distinguishable is what decides whether you are usable as a source.

Does a noindex page still get read by AI crawlers?

noindex speaks to search indexing, not to agent access — the two are separate rules. That mismatch is one of the things we check.

Get it

Web Ready comes with Trust.

One tier, one price. There is no separate Web Ready subscription to work out.

Trust

€24 every 4 weeks

Public security badge, auto-updating
Daily re-scan of your site
Web Ready report — SEO and AEO, private to you
AI-fix prompts — paste into Claude or GPT
Trigger a scan on demand (rate-limited)

Subscribe to Trust — €24 Compare all tiers

✓ EU-hosted · GDPR ✓ Cancel anytime ✓ No credit-card trial trap

Price excl. VAT — added at checkout where applicable. B2B only.

The crawlers this is about.

What you can check yourself right now.

Can they in?

Is there a summary?

Does the page say what it is?

Is the content in the HTML?

It is rarely one big mistake.

A check is a snapshot. A deploy changes everything.

Questions

Web Ready comes with Trust.