Crawling
robots.txt and Twitterbot
How Twitterbot follows robots.txt, the rule that unblocks it on a locked-down site, and the firewall and bot-protection settings that break cards without warning.
Short answer
Twitterbot is the user agent X uses to fetch shared links; it follows robots.txt the way Google does, so a Disallow that covers your pages or images stops cards, and a specific User-agent: Twitterbot group with Allow: / lets it through.
Key takeaways
- Twitterbot obeys robots.txt using Google's rules: the most specific matching user-agent group wins.
- Images are fetched too, so image folders must not be disallowed.
- Firewalls, bot challenges and 'block AI bots' switches can block Twitterbot even when robots.txt allows it.
- Do not block link preview bots if you want previews; they are not search or AI training crawlers.
A robots.txt file is a small text file with a big blast radius. One line meant for search engines or scrapers can quietly remove every card your site gets on X.
Does Twitterbot respect robots.txt?
Yes. X's troubleshooting guide says Twitterbot implements Google's robots.txt specification, and lists a blocking robots.txt as a reason cards fail. That means the usual rules apply: the crawler finds the group whose User-agent line matches it most specifically, and follows only that group. If there is no Twitterbot group, it follows User-agent: *.
Which rules block cards?
Any Disallow that covers the shared page or its image, in the group Twitterbot follows. Some examples:
# Blocks everything for every bot, including Twitterbot
User-agent: *
Disallow: /
# Pages are allowed, but card images in /media/ are not: cards appear without images
User-agent: *
Disallow: /media/
# Staging rule left on production after launch
User-agent: *
Disallow: /blog/
The second case is easy to miss. The page itself can be fetched, so a card may be built, but the image request is refused.
How do I unblock Twitterbot on a locked-down site?
Add a group just for Twitterbot. Because the most specific group wins, this works even if everyone else is blocked:
User-agent: *
Disallow: /
User-agent: Twitterbot
Allow: /
X's guide shows the equivalent older form, an empty Disallow: under User-agent: Twitterbot, which also means "nothing is disallowed". If you only want cards for public marketing pages, allow those paths plus the folder your card images live in:
User-agent: Twitterbot
Allow: /blog/
Allow: /og/
Disallow: /
Under Google's rules the longest matching path wins, so /blog/ and /og/ are allowed and everything else is not.
The same pattern works for other link preview bots, such as facebookexternalhit for Facebook and LinkedInBot for LinkedIn. Slack is the exception: its documentation says it does not currently honor robots.txt at all.
A few robots.txt details that bite
- One file per host.
www.example.com/robots.txtandexample.com/robots.txtare separate files. If your images live on a CDN host such asimg.example.com, that host's robots.txt applies to the image fetch. - Case matters in paths.
Disallow: /Media/does not block/media/, and the reverse is also true. - A missing file allows everything. Google's specification treats a 404 for robots.txt as "no restrictions". A 5xx error is treated more cautiously, as a temporary full block, so a broken robots.txt endpoint can stop crawling.
- Wildcards are supported. Under Google's rules
*matches any sequence and$anchors the end, soDisallow: /*.pdf$blocks PDFs only.
What else can block Twitterbot?
Anything that treats it as an unwanted bot: firewalls, challenges and gates.
| Blocker | Symptom | Fix |
|---|---|---|
| WAF or CDN bot rules | 403 for Twitterbot in logs | Allow the verified bot or the user agent |
| JavaScript or CAPTCHA challenges | Card built from the challenge page, or no card | Exempt link preview bots from challenges |
| One-click "block AI crawlers" settings | Cards vanish after enabling | Check which user agents the setting covers |
| Geo-blocking | Cards fail in some regions | Allow the bot regardless of country |
| Cookie or age walls | Card shows the wall's title | Serve real tags to everyone; overlay the wall with script |
| Rate limiting | Cards fail during traffic spikes | Raise limits for known bots |
Your server logs are the quickest way to see what is going on. Search for Twitterbot around the time you posted a link and look at the status code. A 200 with a small response size can mean a challenge page was served instead of the real one.
Testing your rules
- Read your live file at
https://yourdomain/robots.txt. Check for aTwitterbotgroup and forDisallowlines underUser-agent: *. - Make a request as the bot:
curl -I -A "Twitterbot/1.0" https://yourdomain/page/and the same for the image URL. You want 200 for both. - Paste the page head into the Card Preview & Meta Inspector to confirm the tags themselves are right.
- Post the URL with a fresh query string in the X composer.
A curl request with a Twitterbot user agent is not the real crawler, and some firewalls verify bots by IP range, so a pass here is not proof. A failure, though, almost always points straight at the problem.
Should I welcome other bots too?
That is your call, and it is separate from cards. Link preview bots fetch a page because a person shared it. Search engines index it, and AI crawlers may read it for search answers or training. Many sites, this one included, allow all of them. If you block some, name them individually rather than using a blanket rule, and keep Twitterbot, facebookexternalhit, LinkedInBot and Discordbot allowed if you want previews to keep working.
Questions
What is Twitterbot's user agent string?
X's troubleshooting guide names it Twitterbot, with the versioned form Twitterbot/1.0. Match on the word Twitterbot rather than a full string.
If I block GPTBot or other AI crawlers, does that affect X Cards?
Not if you block them by name in robots.txt. Twitterbot is a separate user agent. Be careful with blanket rules and firewall presets that block many bots at once.
Can I allow Twitterbot only on some pages?
Yes. A Twitterbot group can Allow and Disallow paths like any other group. Remember to allow the image paths your cards use.
Does a noindex tag stop cards?
No. noindex tells search engines not to list the page; it does not stop Twitterbot reading the card tags. robots.txt Disallow is what blocks fetching.