robots.txt Tester & Parser
Fetch and parse the robots.txt of any site into user-agent groups, allow/disallow rules and sitemaps, and test if a path is crawlable.
Result
Press “Run” to see the result.
Give any URL of a site in url; the API fetches /robots.txt from its origin and parses it into groups (user agents with their allow, disallow and crawl_delay rules) and the list of sitemaps. The response also tells whether the file exists, its HTTP status and the raw text (first 20,000 characters).
Add a path (for example /search) and optionally a user_agent (default *) to test whether that path may be crawled: the API picks the group matching the user agent (or *), applies the longest matching rule with * wildcards and $ end anchors, and returns allowed: true or false.
The robots.txt file is read up to 512 KB with a 10 second timeout. Only public http/https hosts are allowed.
Use this in your code
curl "https://apibox.io.vn/v1/robots-txt?url=https://github.com&path=/search&user_agent=Googlebot" \
-H "Authorization: Bearer YOUR_API_KEY"const res = await fetch('https://apibox.io.vn/v1/robots-txt?url=https://github.com&path=/search&user_agent=Googlebot', {
method: 'GET',
headers: {
Authorization: 'Bearer YOUR_API_KEY',
},
});
const data = await res.json();
console.log(data);import requests
res = requests.get(
"https://apibox.io.vn/v1/robots-txt?url=https://github.com&path=/search&user_agent=Googlebot",
headers={"Authorization": "Bearer YOUR_API_KEY"},
timeout=30,
)
print(res.json())<?php
$ch = curl_init('https://apibox.io.vn/v1/robots-txt?url=https://github.com&path=/search&user_agent=Googlebot');
curl_setopt_array($ch, [
CURLOPT_RETURNTRANSFER => true,
CURLOPT_HTTPHEADER => ['Authorization: Bearer YOUR_API_KEY'],
]);
$response = curl_exec($ch);
print_r(json_decode($response, true));API documentation → Get a free API key
FAQ
How do I test if a URL is blocked by robots.txt?
Send url (the site), path (e.g. /private/page) and user_agent (e.g. Googlebot). The response contains allowed: true or false.
Are wildcards supported in rules?
Yes. * matches any characters and a trailing $ anchors the end of the path. The longest matching rule wins, and Allow wins a tie.
What if the site has no robots.txt?
The response has exists: false, the HTTP status, empty groups, and any tested path is reported as allowed.