robots.txt Parser API

Q: How do I test if a URL is blocked by robots.txt?

Send `url` (the site), `path` (e.g. /private/page) and `user_agent` (e.g. Googlebot). The response contains `allowed: true` or `false`.

Q: Are wildcards supported in rules?

Yes. `*` matches any characters and a trailing `$` anchors the end of the path. The longest matching rule wins, and Allow wins a tie.

Q: What if the site has no robots.txt?

The response has `exists: false`, the HTTP status, empty groups, and any tested path is reported as allowed.

Fetch and parse the robots.txt of any site into user-agent groups, allow/disallow rules and sitemaps, and test if a path is crawlable.

GETPOSThttps://apibox.io.vn/v1/robots-txt

Web & SEO▶ Try it online

Give any URL of a site in url; the API fetches /robots.txt from its origin and parses it into groups (user agents with their allow, disallow and crawl_delay rules) and the list of sitemaps. The response also tells whether the file exists, its HTTP status and the raw text (first 20,000 characters).

Add a path (for example /search) and optionally a user_agent (default *) to test whether that path may be crawled: the API picks the group matching the user agent (or *), applies the longest matching rule with * wildcards and $ end anchors, and returns allowed: true or false.

The robots.txt file is read up to 512 KB with a 10 second timeout. Only public http/https hosts are allowed.

Parameters

Name	In	Type	Required	Default
`url`	body\|query	string	yes		Any URL of the site; robots.txt is read from its origin (max 4096 characters).
`path`	body\|query	string	no		Path to test, e.g. `/search`. When given, the response includes `allowed`.
`user_agent`	body\|query	string	no	`*`	User agent used for the `path` test, e.g. `Googlebot`.

Example

curl "https://apibox.io.vn/v1/robots-txt?url=https://github.com&path=/search&user_agent=Googlebot" \
  -H "Authorization: Bearer YOUR_API_KEY"

const res = await fetch('https://apibox.io.vn/v1/robots-txt?url=https://github.com&path=/search&user_agent=Googlebot', {
  method: 'GET',
  headers: {
    Authorization: 'Bearer YOUR_API_KEY',
  },
});
const data = await res.json();
console.log(data);

import requests

res = requests.get(
    "https://apibox.io.vn/v1/robots-txt?url=https://github.com&path=/search&user_agent=Googlebot",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    timeout=30,
)
print(res.json())

<?php
$ch = curl_init('https://apibox.io.vn/v1/robots-txt?url=https://github.com&path=/search&user_agent=Googlebot');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HTTPHEADER => ['Authorization: Bearer YOUR_API_KEY'],
]);
$response = curl_exec($ch);
print_r(json_decode($response, true));

Response

{
  "robots_url": "https://github.com/robots.txt",
  "status": 200,
  "exists": true,
  "groups": [
    {
      "user_agents": [
        "GPTBot",
        "OAI-SearchBot",
        "ChatGPT-User",
        "ClaudeBot"
      ],
      "allow": [
        "/*?tab=achievements&achievement=*",
        "..."
      ],
      "disallow": [
        "/account-login",
        "/copilot/",
        "/gist/",
        "/search$",
        "..."
      ],
      "crawl_delay": 1
    },
    {
      "user_agents": [
        "Bytespider"
      ],
      "allow": [],
      "disallow": [
        "/"
      ]
    },
    {
      "user_agents": [
        "*"
      ],
      "allow": [
        "/*?tab=achievements&achievement=*",
        "..."
      ],
      "disallow": [
        "/account-login",
        "/copilot/",
        "/gist/",
        "/search$",
        "..."
      ]
    }
  ],
  "sitemaps": [
    "https://github.com/sitemap_marketing.xml"
  ],
  "path": "/search",
  "user_agent": "Googlebot",
  "allowed": false,
  "raw": "# If you would like to crawl GitHub contact us via https://support.github.com?tags=dotcom-robots\n# We also provide an extensive API: https://docs.github.com\nSitemap: https://github.com/sitemap_marketing.xml\n\nUser-agent: GPTBot\n..."
}

Use cases

Check if Googlebot or another crawler may access a page
Find the sitemaps declared by a website
Make your own crawler or scraper respect robots.txt rules
Audit AI bot rules (GPTBot, ClaudeBot...) of a site

FAQ

How do I test if a URL is blocked by robots.txt?

Send url (the site), path (e.g. /private/page) and user_agent (e.g. Googlebot). The response contains allowed: true or false.

Are wildcards supported in rules?

Yes. * matches any characters and a trailing $ anchors the end of the path. The longest matching rule wins, and Allow wins a tie.

What if the site has no robots.txt?

The response has exists: false, the HTTP status, empty groups, and any tested path is reported as allowed.

robots.txt Parser API

Parameters

Example

Response

Use cases

FAQ

Related APIs

URL Metadata

HTTP Headers

Redirect Checker