Sitemap Generator API

Crawl a website from its home page and build sitemap.xml: same host only, robots.txt respected, noindex and canonical honoured, lastmod from headers.

GETPOSThttps://apibox.io.vn/v1/sitemap-generate

Web & SEO▶ Try it online

The sitemap generator crawls a site from the url you give, following links on the same host (after the first redirect, so http → https and www are handled), six pages at a time. It respects robots.txt (rules for APIBoxBot or *), skips pages marked noindex (meta robots or X-Robots-Tag), does not follow nofollow links, and when a page declares a canonical URL elsewhere, the canonical page is listed instead. Images, PDFs, scripts and other files are ignored.

Shape the crawl with max_pages, max_depth (0–10, default 5), path_prefix (only /blog/…), exclude (a regular expression), keep_query=true for sites whose pages differ by query string, and ignore_robots for your own site. Each URL gets its lastmod (from the Last-Modified header), an optional changefreq and a priority that decreases with link depth (priority=false drops it).

The JSON answer lists the pages with their titles and depth, what was skipped and why, the sitemaps declared in robots.txt and the XML itself; format=xml returns the sitemap.xml file directly. Pages per call depend on your plan (Free 50, Starter 200, Pro 500, online tool 20), and a crawl stops after about 25 seconds.

Parameters

Name	In	Type	Required	Default
`url`	query\|body	string	yes		Start URL (usually the home page).
`max_pages`	query\|body	number	no		Pages to include (default and maximum: your plan’s cap).
`max_depth`	query\|body	number	no	`5`	Link depth from the start page (0–10).
`path_prefix`	query\|body	string	no		Only URLs whose path starts with it, e.g. `/blog/`.
`exclude`	query\|body	string	no		Regular expression of URLs to leave out.
`keep_query`	query\|body	boolean	no	`false`	Keep query strings (default: removed).
`ignore_robots`	query\|body	boolean	no	`false`	Ignore robots.txt (for your own site).
`changefreq`	query\|body	string	no		`always`, `hourly`, `daily`, `weekly`, `monthly`, `yearly` or `never`.
`priority`	query\|body	boolean	no	`true`	Add `<priority>` by depth.
`format`	query\|body	string	no	`json`	`json` or `xml` (the sitemap.xml file).

Example

curl "https://apibox.io.vn/v1/sitemap-generate?url=https://www.sitemaps.org/&max_pages=10" \
  -H "Authorization: Bearer YOUR_API_KEY"

const res = await fetch('https://apibox.io.vn/v1/sitemap-generate?url=https://www.sitemaps.org/&max_pages=10', {
  method: 'GET',
  headers: {
    Authorization: 'Bearer YOUR_API_KEY',
  },
});
const data = await res.json();
console.log(data);

import requests

res = requests.get(
    "https://apibox.io.vn/v1/sitemap-generate?url=https://www.sitemaps.org/&max_pages=10",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    timeout=30,
)
print(res.json())

<?php
$ch = curl_init('https://apibox.io.vn/v1/sitemap-generate?url=https://www.sitemaps.org/&max_pages=10');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HTTPHEADER => ['Authorization: Bearer YOUR_API_KEY'],
]);
$response = curl_exec($ch);
print_r(json_decode($response, true));

Response

{
  "url": "https://www.sitemaps.org/",
  "site": "https://www.sitemaps.org",
  "pages": 5,
  "complete": true,
  "limits": {
    "max_pages": 10,
    "max_depth": 5,
    "plan_cap": 500
  },
  "robots_txt": "respected",
  "sitemaps_in_robots": [
    "https://www.sitemaps.org/sitemap.xml"
  ],
  "skipped": {
    "robots": 0,
    "noindex": 0,
    "not_html": 2,
    "errors": 0,
    "canonical_elsewhere": 0,
    "redirected_away": 0
  },
  "queued_not_crawled": 0,
  "errors": [],
  "urls": [
    {
      "loc": "https://www.sitemaps.org/",
      "lastmod": "2021-09-23",
      "depth": 0,
      "title": "sitemaps.org - Home"
    },
    {
      "loc": "https://www.sitemaps.org/faq.html",
      "lastmod": "2021-07-08",
      "depth": 1,
      "title": "sitemaps.org - FAQ"
    },
    {
      "loc": "https://www.sitemaps.org/protocol.html",
      "lastmod": "2022-12-15",
      "depth": 1,
      "title": "sitemaps.org - Protocol"
    }
  ],
  "xml": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<urlset xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\">\n  <url><loc>https://www.sitemaps.org/</loc><lastmod>2021-09-23</lastmod><priority>1.0</priority></url>\n  <url><loc>https://www.sitemaps.org/faq.html</loc><lastmod>2021-07-08</lastmod><priority>0.8</priority></url>\n  <url><loc>https://www.sitemaps.org/protocol.html</loc><lastmod>2022-12-15</lastmod><priority>0.8</priority></url>\n  …\n</urlset>\n"
}

Use cases

Create a first sitemap.xml for a site without a CMS plugin
Audit which pages search engines can reach
Compare the crawled pages with the declared sitemap
Find pages hidden by noindex or a wrong canonical

FAQ

Why are some pages missing?

They are not linked from crawled pages, are blocked by robots.txt, noindex or a canonical elsewhere, or the page limit was reached: see skipped and stopped_by.

Can it crawl JavaScript-only sites?

No, links are read from the HTML returned by the server; pages rendered only in the browser are not discovered.

How many pages can it crawl?

Up to 50 (Free), 200 (Starter) or 500 (Pro) pages per call, 20 on the online tool.

Sitemap Generator API

Parameters

Example

Response

Use cases

FAQ

Related APIs

URL Metadata

HTTP Headers

Redirect Checker