Sitemap URL extractor: every URL a website lists in its XML sitemaps, up to 100 sites per run: sitemaps are found through robots.txt and the usual paths, sitemap indexes and gzip are followed, and each URL comes with its sitemap, last modification date, change frequency and priority. Pay per URL.
One sitemap URL delivered. Sites without a sitemap are never charged.
ACTOR=steadydata~sitemap-urls URL="https://api.apify.com/v2/acts/$ACTOR/run-sync-get-dataset-items" curl -X POST "$URL?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"sites": ["apify.com"], "maxUrlsPerSite": 5000}'
import os, requests ACTOR = "steadydata~sitemap-urls" URL = f"https://api.apify.com/v2/acts/{ACTOR}/run-sync-get-dataset-items" rows = requests.post( URL, params={"token": os.environ["APIFY_TOKEN"]}, json={'sites': ['apify.com'], 'maxUrlsPerSite': 5000}, timeout=900, ).json() # every row carries a status; failures are records, not exceptions ok = [r for r in rows if r.get("status") == "ok"] print(len(ok), "rows delivered")
const ACTOR = "steadydata~sitemap-urls"; const url = `https://api.apify.com/v2/acts/${ACTOR}/run-sync-get-dataset-items` + `?token=${process.env.APIFY_TOKEN}`; const rows = await fetch(url, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({"sites": ["apify.com"], "maxUrlsPerSite": 5000}), }).then((r) => r.json()); // one object per delivered row, same shape every time console.log(rows.filter((r) => r.status === "ok").length, "rows");
Point an MCP client at Apify's hosted server with this dataset in the tools list, or run our own server locally. The field explanations in the table below travel with the schema, so your agent reads them straight from the platform instead of from this page.
https://mcp.apify.com?tools=steadydata/sitemap-urls # the tool arrives in your agent as steadydata--sitemap-urls
| Field | Type | Name | What it does |
|---|---|---|---|
sitesrequired | array of string | Sites or sitemap URLs | One per row, up to 100: a domain (example.com), a site URL, or a sitemap URL (https://example.com/sitemap.xml). |
maxUrlsPerSite | integer | Max URLs per site | Cost ceiling per site, in sitemap order. |
{
"site": string | null,
"sitemapUrl": string | null,
"url": string | null,
"lastmod": string | null,
"changefreq": string | null,
"priority": number | null,
"imageCount": integer | null,
"status": string | null,
"input": string | null,
"errorCode": string | null,
"error": string | null
}| Field | Type | When it is filled |
|---|---|---|
site | string, null | |
sitemapUrl | string, null | |
url | string, null | |
lastmod | string, null | Filled when the sitemap publishes a last-modified date for the URL. |
changefreq | string, null | |
priority | number, null | |
imageCount | integer, null | |
status | string, null | Either 'ok' or 'error'. An error row carries errorCode and error, and leaves the data fields empty; it is never charged. |
input | string, null | The input this row was built from, so a row can always be traced back. |
errorCode | string, null | Filled on an error row only; a delivered row leaves it empty. |
error | string, null | Filled on an error row only; a delivered row leaves it empty. |
One health report per domain, up to 500 per run: registration and expiry from the official RDAP registry, DNS records with the mail and nameserver provider, SPF and DMARC policy, the TLS certificate with days remaining, and a ranked list of issues. Official protocols only.
$20.00 per 1,000 · Domain reportedOne deliverability report per domain, up to 500 per run: MX and mail provider, the SPF record with its DNS lookup count against the limit of ten, DKIM selectors that really exist, the DMARC policy and reporting, MTA-STS mode, TLS-RPT, BIMI and DNSSEC, plus ranked issues and a score.
$3.00 per 1,000 · Domain checkedText extraction (OCR) from images by URL, up to 200 per run: every line with its confidence and position, the full text joined, image size and format, for screenshots, scans, receipts, menus and product labels, without a browser or an API key. Pay per image with text.
$2.00 per 1,000 · Image readAlready using this one? Ratings are the first thing other buyers look at, and this dataset has none yet. If it does a job for you, a rating on its Apify page is the one thing that helps. It takes a minute and it is the only thing we ask for.