URL fetcher

The URL converter fetches a public page only after a user submits that exact URL. Requests identify themselves as markitdown.ai-fetcher/1.0 (+https://markitdown.ai/docs/fetcher). This page is the operator-facing identity document for that user agent. Product behavior for researchers lives on /url-to-markdown. API clients that pass a url to POST /v1/convert/url go through the same fetch rules.

• Private, reserved, and non-web addresses are blocked before connection; only ports 80 and 443 are used.

• Response time (15 seconds by default), decompressed size, redirects (at most 4), and requests per target host are limited.

• Requests send Accept: text/markdown, text/html;q=0.9. A site that answers with its own Markdown has it used directly when it passes a quality check; otherwise the HTML page is fetched and converted.

• When a page's first HTML is an empty JavaScript shell, it is rendered once in a headless browser. The browser's user agent ends with the same markitdown.ai-fetcher/1.0 token, and it requests only the page, its scripts, and its data calls — no images, fonts, or stylesheets. Every one of those requests passes the same private-address checks.

• To block the fetcher immediately, deny the markitdown.ai-fetcher user agent at your edge. Contact support@markitdown.ai if you need help or want a domain blocked.

The fetcher requests only the exact public URL a user submitted for conversion. It identifies itself as markitdown.ai-fetcher/1.0 (+https://markitdown.ai/docs/fetcher). It does not log into paywalls, and it does not crawl a site looking for extra pages. If the first HTML is an empty JavaScript shell, the page is rendered once in a headless browser: the page's own scripts and data requests run, but nothing is clicked, scrolled, or submitted, and images, fonts, and stylesheets are not downloaded. Saved .html files belong on the POST /v1/convert/html upload path, not this fetch path. See /url-to-markdown for the product landing that uses this fetcher.

Deny the markitdown.ai-fetcher user agent at your edge. Private, reserved, and non-web addresses are already blocked before connection. Response time, decompressed size, redirects, and requests per target host are limited. Email support@markitdown.ai if you want a domain blocked on our side. The fetcher is used only when a person or API client submits that exact URL. It is not a site-wide crawler and it does not follow arbitrary in-page links. Operators who already block unknown bots can add this user agent to the same list. Blocking it stops URL conversion for that host, including browser rendering, because the rendering browser carries the same user agent token; uploaded HTML files are unaffected.