URL fetcher

The URL converter fetches a public page only after a user submits that exact URL. Requests identify themselves as markitdown.ai-fetcher/1.0 (+https://markitdown.ai/docs/fetcher). This page is the operator-facing identity document for that user agent. Product behavior for researchers lives on /url-to-markdown. API clients that pass a url to POST /v1/convert/url go through the same fetch rules.

• Private, reserved, and non-web addresses are blocked before connection; only ports 80 and 443 are used.

• Response time (15 seconds by default), decompressed size, redirects (at most 4), and requests per target host are limited.

• To block the fetcher immediately, deny the markitdown.ai-fetcher user agent at your edge. Contact support@markitdown.ai if you need help or want a domain blocked.

The fetcher requests only the exact public URL a user submitted for conversion. It identifies itself as markitdown.ai-fetcher/1.0 (+https://markitdown.ai/docs/fetcher). It does not log into paywalls, and it does not crawl a site looking for extra pages. JavaScript-only applications may return an empty shell because the fetcher reads the first server-rendered HTML. Saved .html files belong on the POST /v1/convert/html upload path, not this fetch path. See /url-to-markdown for the product landing that uses this fetcher.

Deny the markitdown.ai-fetcher user agent at your edge. Private, reserved, and non-web addresses are already blocked before connection. Response time, decompressed size, redirects, and requests per target host are limited. Email support@markitdown.ai if you want a domain blocked on our side. The fetcher is used only when a person or API client submits that exact URL. It is not a site-wide crawler and it does not follow arbitrary in-page links. Operators who already block unknown bots can add this user agent to the same list. Blocking it stops URL conversion for that host; uploaded HTML files are unaffected.