---
title: "URL fetcher"
description: "How markitdown.ai fetches public pages submitted for URL conversion, including identity and opt-out."
url: https://markitdown.ai/docs/fetcher
updated: 2026-09-25
source: markitdown.ai
---

# URL fetcher

> How markitdown.ai fetches public pages submitted for URL conversion, including identity and opt-out.

## Identity and purpose

The URL converter fetches a public page only after a user submits that exact URL. Requests identify themselves as `markitdown.ai-fetcher/1.0 (+https://markitdown.ai/docs/fetcher)`. This page is the operator-facing identity document for that user agent. Product behavior for researchers lives on /url-to-markdown. API clients that pass a `url` to `POST /v1/convert/url` go through the same fetch rules.

## Limits and opt-out

- Private, reserved, and non-web addresses are blocked before connection; only ports 80 and 443 are used.
- Response time (15 seconds by default), decompressed size, redirects (at most 4), and requests per target host are limited.
- To block the fetcher immediately, deny the `markitdown.ai-fetcher` user agent at your edge. Contact support@markitdown.ai if you need help or want a domain blocked.

## What does the fetcher request?

The fetcher requests only the exact public URL a user submitted for conversion. It identifies itself as `markitdown.ai-fetcher/1.0 (+https://markitdown.ai/docs/fetcher)`. It does not log into paywalls, and it does not crawl a site looking for extra pages. JavaScript-only applications may return an empty shell because the fetcher reads the first server-rendered HTML. Saved `.html` files belong on the `POST /v1/convert/html` upload path, not this fetch path. See /url-to-markdown for the product landing that uses this fetcher.

## How do I block the fetcher?

Deny the `markitdown.ai-fetcher` user agent at your edge. Private, reserved, and non-web addresses are already blocked before connection. Response time, decompressed size, redirects, and requests per target host are limited. Email support@markitdown.ai if you want a domain blocked on our side. The fetcher is used only when a person or API client submits that exact URL. It is not a site-wide crawler and it does not follow arbitrary in-page links. Operators who already block unknown bots can add this user agent to the same list. Blocking it stops URL conversion for that host; uploaded HTML files are unaffected.

## Related

- [Errors & limits](https://markitdown.ai/docs/errors-and-limits)
- [Introduction](https://markitdown.ai/docs)
- [Document Conversion APIs](https://markitdown.ai/developers)