@pipeworx/web-fetch

Connect: https://pipeworx.io/mcp — every tool in the catalog, including @pipeworx/web-fetch’s. Install: one-click buttons

No MCP client? Skip the connection: POST https://gateway.pipeworx.io/v1/tools/search_packs {"query":"..."} to find a tool below, GET /v1/tools/<name> for its schema, POST the same URL with arguments for the data — see For AI agents.

Tools: 2

Fetch the exact contents of one public URL, live from its origin server, and get back the bytes (or text decoded from them) with a SHA-256 hash of exactly what the origin sent, plus the redirect chain and the response headers. Also lists and extracts named members from a ZIP archive at a public URL, without downloading the whole archive when the origin honours HTTP Range.

Tools

• fetch_url(url, format?, max_bytes?, offset?) — one GET of one public http(s) URL. Returns status, url_final, redirects (present only when the URL redirected), headers (content-type, content-length, last-modified, etag, content-language, date), content_family (text | pdf | archive | binary), bytes_total, sha256 over the complete body, and a window of the body:

  • format: "auto" (default) — readable text for HTML and PDF, the raw body for everything else.
  • format: "raw" — the body itself: decoded text for text types, base64 for binary (zip, xlsx, gzip, PDF). sha256_returned hashes exactly the bytes in the window.
  • format: "text" — extracted text (text_extraction: html | pdf | none | failed). PDF extraction warnings are passed through in warnings.
  • max_bytes (default 100,000; at most 1,000,000 for text, 512,000 for base64) and offset page through a large body; follow next_offset.

  Text is decoded from the bytes, never guessed: the charset comes from a BOM, then the Content-Type header, then the page’s own <meta charset> or XML declaration, then UTF-8 if the bytes are valid UTF-8, else windows-1252. charset_source says which rule decided and decode_replacements counts any characters lost in decoding (0 means none).

• extract_archive_member(url, member?, format?, max_bytes?, offset?) — list or extract a named member from a ZIP archive at url.

  • No member: lists members — name, compressed_size, uncompressed_size, method (stored | deflate | method_N), crc32, local_header_offset, encrypted. members_truncated says if there were more than members_returned.
  • With member (exact name, falling back to a case-insensitive match — matched_by says which): decompresses that one entry and returns it — format: "auto"/"raw" give decoded text for a textual member and base64 for binary; "text" forces a text decode (falls back to base64 with a warning if the member looks binary). Windowed by max_bytes/offset exactly like fetch_url, over the DECOMPRESSED bytes. crc32_verified confirms the decompressed bytes match the archive’s own checksum.
  • Reads only the end-of-central-directory record and the central directory — at the tail of the file — to list members, and only the wanted member’s local header + payload to extract it, all via HTTP Range. A multi-GB archive costs the same few requests as a tiny one, as long as the origin honours Range (most do, including Internet Archive Wayback Machine captures); otherwise it falls back to a capped full download (range_supported: false).
  • Zip-bomb protection is two independent layers: the central directory’s declared uncompressed size is checked against the cap BEFORE any byte is decompressed, and the real decompressed byte count is ALSO capped DURING decompression — so a member that lies about its own declared size is still caught. Encrypted entries, unsupported compression methods (only stored and deflate), and archives whose own end-of-central-directory record is ZIP64 (more than 65,535 entries, or a 4GB+ central directory) are refused by name rather than mis-read.
  • Refusals: member_not_found, encrypted_entry, unsupported_compression_method, decompressed_too_large (checked both before and during decompression — see above), too_many_members, directory_too_large, zip64_not_supported, too_large (Range not honoured and the archive exceeds the read cap), invalid_archive (not a ZIP, or corrupt), plus fetch_url’s own blocked_url / upstream_refused / upstream_down / budget_exhausted for reaching the URL in the first place.

Checking “verbatim” yourself

sha256 is taken over the entity body after transfer compression is removed — the same bytes curl --compressed -s <url> | shasum -a 256 writes. Two caveats: a dynamic page can legitimately hash differently a second later, and some sites serve different content by location, so the answer is verbatim as served to this fetch (colo, when known, names the edge location).

Refusals — always loud, never an empty body

Limits

• One GET, no cookies, no login, no caller headers, no JavaScript execution. likely_needs_js: true flags a page that renders its content with scripts.
• 10 MB read per call; a larger body returns the first window with truncated: true, body_complete: false and sha256: null.
• 25 s per call, at most 4 redirects.
• Per site (registrable domain), across all callers: 30 requests a minute and 1,000 a day (sec.gov: 5 a minute). A 200 response is reused for 10 minutes (served_from: "cache", cache_age_s).
• Identifies itself honestly as Pipeworx/1.0 (pipeworx.io); on a 403 it retries once as Pipeworx/1.0 (+https://pipeworx.io; support@pipeworx.io) and reports which it used in user_agent_used.

Auth

Keyless.

Data sources

• Whatever public URL the caller names. Nothing is fetched except that URL and its redirects, plus a DNS-over-HTTPS lookup at https://cloudflare-dns.com/dns-query to refuse names that resolve to private addresses.

Tools

• fetch_url — Fetch the exact contents of ONE public web URL live from its origin server and return them verbatim with a SHA-256 hash of the bytes received.
• extract_archive_member — List or extract a named member from a ZIP archive at a public URL, without downloading the whole archive when the origin honours HTTP Range (most do, including Internet Archive Wayback Machine capture