Skip to main content
VoxistryRequest a demo

Voxistry research fetcher

Last updated: September 25, 2026

What it is

VoxistryResearch is the web page fetcher Voxistry uses when a customer's Workroom request, such as a proposal, includes a research step. It fetches individual public web pages so that the request can read and cite them.

It runs only inside a customer's request. It does not crawl or index the web, and it does not follow links it finds on the pages it reads.

How it identifies itself

Every request, including robots.txt requests, is sent with this User-Agent header:

Mozilla/5.0 (compatible; VoxistryResearch/1.0; +https://voxistry.com/research-bot)

The product token it matches in robots.txt is VoxistryResearch.

Blocking it with robots.txt

To block it from your whole site, add this to your robots.txt:

User-agent: VoxistryResearch
Disallow: /

To block only part of your site:

User-agent: VoxistryResearch
Disallow: /private/

The User-agent line is matched without regard to case. If no group names VoxistryResearch, the rules in your User-agent: * group apply.

How it reads robots.txt

  • Before fetching a page, it reads https://<host>/robots.txt for that host. It reads each host's file once per research step.
  • It follows up to five redirects for robots.txt and reads at most 512 KiB of the file.
  • A page your robots.txt disallows for it is not fetched. This is checked again for every redirect target.
  • If robots.txt answers with a 4xx status (for example 404), it proceeds as if the site has no robots.txt rules.
  • If robots.txt answers with a 5xx status, or the site cannot be reached (a network error or a timeout), it does not fetch any page from that site.
  • It does not read Crawl-delay.

What it fetches

  • Only URLs the request already has: a web search result, a page already fetched in the same request, or a URL in the material the customer provided.
  • HTTPS on port 443 only. Plain http URLs are refused, and so are hosts that resolve to private or otherwise non-public addresses.
  • Plain GET requests with Accept: text/html, application/xhtml+xml, text/plain and Accept-Encoding: identity. It follows at most three redirects per page.
  • It uses only responses with status 200 and a content type of text/html, application/xhtml+xml, or text/plain. For any other content type, including PDF, the body is not read.
  • Responses sent with a compressed Content-Encoding are refused.
  • It reads at most 5 MiB of a page and abandons anything larger.
  • It waits up to 5 seconds to connect and up to 15 seconds for each request.

How much it fetches

Each request has a fixed ceiling on page fetches. A proposal request allows at most 12, and no request can be configured to allow more than 120. There is no separate per-site rate limit. A customer's administrators can switch web research off for their account.

Contact

Contact: support@voxistry.com for questions, complaints, or removal requests. Please include your site's host name and the approximate time of the requests you saw.