EightVisionsCrawler

Overview

EightVisionsCrawler is an automated web crawler operated by 8 Visions. It is used for discovering and indexing publicly available web pages in order to improve our search engine and related research tools.

The crawler is designed to be polite and to respect robots.txt rules as well as standard web crawling conventions.

What the crawler collects

EightVisionsCrawler only collects technical and SEO-related metadata from publicly accessible pages. It does not intentionally collect or store personal information.

• Page URL
• HTTP status code
• Content type (e.g., text/html)
• Approximate content length
• HTML <title> text
• <meta name="description"> content (if present)
• <meta name="robots"> and X-Robots-Tag headers
• Internal links used to discover other pages on the same site

The crawler does not store full page content and attempts to avoid storing obvious personal identifiers (such as email addresses or phone numbers) in its metadata fields.

Respect for robots.txt and crawling rules

EightVisionsCrawler obeys the Robots Exclusion Protocol (robots.txt) .

• Before crawling a site, the crawler requests /robots.txt and interprets rules for its specific User-Agent and for User-agent: *.
• If robots.txt is missing, the site is treated as crawlable by default (subject to standard HTTP errors and availability).
• If robots.txt cannot be read or explicitly disallows /, the crawler treats the site as non-crawlable.
• The crawler also respects noindex and nofollow directives in X-Robots-Tag headers and in HTML <meta name="robots">.

How to block EightVisionsCrawler using robots.txt

If you do not want EightVisionsCrawler to crawl your site, you can block it in your robots.txt file. Place a file named robots.txt at the root of your domain (for example: https://example.com/robots.txt) and add rules like the following:

User-agent: EightVisionsCrawler
Disallow: /

User-agent: *
Disallow:

The example above:

• Blocks EightVisionsCrawler from requesting any paths on your domain.
• Allows all other crawlers (matching User-agent: *), unless they have their own specific rules.

If you want to allow most of your site but block specific sections, you can do:

User-agent: EightVisionsCrawler
Disallow: /private/
Disallow: /tmp/

User-agent: *
Disallow:

In this example, EightVisionsCrawler may crawl your public pages but will not crawl anything under /private/ or /tmp/.

Rate limiting and politeness

EightVisionsCrawler is designed to crawl politely:

• It spaces out requests to avoid overloading your server.
• It honors Crawl-delay directives when present in robots.txt.
• It does not attempt to bypass authentication, paywalls, or other access controls.

Contact

If you believe EightVisionsCrawler is not respecting your robots.txt rules or is causing issues for your site, please contact the site operator and include:

• Your domain name
• Relevant log excerpts (including timestamps and User-Agent)
• A brief description of the issue

We will investigate and take action to ensure the crawler continues to behave politely and in accordance with your published rules. support@8visions.online