UpContent Bot

UpContent Bot intelligently determines the number of requests to make to your site to ensure that requests are minimized, sporadic, and only made when new information is available to be gathered.

Base Rate Limit: UpContent Bot limits page requests to a maximum of two requests per five seconds per hostname to minimize burden on your servers.

Network conditions and other factors may cause UpContent Bot to occasionally exceed this rate, but all efforts are being made to ensure this is rare. If UpContent Bot continues to request pages at a faster rate for more than two minutes, please contact us and we will address immediately.

UpContent Bot will also limit the number of redirects followed to three per initial URL. Following redirects may temporarily increase our request rate.

UpContent Bot fully respects robots.txt files. We honor both:

• User-agent restrictions: We will not access any paths disallowed for our bot in your robots.txt
• Crawl-delay directives: If your robots.txt specifies a crawl-delay, we will respect it and adjust our request rate accordingly

The crawl-delay directive takes precedence over our default rate limit. For example, if you specify Crawl-delay: 10, we will wait at least 10 seconds between requests to your domain.

You can identify UpContent Bot traffic by the User-Agent header in the HTTP requests we send. This value is typically logged in your server's access logs.

Our User-Agent string is:

Mozilla/5.0 (compatible; UpContent-Test/1.0; +https://qaplatform.upcontent.com/bot)

You can also verify the identity of a particular server that claims to be UpContent Bot by performing a reverse DNS lookup on the IP address of that server.

Using robots.txt (Recommended)

To control UpContent Bot's access to your site, add the following to your robots.txt file:

User-agent: UpContent
Disallow: /path/to/block/
Crawl-delay: 10

To block UpContent Bot entirely:

User-agent: UpContent
Disallow: /

Using HTTP Response Codes

Alternatively, you can configure your server to respond to requests with our user agent with an HTTP 403 Forbidden, 404 Not Found, or 410 Gone response.

If you feel that UpContent Bot is accessing your site too frequently, or if you have any concerns about our crawler's behavior, please contact us at support@upcontent.com.

We will respond to all requests within two business days and work with you to address any concerns.

If you wish to prevent UpContent Bot from accessing content on your site, we would appreciate you letting us know why you do not wish for your site to be accessed by UpContent Bot so we can improve our service.