PepGrades
Menu

Data collection

How PepGrades collects prices.

PepGrades reads public vendor pages so price comparisons stay current. This page explains what we fetch, how often, the limits we keep, and how to reach us.

What we fetch.

  1. Public product, price and stock information: product pages and the public store APIs behind them, such as WooCommerce and Shopify product listings.
  2. Public promotion pages: the home page banner, sale, promotion, FAQ, rewards and membership pages, and the sitemaps used to find them.
  3. Anonymous cart checks for automatic deals, only for offers whose deal could change the price we show: we add the item to a fresh, anonymous cart and read the result. We never start checkout and never pay.
  4. Public certificates of analysis (COAs) that vendors publish.

How often.

  1. Prices and stock: once a day.
  2. Deal checks: when a vendor's promotion text changes or a price we would show changes, and otherwise about once a week.
  3. COAs: once a week.

The limits we keep.

  1. We follow each site's robots.txt, including up to five redirects from it. If robots.txt is missing (a 4xx answer) or comes back as an ordinary web page, we treat the site as having no rules for us; if its server errors or does not answer, we request nothing else from that site that day. Our weekly certificate check is stricter: it also stops if robots.txt answers 401 or 403.
  2. One request at a time to a site, at least 2 seconds apart.
  3. At most 25 requests per vendor per day for prices and deal checks together. Anything that does not fit waits for the next day. The weekly COA check has its own limit of 40 requests per site.
  4. If a site answers a deal check or COA check with a rate limit (429), a refusal (403) or a bot challenge, or answers the price check with a rate limit or a challenge, we pause deal checks and COA checks on that site for 6 days. The daily price check continues with a few requests.
  5. No logins, no account creation, no contact or coupon forms, and no checkout. The only thing we ever submit is an anonymous add-to-cart request for a deal check.
  6. When a store's catalogue only appears after its own scripts run, we load those catalogue pages the way a browser does, which also fetches the page's own scripts and images.

How to recognize us.

Every request identifies itself with the user agent PepGrades/1.0 (+https://pepgrades.com/data-collection). To set rules for us in robots.txt, use the name PepGrades.

Questions or changes.

If you run a store we check and want us to slow down, use a particular feed, allowlist our requests, or stop collecting from your site, email admin@pepgrades.com with your site address and what you would like changed.