返回博客

How to protect a website from price scraping and check the result

Spot automated visits to product pages, try a scoped HotWall rule, and check that shoppers and search crawlers can still access your catalogue.

When requests to your catalogue spike, find out which pages received them and where they came from. A promotion, a product feed update, or an automated price scraper can all produce a spike. Start with your web server's access logs for the affected period.

What price scraping can look like in your logs

  1. Pick the paths used by product pages, such as /catalog/. Collect request times, paths, status codes, and client addresses. If your server is behind a proxy, use the client address logged by your trusted proxy or a field configured according to your hosting provider's documentation.
  2. Compare request counts and the number of distinct product pages per source over equal intervals. Use your store's usual traffic at that hour as a baseline, and check for promotions, partner feeds, or catalogue updates during the spike.
  3. Look for repeated sequences of requests across many products alongside changes in server load. Check the schedule of your own integrations. When a client claims to be a crawler, verify its address using the Yandex or Google instructions.

Example using sample data. One source opened 48 distinct product pages in two minutes, from /catalog/chairs/p101 to /catalog/chairs/p148. The sequence repeated ten minutes later. Compare it with normal traffic to that category and check whether a known product feed ran at the same time. Use your store's baseline when deciding what warrants a closer look.

Test a rule on one part of your catalogue

Open Custom rules for a domain you already protect with HotWall. In the visual builder, choose Path, select Contains, and enter a path fragment for the affected category, such as /catalog/chairs/. Add a second condition with AND for the source you verified, such as its IP address. Check whether shoppers or a partner integration could share that address. When your checks point to price scraping, select Challenge and enable the rule for an observation window.

Record when you enabled the rule, the paths it covers, and the metrics you will compare. Start with one rule and expand coverage after checking its effects.

Check shoppers, crawlers, and traffic

  • Shoppers. Open a product page in a fresh browser session, select a product and add it to the cart. Repeat on a phone and check the usual catalogue navigation.
  • Search crawlers. Test the live product URL in Yandex Webmaster and Google Search Console. Confirm the crawler can reach the page and see its product content.
  • Traffic. Compare the category's request counts and status codes over equal windows before and after the change. If shoppers or verified crawlers encounter errors, disable the rule, check access again, and narrow its conditions.

Learn more about HotWall bot management if you want to apply this workflow to your store.

分享: