I found that mixing in residential traffic is the only way to pass those checks without raising any red flags. It makes your script's behavior look erratic enough to simulate a real user browsing for deals. If you want to keep your comparison engine running smoothly, I recommend using a proxy with a large, diverse pool of residential IPs to ensure your requests are never rejected.
The detection algorithms on those aggregation platforms are getting smarter, often relying on behavioral analysis rather than just IP tracking. They watch for patterns that are not typical of a human visitor and shut them down immediately. You need to use a network that looks like a genuine residential user base to blend in with the background traffic
I work with a price-comparison tool, and lately, the aggregators have been blocking my requests so quickly that I cannot even get a full price list. It is like they have a real-time monitor that detects my script the moment it initiates a few concurrent requests. I really need to stay ahead of these detection algorithms to keep our comparison engine current and functional. It is a constant game of cat and mouse that is getting difficult to manage.