Last time I saw some excessive crawling on CP I figured it was easier just to throttle the crawler or ban them outright. Many crawlers obey the robots.txt file.
It can be a bit cat and mouse if the crawler ignores that and has a large pool of IPs but I'd be surprised if they overlapped much with real users' IP addresses.
__________________
Uncertainty is an uncomfortable position.
But certainty is an absurd one.
|