Tuesday, May 19, 2026

When most of the spiders skid to a stop

 kw: blogs, blogging, spider scanning, AI training

For the past three days the number of daily views has been around 2,000 rather than 20,000 (up to 60,000 a time or two). I checked the "Now" view, which shows minute-by-minute activity in a two-hour span, and I got lucky.

As the chart shows, for the first 82 minutes things were rocking along at a rate of about 3.6 per minute, or about 5,200 per hour, a significant increase over the prior day. The last 38 minutes, only 20 views occurred, a rate of about 760 per day.

What are the chances that the "legitimate" hit rate is in this range? Or is it still about 100 per day? I surmise that several spidering efforts came to a halt about 4:35, and one spider is still running.

No comments: