Surging spiders

 kw: blogs, blogging, spider scanning, AI training

I checked on the blog just now, and saw that the view rate more than doubled in the past day to 56,000. Here is the chart for the past 24 hours:

The peak rate in the seven-o'clock hour was 11,440. The average rate during that hour was more than three per second. A full day at that rate would be almost 275,000 views! I checked the "Now" view, to see the past two hours:

The surge around 7:40 (my time, EDT) peaked at around 470, and that 10-minute surge totaled about 4,000 views (~6.5/second), 27% of the total for these two hours, and a daily rate over 575,000. The plateau that followed, at 150/minute, totaled about 5,800, and since then the per-minute rate has been around 30, varying from 15-60.

I have belatedly considered the number of views of the posts themselves, and the numbers don't add up. In a "posts statistics" view I am shown the number of views for the twenty most popular posts for a period, from which I can project the rest (there are 2,717 today) by fitting a lognormal distribution. The total views of the blog in any period, from an hour to a year, is five to ten times the sum of the views of all the posts. 

I conclude that 80-90% of the views recorded are hits on the blog that are not taken further. That is, no posts are opened. Whatever algorithm it is that "finds" this blog apparently checks the home page, and most of the time, drops it without going further. Oh, well. If the 10:1 ratio is the most likely, it still means that in the past 24 hours, about (at least?) 5,600 posts have been opened, and the hits have been scattered throughout the blog's history, on average opening each one twice. Compare this to the likely "real" activity of about four views per hour, wherever someone's search for a topic lands them.