Surging spiders

 kw: blogs, blogging, spider scanning, AI training

I checked on the blog just now, and saw that the view rate more than doubled in the past day to 56,000. Here is the chart for the past 24 hours:

The peak rate in the seven-o'clock hour was 11,440. The average rate during that hour was more than three per second. A full day at that rate would be almost 275,000 views! I checked the "Now" view, to see the past two hours:

The surge around 7:40 (my time, EDT) peaked at around 470, and that 10-minute surge totaled about 4,000 views (~6.5/second), 27% of the total for these two hours, and a daily rate over 575,000. The plateau that followed, at 150/minute, totaled about 5,800, and since then the per-minute rate has been around 30, varying from 15-60.

I have belatedly considered the number of views of the posts themselves, and the numbers don't add up. In a "posts statistics" view I am shown the number of views for the twenty most popular posts for a period, from which I can project the rest (there are 2,717 today) by fitting a lognormal distribution. The total views of the blog in any period, from an hour to a year, is five to ten times the sum of the views of all the posts. 

I conclude that 80-90% of the views recorded are hits on the blog that are not taken further. That is, no posts are opened. Whatever algorithm it is that "finds" this blog apparently checks the home page, and most of the time, drops it without going further. Oh, well. If the 10:1 ratio is the most likely, it still means that in the past 24 hours, about (at least?) 5,600 posts have been opened, and the hits have been scattered throughout the blog's history, on average opening each one twice. Compare this to the likely "real" activity of about four views per hour, wherever someone's search for a topic lands them.

When most of the spiders skid to a stop

 kw: blogs, blogging, spider scanning, AI training

For the past three days the number of daily views has been around 2,000 rather than 20,000 (up to 60,000 a time or two). I checked the "Now" view, which shows minute-by-minute activity in a two-hour span, and I got lucky.

As the chart shows, for the first 82 minutes things were rocking along at a rate of about 3.6 per minute, or about 5,200 per hour, a significant increase over the prior day. The last 38 minutes, only 20 views occurred, a rate of about 760 per day.

What are the chances that the "legitimate" hit rate is in this range? Or is it still about 100 per day? I surmise that several spidering efforts came to a halt about 4:35, and one spider is still running.

Spiders of Brazil and US now neck-and-neck

 kw: blogs, blogging, spider scanning, AI training

Before Noon today (5/13) I checked the stats and saw that today and the previous four days the activity has been 20,000 or greater. The first chart (second image) shows the hits over the 24 hours up until about Noon today. The world stats, seen in the third image, show that between about 8am and 10am, Brazil and the US were a close 1-2.

Earlier posts on this subject showed spider cartoons that were generated by Gemini. I couldn't induce it to produce an image I liked this time; this image is from Leonardo AI using the Recraft V4 engine.

A day contains 1,440 minutes; 20,000 views are on average nearly 14 per minute. At a 30,000 rate, nearly 21. But this chart shows that on an hourly scale, the rate is not very regular.

Focusing on two hours (which I did first), we see the country breakdown in that short interval.

This is notable for a missing country: China. It might be among "other", which comprises 64% of the data, but I think not. China has recently cracked down severely on external access by its citizens, and is not accessible via VPN…not that anyone from a free country would be willing to try to use the Internet from "inside" China. 

Considering that this blog has a little less than 3,000 posts, any location running at a rate of 125 views per hour, and spreading their views around, would be downloading the entire blog each 24 hours. Brazil, with 191 views in two hours, is apparently snarfing up the whole thing about every 30 to 31 hours.

As I've said before, I hope these folks are getting value for their effort.

Brazil continues to top the spider races

 kw: blogs, blogging, spider scanning, AI training

The past two days this blog's activity has been in the range of 20,000 views daily. Spiders continue to run rampant. As the chart below shows, Brazil tops the list, with 15% of the total.

First, it's interesting to see the daily totals in the first graph. Between a day with ~20k and the past two days at the same level we see a few days in the 5,000 range. This seems to be the new normal, even though actual readers of the articles being "viewed" probably amount to about 100 per day.

The worldwide breakdown is interesting. While the US is second with 6% of total views for the week, quite a variety of countries are represented with at least a couple of percent each. Also, "Other" is still more than half the total, to be divvied up among the other 180 or so countries of the world.

I had read that Brazil has become a hotbed of AI training. This supports that. I suspect many of the seemingly offbeat countries in the mix are actually cases of VPN attribution, with origins mainly in the US and Brazil.

Spiders multiplying?

 kw: blogs, blogging, spider scanning, AI training

Wow! I thought spider scanning the past few days was heavy. Today the total is about 50,000 views. A new record by a large factor. The spiders seem to be multiplying.

This is the rate chart for the past 24 hours:

A blast of activity, that peaked above 6,500/hour, hit a lull, and then rose to roughly one hit per second (~3,600/hr) for the past eight hours. I looked at the "now" record to see that in detail, there are peaks of 2-3 per second with little dips between, but few minutes pass that contain fewer than 20 hits each. Here is the country breakdown for the past 24 hours:

Brazil again tops the list with about 17% of the total, with the US and Iraq (!) tied for second with 5% each. As usual, many countries have large counts. I can only attribute this to AI training. I mean, now much spying could be going on related to the Ukraine and Iran wars? Well, perhaps a lot. Particularly with a critical deadline arriving in three hours as I write this.

I guess more frequent checking is warranted. This may be a barometer of something…

A spider blast

 kw: blogs, blogging, spider scanning, ai training

I thought yesterday's spider outbreak was the peak. The peak is being sustained, or perhaps I should speak of a range of peaks. Here is the profile of views in the past 24 hours (captured at 10am):

A simple, twelve hour sawtooth burst. At the peak, 3,012 views per hour is almost one per second. To me, that seems like a lot, but I considered popular blogs that have millions of followers; they probably have thousands of views per second. Nonetheless, the barely visible background rate of perhaps four or five per hour is what I consider normal for this blog.

The geographic chart carries little useful information:

I didn't even bother opening up the "Other" section, which encompasses 43% of the 20,700 hits in the past day. A more normal day's activity would consist of less than 100 views, 2/3 from the US, and a total of no more than 20 countries.

I have been attributing this to AI/LLM training. Now I wonder whether the various AI Assistants such as CoPilot and Gemini 3 are performing blog scans as part of their "research" when answering a question. That seems to me better than the prospect of all this activity being only for LLM training.

Spiders on a rampage

 kw: blogs, blogging, spider scanning, ai training

In the past two hours this blog has received 4,016 views. I am pretty sure that no more than 100 of them are legitimate persons who might, perhaps, read a post or two. The rest are spiders of one kind or another. When I first noticed surges of spidering a few years ago, the source was Russia, most likely Russian hackers, or perhaps Russian governmental agents looking for negative press in the blogosphere. In the past year or so, multiple governments are interested in what bloggers have to say, but the bigger source of suction is AI training.

4,016 views averages 33+ per minute. Longer term, in the past two days, the daily total is about 12,000. Here is the minute-by-minute rate:

This minute-by-minute variability is quite different from the steady blasts of 50 hits at a time, every few minutes, that characterized the Russian spidering a decade ago. And the next chart shows how the hits are scattered around the world:

This shows that numerous entities are active, probably all using VPN's. There's no telling where any of these actually originate. And notice the large remnant in "Other", fully one-third of the total, scattered among the other 190 or so countries of the Earth. To reprise: The number of "legitimate" hits in this two-hour interval is about five.

Considering the tendency of blogs to be more inflammatory and biased than actual journalism, and even much more so than published (via a publishing house or peer-reviewed journal) material, training LLM's via the blogosphere ought not be done. It is creating digital "snowflakes" and digital sociopaths in great numbers.

Spiders aroar - AI training?

kw: blogs, blogging, spider scanning, AI training

About two months ago, July 16, the number of views this blog experienced was 77. That is "about right", historically. For the last six months, that was the closest to a "normal" day; no other day has seen fewer than 200 views, and the rate has frequently been in the thousands per day.

Firstly, here we see the past 24 hours (from Noon 9/4/25 to Noon today):

Even the "dull roar" of the first 14 hours of this record (~100/h) exceeds the July 16 (~3/h) rate by a factor of 30, and the rate since then has been has high as 540/h. World distribution for this period is interesting, and a bit surprising:

Surprise #1: The US is at the top; its total slightly exceeds the combined total for Singapore and Hong Kong.

Surprise #2: The top six countries each exceed 77 views per day, and "other" represents a rate that would hit every post in this blog over less than five days.

Let's look at the past month:

The daily rate on August 20 was around 500 views that day. All others were greater. Switching to the other end of the time scale, here is the "Now" picture, the past two hours:

The "clutter" through most of this period is 3-5 views per minute, never less than 2. That comes to around 3,000 per day. The five spikes in the 40-90 view/min range add another 300 or so views, making up more than half the 595 total for this two-hour period. 595 views/2h = 7,140/day. Probably half of that is in such spikes, the rest as ongoing "clutter". For reference, 77 views/day comes to ~6.4/2h, or 0.053 views/min.

I like to think I am truthful and mostly correct. I hope that is a positive influence for the AI agents that are probably being trained with material from this blog. Considering that there are close to two million blogs more popular than this one (in ordinary times, at least), my influence is likely in the range of 0.00001%. Ha!