Winkling out spider behavior

 kw: blogs, blogging, spider scanning, inferences

I just opened this blog to write a new post and I first checked the statistics. Each day for the past week or so there has been a burst of activity between 8am and 10am local time (EDT), amounting to 15,000 to 20,000 views. I saw that today there had been a sudden burst of extra activity starting about 9am, that was still going on. Here is the "Now" view of the stats for the past two hours (~9:30-11:30):

I just barely caught this in time to see the tail end of the pattern. Prior to 11am, the hit rate was around 2.5 per second or ~9,000 per hour (150-ish per minute), according to the "Past 24 hours" view. Here we see that after 11 it dropped to about half that for twenty minutes, then went to the more ordinary "background" rate of just one view or fewer per minute. The total number of hits in this period is about 15,400.

When I looked at the post-by-post stats, I realized that the majority of these views were of the landing page, which contains the most recent 10-15 posts, depending on their size. To get to individual pages requires opening a link or two in the navigator then linking to the page one wishes to view. The number of page views during these two hours was about 700, or about 4.5% of the total hits.

I infer that when a spider—whether of the spying type or the AI training type—encounters this blog, it usually snarfs up what it can from the home page and then goes elsewhere. Sometimes or in some cases a spider may go deeper, because 700 page views in two hours is still at least ten times the "honest" page view rate, the rate of humans actually going to a page to read it.

Considering that the oldest posts in this blog are 21 years old, it makes sense for an AI training spider to concentrate on newer material.

Surging spiders

 kw: blogs, blogging, spider scanning, AI training

I checked on the blog just now, and saw that the view rate more than doubled in the past day to 56,000. Here is the chart for the past 24 hours:

The peak rate in the seven-o'clock hour was 11,440. The average rate during that hour was more than three per second. A full day at that rate would be almost 275,000 views! I checked the "Now" view, to see the past two hours:

The surge around 7:40 (my time, EDT) peaked at around 470, and that 10-minute surge totaled about 4,000 views (~6.5/second), 27% of the total for these two hours, and a daily rate over 575,000. The plateau that followed, at 150/minute, totaled about 5,800, and since then the per-minute rate has been around 30, varying from 15-60.

I have belatedly considered the number of views of the posts themselves, and the numbers don't add up. In a "posts statistics" view I am shown the number of views for the twenty most popular posts for a period, from which I can project the rest (there are 2,717 today) by fitting a lognormal distribution. The total views of the blog in any period, from an hour to a year, is five to ten times the sum of the views of all the posts. 

I conclude that 80-90% of the views recorded are hits on the blog that are not taken further. That is, no posts are opened. Whatever algorithm it is that "finds" this blog apparently checks the home page, and most of the time, drops it without going further. Oh, well. If the 10:1 ratio is the most likely, it still means that in the past 24 hours, about (at least?) 5,600 posts have been opened, and the hits have been scattered throughout the blog's history, on average opening each one twice. Compare this to the likely "real" activity of about four views per hour, wherever someone's search for a topic lands them.

When most of the spiders skid to a stop

 kw: blogs, blogging, spider scanning, AI training

For the past three days the number of daily views has been around 2,000 rather than 20,000 (up to 60,000 a time or two). I checked the "Now" view, which shows minute-by-minute activity in a two-hour span, and I got lucky.

As the chart shows, for the first 82 minutes things were rocking along at a rate of about 3.6 per minute, or about 5,200 per hour, a significant increase over the prior day. The last 38 minutes, only 20 views occurred, a rate of about 760 per day.

What are the chances that the "legitimate" hit rate is in this range? Or is it still about 100 per day? I surmise that several spidering efforts came to a halt about 4:35, and one spider is still running.

Spiders of Brazil and US now neck-and-neck

 kw: blogs, blogging, spider scanning, AI training

Before Noon today (5/13) I checked the stats and saw that today and the previous four days the activity has been 20,000 or greater. The first chart (second image) shows the hits over the 24 hours up until about Noon today. The world stats, seen in the third image, show that between about 8am and 10am, Brazil and the US were a close 1-2.

Earlier posts on this subject showed spider cartoons that were generated by Gemini. I couldn't induce it to produce an image I liked this time; this image is from Leonardo AI using the Recraft V4 engine.

A day contains 1,440 minutes; 20,000 views are on average nearly 14 per minute. At a 30,000 rate, nearly 21. But this chart shows that on an hourly scale, the rate is not very regular.

Focusing on two hours (which I did first), we see the country breakdown in that short interval.

This is notable for a missing country: China. It might be among "other", which comprises 64% of the data, but I think not. China has recently cracked down severely on external access by its citizens, and is not accessible via VPN…not that anyone from a free country would be willing to try to use the Internet from "inside" China. 

Considering that this blog has a little less than 3,000 posts, any location running at a rate of 125 views per hour, and spreading their views around, would be downloading the entire blog each 24 hours. Brazil, with 191 views in two hours, is apparently snarfing up the whole thing about every 30 to 31 hours.

As I've said before, I hope these folks are getting value for their effort.

Brazil continues to top the spider races

 kw: blogs, blogging, spider scanning, AI training

The past two days this blog's activity has been in the range of 20,000 views daily. Spiders continue to run rampant. As the chart below shows, Brazil tops the list, with 15% of the total.

First, it's interesting to see the daily totals in the first graph. Between a day with ~20k and the past two days at the same level we see a few days in the 5,000 range. This seems to be the new normal, even though actual readers of the articles being "viewed" probably amount to about 100 per day.

The worldwide breakdown is interesting. While the US is second with 6% of total views for the week, quite a variety of countries are represented with at least a couple of percent each. Also, "Other" is still more than half the total, to be divvied up among the other 180 or so countries of the world.

I had read that Brazil has become a hotbed of AI training. This supports that. I suspect many of the seemingly offbeat countries in the mix are actually cases of VPN attribution, with origins mainly in the US and Brazil.

Spiders multiplying?

 kw: blogs, blogging, spider scanning, AI training

Wow! I thought spider scanning the past few days was heavy. Today the total is about 50,000 views. A new record by a large factor. The spiders seem to be multiplying.

This is the rate chart for the past 24 hours:

A blast of activity, that peaked above 6,500/hour, hit a lull, and then rose to roughly one hit per second (~3,600/hr) for the past eight hours. I looked at the "now" record to see that in detail, there are peaks of 2-3 per second with little dips between, but few minutes pass that contain fewer than 20 hits each. Here is the country breakdown for the past 24 hours:

Brazil again tops the list with about 17% of the total, with the US and Iraq (!) tied for second with 5% each. As usual, many countries have large counts. I can only attribute this to AI training. I mean, now much spying could be going on related to the Ukraine and Iran wars? Well, perhaps a lot. Particularly with a critical deadline arriving in three hours as I write this.

I guess more frequent checking is warranted. This may be a barometer of something…

Brazil spiders go bonkers, leading the world

 kw: blogs, blogging, spider scanning

The stats on this blog went bonkers the past two days, ~20,000 hits yesterday and ~10,000 so far today. Here is the view for the past week, of the places of origin; about 46,000 hits for the whole week:

Sorry for the fuzziness; I had to reduce scale to get everything on one screen. Brazil tops the list, yet it accounts for only 14.4% of the world total. There appear to be numerous AI training exercises going on, via VPN, and I suspect most of them originate in the US and Vietnam.

Using the "Now" selection, here is a shorter list of places:

This is the "hit list" for the past two hours. Prior to AI training and other uses of automated spider scanning, this blog's peak popularity varied between 100-200 views per day, and for most of the past 10 years, 50-100 per day. The minute-by-minute breakdown:

A few of these peaks exceeded 30 views per minute. On average, it's 8-9 per minute. I sure hope somebody is getting value for their effort! I'd like to think that some substantial amount of this activity represents humans actually reading my posts. Yeah, and bees have lips!

A new spider racing record

 kw: blogs, blogging, spider scanning

I finished the prior post and looked at the stats. Today and yesterday, the number of views exceeds 33,000 per day. That means my entire blog was downloaded about twelve times each day! The spiders are racing through the blogosphere.

Gemini produced this image using Nano Banana 2. It also provided cute commentary on the details, such as the expressions, the spiders in the grandstand, etc. Seems rather proud of itself.

This is the hour-by-hour trace for the past 24h:

At the peak, the rate is nearly 2 hits per second, for an entire hour. 35,200 in 12 hours...with a good, long run, about six hours, exceeding one hit per second.

I hope someone is getting something useful from this!

A spider blast

 kw: blogs, blogging, spider scanning, ai training

I thought yesterday's spider outbreak was the peak. The peak is being sustained, or perhaps I should speak of a range of peaks. Here is the profile of views in the past 24 hours (captured at 10am):

A simple, twelve hour sawtooth burst. At the peak, 3,012 views per hour is almost one per second. To me, that seems like a lot, but I considered popular blogs that have millions of followers; they probably have thousands of views per second. Nonetheless, the barely visible background rate of perhaps four or five per hour is what I consider normal for this blog.

The geographic chart carries little useful information:

I didn't even bother opening up the "Other" section, which encompasses 43% of the 20,700 hits in the past day. A more normal day's activity would consist of less than 100 views, 2/3 from the US, and a total of no more than 20 countries.

I have been attributing this to AI/LLM training. Now I wonder whether the various AI Assistants such as CoPilot and Gemini 3 are performing blog scans as part of their "research" when answering a question. That seems to me better than the prospect of all this activity being only for LLM training.

Spiders on a rampage

 kw: blogs, blogging, spider scanning, ai training

In the past two hours this blog has received 4,016 views. I am pretty sure that no more than 100 of them are legitimate persons who might, perhaps, read a post or two. The rest are spiders of one kind or another. When I first noticed surges of spidering a few years ago, the source was Russia, most likely Russian hackers, or perhaps Russian governmental agents looking for negative press in the blogosphere. In the past year or so, multiple governments are interested in what bloggers have to say, but the bigger source of suction is AI training.

4,016 views averages 33+ per minute. Longer term, in the past two days, the daily total is about 12,000. Here is the minute-by-minute rate:

This minute-by-minute variability is quite different from the steady blasts of 50 hits at a time, every few minutes, that characterized the Russian spidering a decade ago. And the next chart shows how the hits are scattered around the world:

This shows that numerous entities are active, probably all using VPN's. There's no telling where any of these actually originate. And notice the large remnant in "Other", fully one-third of the total, scattered among the other 190 or so countries of the Earth. To reprise: The number of "legitimate" hits in this two-hour interval is about five.

Considering the tendency of blogs to be more inflammatory and biased than actual journalism, and even much more so than published (via a publishing house or peer-reviewed journal) material, training LLM's via the blogosphere ought not be done. It is creating digital "snowflakes" and digital sociopaths in great numbers.

Spiders aroar - AI training?

kw: blogs, blogging, spider scanning, AI training

About two months ago, July 16, the number of views this blog experienced was 77. That is "about right", historically. For the last six months, that was the closest to a "normal" day; no other day has seen fewer than 200 views, and the rate has frequently been in the thousands per day.

Firstly, here we see the past 24 hours (from Noon 9/4/25 to Noon today):

Even the "dull roar" of the first 14 hours of this record (~100/h) exceeds the July 16 (~3/h) rate by a factor of 30, and the rate since then has been has high as 540/h. World distribution for this period is interesting, and a bit surprising:

Surprise #1: The US is at the top; its total slightly exceeds the combined total for Singapore and Hong Kong.

Surprise #2: The top six countries each exceed 77 views per day, and "other" represents a rate that would hit every post in this blog over less than five days.

Let's look at the past month:

The daily rate on August 20 was around 500 views that day. All others were greater. Switching to the other end of the time scale, here is the "Now" picture, the past two hours:

The "clutter" through most of this period is 3-5 views per minute, never less than 2. That comes to around 3,000 per day. The five spikes in the 40-90 view/min range add another 300 or so views, making up more than half the 595 total for this two-hour period. 595 views/2h = 7,140/day. Probably half of that is in such spikes, the rest as ongoing "clutter". For reference, 77 views/day comes to ~6.4/2h, or 0.053 views/min.

I like to think I am truthful and mostly correct. I hope that is a positive influence for the AI agents that are probably being trained with material from this blog. Considering that there are close to two million blogs more popular than this one (in ordinary times, at least), my influence is likely in the range of 0.00001%. Ha!

Normality without spider scanning

 kw: blogs, blogging, spider scanning

I have this browser set to show Blog Stats when it opens. Preparing to review a book, I noticed that the excessive scanning of past weeks had settled down. At about 9:30 local time (EDT), this blog had 34 views, and 209 the day before. This chart shows progress over the past week:

Expected activity without scanning going on is less than 100 per day, so 209 is "getting normal". Here is the picture for the 24 hours ending at 9am today:

Counting back to reach 34 views I find that the "day" counted by the Blogger software began about 7pm the prior day. That means a five-hour offset. If I understand how time zones work correctly, that places the "zero point" in Hawaii or the Aleutian Islands. Counting the rightmost twelve hours on this chart I find 27 views in 12 hours, or 2.25 views per hour, average, a total of 54 views per day. The figure of 97 a the top of this chart indicates a little excess activity, which ended about 16 hours ago (counting from 10am local time). Here is the world chart for the past 24 hours:

This looks a lot like near-normal activity. If this lull continues, I can get a better handle on "the usual."

Spiders multiply and spread out

 kw: blogs, blogging, spider scanning

After a lull, activity—views of my blog posts—has ramped up. I consider a view rate between 50 and 100 per day "normal", or at least ordinary. Let's look at the totals for the past week, on the world map:

The total for the week is more than 16,400, or an average of about 2,300 per day. Just for the 24 hours prior to about 10:00 am EDT, when I captured the view below, the sum is 2,670. But there's quite a difference in where the viewers are reported to be. For the week, Singapore topped the list, followed by Brazil, Vietnam, and the US. For the 24-hour period, Brazil moved to the top, with Vietnam close behind and Singapore a more distant third, with the US about where I'd expect it to be. Now that VPN's are nearly ubiquitous, it isn't even clear that more than one big spider is actually doing all this.

I hope they are all having fun reading nerdy book reviews.

Spiders via Vietnam go hog-wild

 kw: blogs, blogging, spider scanning

I entered to add a post and found the highest activity rate to date. This is the past 24 hours (prior to 10:00 am EDT):

In the recent past Hong Kong and Singapore have provided the bulk of activity, but in the pass few weeks they have been overtaken by Vietnam. Almost 33,000 views! I was curious about the ongoing rate, so I checked the "Now" screen:

The scale is views per minute for the past two hours. The "plateau" sits between 0.7 and 1.3 views per second, and just that 80-minute stretch comes to about 5,000 views, more than 90% tagged to Vietnam.

I say "tagged" because I suspect the country has some fast DNS servers and Asian VPN's are using them. So why the activity? I have one surmise: AI training. Perhaps many blogs are being gulped down wholesale for such purposes.

Hey, guys, if you're relying on my writings for AI training, you're in real trouble!! You'll generate Artificial Nerdism. Maybe that's a redundancy.

Spider activity subsiding

 kw: blogs, blogging, spider scanning

First thing in the morning, what do I see?
No more spiders looking at me.

After several weeks of hot action, things have settled down:

Singapore has been the heavy hitter, but represents only about half of the activity. There's no telling how much of this went through VPN's. I wonder how long things will remain "normal", or perhaps, just ordinary, at 50-100 views per day, or a few thousand per month. To me, that is satisfactory.

American spiders now?

 kw: blogs, blogging, spider scanning

In recent weeks the number of daily views of this blog settled back to fewer than 100, typically 70-80. Today I noticed a big bump. Checking the 24-hour view, I see that whatever is going on is continuing:

It's about 10pm just now. Things started taking off at 4am. Here are the locations:

The top source is the US! How about that!! For the past year or so, when views skyrocketed, either Singapore or Hong Kong led the charge. Not this time. I wonder if the spider source is using a VPN…

Is it spidering or something else?

 kw: blogs, blogging, spider scanning

I was away for a couple of weeks, so it will be a few more days before I have finished reading another book to review. I see that in the past month there was a surge in views for a week or so, with the greatest number emerging from Hong Kong, followed by Singapore:

Viewing from the US and following, with the possible exception of China, represents my usual clientele. Even after this surge, view rates have been high, until just the last day or two. This shows the past week:

One hundred views daily is still well above the usual for this blog. Hong Kong and Singapore still dominate, but by a lesser amount (table follows). For the past 24 hours, we find this activity:

There is a further decrease, but still plenty of activity. To see the trend numerically, here are the percentages:

In spite of a marked decrease in daily views, Hong Kong still dominated today's activity, while Singapore didn't decrease, but its percentage increases because of reduced total activity. I include China because I suspect there is also some "extra" activity from there, but perhaps not all of China is such. The next table shows normalized numbers:

The row labeled "normal" is USA plus Other, and the range of ~800 to ~1,200 is what I expect for a month of activity by genuinely interested viewers, around 30/day.

It is more evident here that activity from Singapore has remained steady while that from Hong Kong reduced to less than 1/3 of the "surge" level. China's activity fell steadily, while "other" stayed level.

How much of this is automatic spidering and how much represents actual persons viewing posts? I can't presume to read minds, not even machine "minds". I suspect that the activity from Hong Kong originates from China as a whole by proxy, while Singapore is its own animal. Perhaps I should say I "speculate" rather than "suspect". I can analyze numbers all day long, but little further insight is likely.

Spiders from Hong Kong outrun those from Singapore

kw: blogging, spiders, spider scanning

 

Just when I'd got used to seeing another extra 400-600 views per day, just from Singapore, today I saw this, for the past 24 hours:

Here, the usual "Singapore plus everyone else" is the low terrain on either side of the "mountain". Naturally, I had to see if this is also Singapore. As it happens:

No! Views from Singapore are indeed many, at 555 for the day, but Hong Kong, in an eight-hour spree, racked up 5,225 views, or about 650 per hour. Considering that the total number of posts in this blog is about 2,500, I suspect that my blog is one of perhaps millions of blogs that are being gulped down wholesale to train AI text models such as ChatGPT and, where images and text occur together, diffusion models such as DALL•E or Playground. I checked into the top posts for the past day:

Everything below the first ten posts has been viewed no more than twice in the past 24 hours. The "text vacuuming" going on is unfocused. At this point, I don't what more to conclude. It would be interesting to learn if other bloggers are experiencing new and unusual numbers of hits, most likely from automated text vacuums.

Singapore spiders, and others

 kw: blogs, blogging, spider scanning

The past few days I've noticed a big increase in views of this blog. It has been a while since the last flurry of spider scanning. Here is what the past month looks like:

The usual daily activity is 30-40 views; about 1,000 per month. Let's look at the past week (Blogger doesn't have a custom date setting):

Matters get a little more clear when we look at the sources:

In the past the primary spider source was Russia, sometimes Ukraine. This time it is Singapore, with supporting roles from Hong Kong and China. If you add these numbers, leaving out those three countries, then add back in 45 for China and 90 for Hong Kong, you get a usual month of views for this blog. In a typical month, China may have 1-2 views daily and Hong Kong 2-4. Singapore is a rare "customer".

I wonder what they were looking for? The total blog has roughly 2,500 posts…