AutoArt folder distribution

 kw: analytical projects, art generation, ai art, statistics, statistical distributions, lognormal, scale free

I began using art generating software in November 2022, when DALL-E2 became available. Since then, I've enjoyed having a series of art generating "engines" available, including numerous engines (called "models") in the aggregators Leonardo AI and OpenArt. As often as I can, I generate images for this blog; in some cases, I download images I find on the Internet. However, my primary artistic pastime is creating images of things and scenes I imagine.

Just in the past few days I was inspired by a heavy snowfall to find short poems about snow, and use them to create wintry images. This image was drawn by Nano Banana Pro, under the Leonardo AI umbrella with "None" as the style; that is, native NB Pro. The aspect ratio was set to 16:9. It displays the entire poem, something NB Pro can do better than any other art engine I have found. The prompt was "Watercolor painting evoked by a poem:", followed by the text of the poem "The First Snow" by Charlotte Zolotow.

The image is particularly evocative in shifting to an exterior view as the window dissolves. I suspect there are a number of images that use this device in the training material for NB Pro.

When I made signed versions of this and several others that were generated in the same session, to be included in a folder for a "screen saver" slide show, I began thinking about the various numbers of different image types I've created in the past three-plus years. Last year I went through my (poorly organized) folder stack of "AutoArt" and reorganized it into 35 categories, each in its own folder. To date, there are 1,472 signed images in 35 folders containing between two and 405 images. My inner statistician began to stir…

The image below shows two analyses of the statistical distribution of the numbers of files in these folders.

Charts like these make it quite evident which statistical treatment is appropriate to a particular set of data. I'll explain what these charts mean and how they were created.

"Scale Free" is a type of power law distribution related to the Pareto distribution. It is easy to analyze, which makes it popular. To analyze a series of numbers graphically in Microsoft Excel:

• Enter the numbers in column B, starting in cell B2.
• Put an appropriate header in cell B1
• Highlight these data (B1:B36 in this case)
• Sort from largest to smallest, using the Sort & Filter section under Editing in the Ribbon.
• Enter 1 in cell A2 and 2 in A3.
• Put a header in cell A1; I usually put "N".
• Highlight cells A2 and A3.
• Double-click the fill handle at the lower right of A3. This will fill the rest of the column with numbers in order, as far as the data goes in column B. In this case, we get numbers from 1 to 35.
• Highlight these two columns to the end of data. In this case, from A1 to B36.
• In the Ribbon, use Insert and in the Charts section, select the icon showing scattered dots with axes; this is X-Y Chart.
• The title of the chart is whatever the header text is in B1. Edit as you wish.
• Double-click one of the axes to open the Format dialog.
• Click Logarithmic Scale near the bottom of the menu.
• Click the other axis and also click Logarithmic Scale. This is now a log-log chart.

The result will be similar to the upper chart. Now for the lognormal analysis, beginning with these two columns of numbers:

• Insert a new column between A and B; this is the new column B.
• In cell B1 enter a header such as "Prob.". You are going to create a probability axis.
• In cell B2 enter this formula (where the largest number in column A is 35):

=NORM.S.INV((A2-0.5)/35)

• Double-click the fill handle at the lower right of A2 to fill the column with the formula.
• Highlight the data in B and C (B1 to C36 in this case).
• Use Insert as before to create an X-Y Chart.
• Edit the chart title.
• Note that the vertical axis is now centered above the zero. 
• Assuming the Format dialog is still open, click the horizontal axis.
• In the middle of the menu in the section "Vertical Axis Crosses", click the bubble at "Axis Value".
• Enter "-3".
• Click the vertical axis and click Logarithmic Scale. This is now a log-probability chart.
• If you want the markers to be a different color, click one of them. The Format Data Series menu appears at the right.
• Select the icon of a paint bucket pouring paint.
• Click the Marker tab
• For both Fill and Border, select the color you want.

This will be similar to the lower chart. For the data I used, the chart shows the points scattered approximately along a straight line. By contrast, in the upper chart there is a definite downward bend. In a log-log chart such a shape is diagnostic that the distribution is not scale free, but is more likely to be lognormal, or even normal (Gaussian). In this case, the second chart shows that lognormal is a good model of the data distribution.

This is an illustration of the Theory of Breakage, formally described by A.N. Kolmogoroff in 1941. When an area is divided (US state or county areas are good examples), the distribution is lognormal. When a sheet of glass is broken, the weights of the pieces also have a lognormal distribution (I've done this experiment). Some recent publications claim that a theory of breakage produces a power law distribution, but this is false. Certain phenomena in nature tend to be normally distributed. The classic example is the height of adult men, or of women (but not both) in a population, such as the residents of a particular town or county. However, most phenomena produce groups of measurements that are lognormally distributed, in which the logarithm of the quantity being measured is distributed as a normal, or Gaussian, curve.

I could go further into this, but this is enough for the purpose of this post.

Studying tartan designs

 kw: analytical projects, plaids, tartans, statistical distributions, scale free, lognormal

Guess what this is? It isn't quite what it looks like. It's a printed plaid, a plaid-like pattern printed on white flannel, the backing for a comforter we made many years ago. Until I looked at it closely (microscopically), I thought it was a woven plaid.

Close inspection also reveals that the weave is single-over-under, rather than the over-2-under-2 of most plaid fabrics. Nonetheless, it is an attractive pattern, one of my favorites!

Some time ago I began to wonder about the distribution of stripe widths on plaids. Long ago I wrote, in GWBASIC, a "screen saver" program that produced plaid patterns on the screen. I used a scale free distribution because it is easy to program. It would generate a bunch of width values and then scramble them by sorting against a set of random numbers; it would assign colors and generate a plaid pattern.

I don't know how plaids are designed. The Scottish tartans such as Black Watch or Douglas can be centuries old, and were selected with aesthetics in mind, and an eye for being imposing because they were worn into battle. Today I suppose artistic designers pick the colors and stripe widths in a purely aesthetic way.

I decided to study the statistical distributions found in my own shirts and other fabrics. I figured out how to wrap a shirt around a dictionary to hold it on a scanner, and did so for 17 flannel shirts and two plaid jackets, plus the pattern above which I photographed because the comforter is large and very thick. I have a number of plaid summer shirts, which I may analyze in the future, but they are not included here.

The large variation in stripe widths led me to consider three model distributions: Normal, Lognormal and Scale Free or Log-Log. When graphed with appropriate coordinates, each of these is a straight line, but, for example, a Normal distribution will graph as a curved line on either Log-Log or Lognormal coordinates. First, we need to see the shapes of these distributions:

The Normal distribution is frequently called the Gaussian distribution, because it was first proposed by the mathematician Carl F. Gauss in the early 1800's. When several random variables are added and measured repeatedly, the distribution of the sum tends toward the center-weighted shape shown in orange. A mathematical proof of this additive tendency is called the Central Limit Theorem.

The Lognormal distribution results when an exponential function is taken for a set of values that have a Normal distribution. The Lognormal shape is shown in green. Also, when several random variables are multiplied and measured repeatedly, the distribution of the sum tends toward a Lognormal distribution. The logarithmic form of the Central Limit Theorem describes this tendency. Furthermore, when an area or extended object is fractured or divided into many pieces via a random process (such as dropping a pane of glass), the areas or weights of the pieces closely approximate a Lognormal distribution. I verified this once in the laboratory using a small piece of glass I broke with a light blow of a hammer, and then weighed a couple hundred pieces. The mathematical proof of this is called the Theory of Breakage, which was propounded by A.N. Kolmogoroff in 1941.

The Scale Free distribution results when a series of measurements are taken of the reciprocals of a uniform random distribution. This is also called a Fractal distribution, based on the work of Benoit Mandelbrot in the 1980's. A theoretical continuous Scale Free distribution has no limit in either direction; no largest or smallest member being predicted. Discrete sets of values that have a Scale Free distribution, however, do have a largest and smallest member. While the theoretical, continuous Normal and Lognormal distributions also have no limits, the probabilities of extreme values are vanishingly small (for a Lognormal distribution, "extreme" means either a very large positive value, or a value that is positive, but very, very close to zero).

Each distribution can be rectified (made to approximate a straight line) by sorting all the values and graphing them in order in an appropriate coordinate system. Idealized examples of these three distributions are all shown together in the three coordinate systems that are relevant to this discussion:

These charts each rectify one of the distributions. Firstly, for "Probability Coordinates", the horizontal axis has units of standard deviation and the vertical axis is linear. The sorted values in a Normal distribution (orange) follow a straight line here. Secondly, for "Log-Probability Coordinates", the horizontal axis is the same, while the vertical axis is the logarithm of the values, which straightens out the Lognormal distribution (green). Thirdly, for "Log-Log Coordinates", the horizontal axis is the logarithm of the ordinal number of the sorted values and the vertical axis is the logarithm of the values. This rectifies the Scale Free distribution (blue). Note that in each case, the "other two" distributions display a distinct curvature.

Now, for sets of more realistic distributions, created by appropriate random processes, we see the same three graphs:

The three coordinate systems are the same as those above. A straight line has been added to each graph to emphasize which set of values has been rectified.

How does all this apply to a study of plaids? I gathered data from the scans of the 20 plaids, measuring each one in both directions. This is because the warp and woof of the weave have different pitches, so the plaid designers adjust the number of threads of each color so the resulting plaid will not look distorted. Here is an example of a set of data for one of the plaids. I used rather generic color names, because the widths of the stripes were the meaningful parameter, not the color pattern.

Note that, while the order of the colors is the same in both directions, the number of threads is seldom the same in direction 2 as compared to direction 1. This enlargement of the pattern shows the threads; it takes a careful look to see that the spacing is different between horizontal and vertical. Look at the white square. It has 9 horizontal threads but 6 vertical threads, yet the "square" appears pretty close to a square.

One benefit of the over-2-under-2 weave is that it makes counting threads in wider bands easier, because I could count by 4.

This is a more overall view of the pattern. Although each "unit" of the pattern contains 5 white stripes, 4 black stripes, 2 navy stripes and only 1 gray stripe, gray dominates because its stripe is so wide, with navy blue running a close second.

What did I do with all these numbers? There are a lot of them. A few patterns had 38-40 stripes, and many had quantities in the 20's. Some plaids have mirror symmetry, a smaller number don't.

I copied all the data, sorted each set (each direction for each plaid), and set up both ordinal and probability axes for them all. I charted them in groups to see how they looked. I was looking for rectified distributions. As we see below, with a few of them as an example, the results are not clear-cut. I had been hoping to see a clear indication that the distributions were primarily either Scale Free or (my preference) Lognormal. The reality is a little of both. The graphs that follow pertain to six non-symmetrical patterns.

The overall view is that many of the lines have a downward curvature at the right, but not all. In particular, the yellow line and the gray line mostly hidden behind it (#16), and the lighter blue and lighter green lines in the midst of the scrum (#10), don't curve down.

The downward curvature indicates that most of these are better modeled as Lognormal. The next graph shows that presentation.

Here many of the lines appear straighter, while some either flatten out or curve oppositely (not really "upward"). We also see that the dark red line and the dark blue that accompanies it also flatten out, even though they have a bit of downward curvature in the other graph.

None of the patterns showed a hint of being closer to Normal than to Lognormal or Scale Free, so I didn't pursue that any further.

"Eyeballing" the charts proved unsatisfactory, so I used a mathematical measure of linearity, relevant to either Log-Log or Lognormal coordinates, to more clearly discern the trends.

I saw from this that some of the patterns were more Lognormal in one direction and more Scale Free in the other. I found the following:

• 7 patterns were Lognormal in both directions.
• 4 patterns were mixed, but leaned Lognormal more than Scale Free.
• 2 patterns were mixed, but leaned Scale Free.
• 7 patterns were Log-Log in both directions.

Here we have, from left to right, #3, which is the most Lognormal of them all, #8, which is the most ambiguous, and #6, which is the most Scale Free of them all.

As it happens, #3 and #8 are favorites of mine, and if the red plaid from our comforter were made into a shirt, as a pattern, it would also be a favorite (although my wife doesn't like me to wear red shirts); it is also a mixed-distribution pattern. I care less for #6; I consider it almost ugly. Just to show that Scale Free patterns are also attractive, another of my favorites is shown here, #10, which is more Scale Free in both directions:

A characteristic of Scale Free distributions is a greater number of narrower stripes, and this one shows that. It illustrates that what we like doesn't have a very strong mathematical basis. I had been thinking just the opposite, but I don't mind being proven wrong.

In the future I may scan my plaid summer shirts and analyze them, to see if these tendencies hold up. This has been an enlightening exercise.

More on Theory of Breakage

kw: analysis, lognormal, power law, statistical distributions

I wrote about the statistical distribution of stellar masses nearly four years ago. The Theory of Breakage, as it is classically derived, indicates that random processes which divide an object or area or substance into many parts will tend to produce "pieces" that have a lognormal distribution of the "size" parameter, whether length or volume or mass. Some scholars, following Benoit Mandelbrot, instead posit that random breakage will produce a distribution of "size" that obeys a power law. Here I apply it to a large collection of computer files.

As I close out a 44-year career in the computer sciences, I need to determine which computer files to pass on to my colleagues, and which can be allowed to vanish. I have had some kind of personal computer on my desk for 32 years, and I have certain files from my mainframe days going back another 12 years. As a long-time member of the Elephant Club (Motto: Do not trust a computer you can see over), I share the computer geek's vice of never throwing anything away. I long ago learned that magnetic disk technology was producing file space much faster than anyone could fill it. So I am a kind of hoarder. My house is not cluttered, but my disk drive certainly is!

I used a DOS command (how few folks even know how to do so!) to gather a complete file and folder listing for the Work disk. The entire corpus came to 35,309 files with an aggregate size of 18 Gbytes. Of this, the Project data consists of 8,297 files that total 8 Gbytes. This Project data are those files directly related to paid work. The rest is support materials and other files kept for historical reasons, such as a great many FORTRAN, Pascal and Perl program source code files that I call my "algorithm collection", presentation files in PowerPoint and older formats such as Framework (by Lotus), spreadsheets (Excel, 1-2-3 and Quattro), flow charts and other drawings in Visio and older tools, plus images, videos and sound files.

With a little bit of fiddling around, I produced two lists of file sizes, and analyzed them two ways. This first chart is a Power Law analysis:

Keep in mind that the "Proj. Files" are included in "All Files". The strongly curved shape of these distributions is diagnostic that they do not follow a Power Law, but are more likely to follow a Lognormal distribution. If you simply project the slope of the upper quarter or third, to the "1" line, you see that it would require trillions of files to extend the line (I estimate 20 trillion for the blue line). By contrast, here is a Lognormal analysis:

In this presentation, it is evident that the distribution is very nearly lognormal. I have no explanation for the few departures from linearity, and they don't really need "explaining", anyway. I simply find it fascinating that these thousands of items, some written, others downloaded or generated by many methods and processes over four decades, should result in such a distribution.

Visual analysis of star mass distribution

kw: analysis, statistical distributions, stars, lognormal, power law

Below I analyze the mass distribution of a couple thousand of the nearest stars. I must first explain what is going on here. It has to do with finding a way to chart a series of data so that they approximate a straight line. This is called linearizing the geodesic.

One of my nerdy obsessions is gathering examples that illustrate the Theory of Breakage. The premise was proven mathematically by Kolmogoroff in 1941 (published in German) and offered to the English-reading public in The Lognormal Distribution by J. Aitchison and J.A.C. Brown in 1957. If you drop a brittle object onto a hard surface so it shatters, then weigh every piece, an analysis would show that the weights of the pieces are distributed lognormally. The analysis of clastic sediments follows lognormal reasoning.

Rather than foist a mathematical proof on you here, I'll support the idea conceptually. The lognormal distribution is based on the normal, or Gaussian, distribution, the famous "bell curve." The relation is thus: If you take the logarithm of a set of items which are distributed lognormally, the resulting distribution will be normal. These six numbers make up a very small normal sequence: 7, 8.4, 9.5, 10.5, 11.6, 13. The mean (and median) value for this distribution is 10. If you add the first and last numbers you get 20, twice the median; so if you add any pair of numbers equally distant from the "gap" between the 9.5 and the 10.5, you get twice the median.

If we take an exponential function of these numbers, we get a lognormal distribution. In this case, let us take 2 to the power of each of the numbers, and divide the resulting series by 100. Then these six numbers form a very small lognormal sequence: 1.29, 3.4, 7.2, 14.5, 30.7, 81.3. As these are lognormally distributed (for they were constructed so), the median is a logmedian, formed by multiplying the extrema and taking the square root: SQRT(1.29*81.3) = 10.2. Again, this works with any pair of numbers equidistant from the "gap" between the 7.2 and the 14.5.

Where a normal distribution is characterized by values clustered about a median value, a lognormal distribution has a cluster on the small side of the median, and a scattering of the largest values, in other words, a heavy tail. In many cases, the largest member of a lognormal sequence is larger than the sum of the rest of the terms.

A bunch of random items that are related by a similar additive process tend to be normally distributed. The Central Limit Theorem, upon which Kolmogoroff's proof is based, states that repeatedly taking the sum of several random numbers produces a normal distribution. Exponentiation turns addition into multiplication, so by the same reasoning, repeatedly taking the product (or the quotient) of several random numbers produces a lognormal distribution. Thus, in nature, additive (and subtractive) processes produce normally distributed values, while multiplicative or divisive processes (exemplified by breakage) produce lognormally distributed values.

I got interested in this subject when I read (I don't recall where) of an alternative hypothesis of breakage, that the resulting pieces might constitute a "scale free" or "power law" distribution. A power law distribution somewhat resembles a lognormal distribution, in that there are many small members and few large ones. However, the quantity of small members is quite a bit greater. A power law sequence is most simply produced by dividing a series of numbers into a constant, but all practical sequences are produced by dividing some power of each member of the series into a constant.

A very small power law sequence, produced by dividing 30 by 6, 5, 4, 3, 2 & 1, is
5, 6, 7.5, 10, 15, 30. Using the generating law to determine the median, we find it is 30/3.5 = 8.57 (The procedure used for a lognormal sequence yields 8.66). The drawback to continuing this series is that it has no finite sum. A power law sequence from a distribution that will converge to a finite sum is found by dividing the 3/2 power of the first six numbers into 30: 2.04, 2.7, 3.8, 5.8, 10.6, 30. This has median 4.6.

While there are software routines that can distinguish whether a sequence is distributed according to a power law or lognormally (and a great many others), there is a simple visual test that I like to administer. The illustrations that follow have larger-sized versions "behind" them, available by clicking. I prepared these using Microsoft Excel, which unfortunately doesn't have a probability axis option for its charting tools, so I use a transformation to linearize a probability axis.

I illustrate the technique here; I generated six sample distributions of 25 members each. Three are power law sequences and three are lognormal sequences. The latter were scaled to have a largest member equal to 10, so we can see their shapes in this chart and the one that follows.

This chart presentation is log-log. On such a chart a power law sequence plots as a straight line. The three blue lines are power law sequences, and the three dark red lines are lognormal sequences with different breadths of distribution (different values of the logvariance). The strong curvature of the red lines indicates that they are far from linearized in this type of chart.

This chart takes a standard normal sequence as the vertical axis; the value "1" means one Standard Deviation from the mean (1σ or "one sigma"). 25 values have probabilities ranging from 0.04 to 0.96, which are from -1.75σ to +1.75σ. The horizontal axis plots the values of the members of the six distributions, and is shown in logarithmic transformation. As before, the blue lines are the power law sequences and the red lines are the lognormal sequences.

Now it is the latter that plot in a straight line on these axes, while the power law sequences plot as distinct curves. This is diagnostic for each type of data; they plot as a straight line on appropriate axes, and as a curve under any other transformation.

Now we can plot the stars' masses. I gathered stellar data from many sources, but I'd like to acknowledge the Nearby Star Observers for their work and for a great collection of links that allowed me to find some of the sources of data I used.

I gathered information on mass where I could, but for most of the stars I used databases that had the stars' spectral and luminosity types, distances and magnitudes, and applied stellar evolution theory to approximate the masses. I eventually had a list of the masses of 2,200 stars out to a distance of 100 light-years. I plotted these, the 688 stars closer than 50 light years, and the 51 stars closer than 25 light years, in log-log coordinates for this chart. Of the 2,200 stars, 1,876 are on the Main Sequence (Class V), and are distributed spectrally as follows:

• B - 2
• A - 38
• F - 50
  
• G - 379
• K - 578
• M - 829

These accord well with other analyses of the relative number of each type of star. The total list also includes some giants and close to 100 white dwarfs, plus a very few brown dwarfs. For the white dwarfs I did my best to infer the mass of the star when it was younger, because the aim here is to investigate the breakup of the gas-and-dust clouds that formed these stars.

The clear conclusion from this plot is that the data I have are not distributed according to a power law. There are two possible reasons: a great many K- and M-type stars may yet be discovered within 100 light years of Earth, or the actual distribution is not power law.

A lognormal analysis yields much straighter lines. Let us focus on the pinkish line. This set of 688 stars is probably very nearly complete.

A sphere twice the radius of another ought to have eight times as many stars within it, compared to the other. But 2,200/688 = 3.2, so many stars in the 50-100 light-year range are probably not discovered yet, and most of them will be small, dim stars (K and M) and brown dwarfs. On the other hand, 688/51 = 13.5, so there is a dearth of stars in the solar neighborhood. This is known from the literature about the "solar bubble".

What does this chart show us? It indicates to me that the distribution of star mass is most likely lognormal, and if so we can infer a few things. Projecting a straight line through the pink sequence to the 4σ lines indicates that in a complete sample of about 32,000 stars there ought to be a smattering of giants with up to 9 or 10 solar masses, and similar smattering of M9 types and brown dwarfs as small as 0.02 solar masses, about 20 Jupiter masses. Extrapolating wildly to 7σ (almost a trillion stars, close to the probable number in the Galaxy), puts us in a realm in which the heaviest star approaches 100 solar masses and the lightest brown dwarfs are no more than twice the mass of Jupiter. These conclusions seem plausible, so I have considerable confidence in a lognormal distribution of stellar mass.