The Stats are out to get you

kw: book reviews, nonfiction, statistics, logical fallacies

I reckon there are a few hundred books with subjects similar to the classic How to Lie With Statistics by Darrell Huff. They are really self-help aimed at helping us resist arguments made using flawed, or fraudulent, statistics. Now I find a book aimed at those who might use statistics to make an argument, to avoid fooling themselves: Standard Deviations: Flawed Assumptions, Tortured Data, and Other Ways to Lie With Statistics by Gary Smith.

As I began to read, I remember thinking, "He ought to title it Nonstandard Deviations", but I soon realized that proper statistical thinking is so rare, even among scientific writers, that the deviations the book presents are indeed standard practice. It is trouble enough that cynical marketers and politicos are using statistics fraudulently to deceive us; the larger problem is how many different ways proponents can lie to themselves!

The key chapter is #2: "Garbage In, Gospel Out". Although there are 16 more chapters exposing at least as many errors of statistical logic, and a great summary titled "When to Be Persuaded and When to Be Skeptical", those 16 chapters show all the common ways of using numbers to create nonsense. Several are based on faulty assumptions about trends.

We live in a world with two kinds of time. We are embedded in the cycles of the seasons: days, weeks, months, years, decades and centuries. Every day the sun rises, crosses the sky, and sets (unless you live in the high Arctic or on Antarctica). Every year the seasons come and go in sequence. Our most basic, gut-level experience of time is cyclic. But we also have linear time. Plant a tree and it grows taller every year. Some trees keep that up for a thousand years or more. We see continual population growth in most countries and in the whole world (Germany, France and a few other countries have reducing populations, but we don't think about that much). We have ancestors in the past, going all the way back to Noah or Adam or whatever progenitor we believe in; we also expect to have descendants going pretty much forever into the future, or at least "until Kingdom come".

We are less familiar with linear time, though, and tend to think linear trends can continue without limit. The key to unlocking this quandary is to realize that time itself is linear, but things that happen in time have a beginning and an end, and typically rise and fall in between. An evangelical "young-Earth" Christian believes in a strictly limited span of time, beginning about 6,000 years ago, maybe as much as 10,000 years, and ending within the next hundred or so. A purely agnostic scientist who knows cosmology believes time, or at least the current phase of phenomena in time, began 13.8 billion years ago, but there are a few hundred competing theories about when or whether it will end. Nonetheless, the end of life on Earth is pretty well understood to be a billion years from now, because the Sun is slowly heating up, and the end of the Earth itself will follow 3-4 billion years later, when the planet is crisped and perhaps evaporated by the Sun's red giant phase.

A few billion years is plenty of time enough for some trends to go along and go along for a long, long time. The human population of Earth has been steadily increasing for at least the last 50,000-70,000 years. The hope of many "zero population growth" advocates is that human population will stabilize within the coming 50-100 years, and even begin to shrink. However, if you want to start a business that requires population growth to continue, and you're satisfied with a run of 20-40 years, go for it. It'll take at least that long for growth to slow to the point you'd have a hard time keeping the business going. But the usual business cycle is about 6 years. Plan on some kind of downturn in the next few years. If you survive that into the next cycle, you just might keep that business going until your kids are grown.

The author exhorts us, again and again, to think. The motto of IBM used to be "THINK". Statistical reasoning doesn't come naturally, even for statisticians. He uses humorous stories of "experts" who ran afoul of their own wishful thinking. It takes a lot of data to prove a statistical inference. A key concept of statistics is "significance". Scientific journals are filled with articles that employ statistical tests and declare that some finding is "significant to the x% level". That "x%" is typically 95%, which is frequently stated as 0.95. That means that there is at least a 95% chance that the "significant" finding is true. But there's a 5% chance that it is not true.

Let's suppose that every scientific experiment resulted in a publication telling the results. Further, let's suppose that only one in ten reported "significant" results. Think a minute: why do scientists use statistics? It is because they don't get a clear-cut result. If using widget A was always lots better than using widget B, statistics would not be needed. The article could be very short: "In 100 trials, widget A always did a better job than widget B". Then you'd question whether the scientist were sane: after about 10 trials, you can stop already! That depends on just how much A was better.

More typically, there is overlap. Suppose that some scoring method showed that A is better 64% of the time. If that was 64 out of 100, it is probably a significant result, but if it was 16 out of 25, you could be in trouble with the law of small numbers. This is analogous to flipping a coin 25 times to see if it is a fair coin. You get 16 heads. How likely is that? Many people think there ought to be a nearly exact even split, either 12 or 13 heads. Here is how to analyze it:

• For 25 coin flips, there are 33,554,432 possible outcomes, from all heads to all tails, but in 33,554,430 out of 33,554,432 cases, it'll be some mix. 
• An outcome of 12 heads occurs 5,200,300 different ways, as does an outcome of 13 heads. Together they total 30.1% of all outcomes. That is, intuition is correct less than 1/3 of the time!
• An outcome of exactly 16 heads occurs 2,042,875 different ways. Thus, the chance you'll get 16 heads is 6.1%. 
• There is thus a 6.1% probability that this outcome indicates there is no difference between the two widgets. The result is not sufficiently "significant".

This analysis was done using Pascal's Triangle, and there is plenty of software out there that can do such an analysis. You just have to know enough to set it up. By the way, if this were the result of 50 trials, with 32 heads, you'd have a different conclusion. Firstly, getting exactly 32 heads in 50 throws occurs 1.6% of the time. You could also say that getting at least 64% occurs 3.2% of the time by chance alone. Thus, the "significance level" is 96.8%, which is better than 95%, so there is support to say that widget A is actually better than widget B.

This is not a lock. Remember, I posited a world in which every result is published, whether favorable or unfavorable to the initial conjecture. Do you think negative results are published? Nearly never!! So in a world of "publish everything", if 1/10th report "significant" results, some of those are likely to be due to chance alone. Perhaps one in 20, or 2 of the original 100 articles. But in the real world, the proportion may be quite a bit higher. It is certain to be at least 1 in 20.

OK, that's a long-winded excursion into just one item that struck my fancy. As in most endeavors, there is a very short list of ways to do it right, and a near-infinite number of ways to go wrong. That's why we need to expose our ideas to a great variety of folks with different backgrounds and viewpoints. Many times, though, the proponent(s) of an idea will circulate only among those who think alike.

It is also shown that wanting a certain result is the most powerful enemy of truth. I recall an old story of someone seeking a simple answer, because he didn't know how to figure it for himself. He got a variety of answers from people he knew, until he asked a political lobbyist, who responded, "What do you want it to be?" Well, that joke may be more political than statistical, but it is sobering. No matter how much we may want this or that to be true, the actual case is the actual case, the truth is the truth, and will outlive you and your most heartfelt desire.

The illusion of thinking clearly

kw: book reviews, nonfiction, thinking, logical fallacies

A few years ago I met someone at a reception. He told me he was a philosopher, and that his specialty was the fallacies of formal logic. I happen to know that there are 16 formal fallacies, which are errors in the logic of an argument. I also know, and had recently read a treatise upon, the informal fallacies, which are unknown in number, but there are more than 100 (the Wikipedia article "list of fallacies" notes 59). For some reason, this quite incensed my new acquaintance, to the point that I was concerned he may become violent (a common informal fallacy on his part!).

Winston Churchill said, "Even a fool is right once in a while." This caution alerts us to avoid the most common informal fallacy, the ad hominem attack, which we could describe as, "You must be wrong because you are a bad person", or, "…because I don't like you", or, "…because you are a [substitute your stereotype of choice]". Interestingly, this fallacy is not discussed in The Art of Thinking Clearly by Rolf Dobelli. I suppose he felt is has been sufficiently treated elsewhere, a great many elsewheres. But he does discuss 99 common errors that are so common, so very common it is a surprise any of us can decide anything at all!

Dobelli is Swiss and writes in German; the book was translated to English by Nicky Griffin. Kudos to the translator. Many translations from German produce nearly unreadable English. Clearly, Dobelli has a smooth, conversational writing style which Griffin has captured masterfully. It is great fun to read.

I can't hope to comment on more than a few of the items discussed. I picked a few favorites:

• Reciprocity, in the chapter "Don't Accept Free Drinks" – Dobelli calls appeals for donations that come in the mail with a "free gift" inside a "kind of gentle blackmail" (I'd call them extortion rather than blackmail). An allied principle is TANSTAAFL: There Ain't No Such Thing As A Free Lunch. Depending on how much spending authority you have, the "free lunch" could range from sports tickets to a "free" vacation. Then there is all the "free" stuff offered if you'll spend 90 minutes listening to a timeshare presentation. It is good to learn to say, "I don't think I can afford your 'free gift'."
• Confirmation Bias, in two chapters, the second being "Murder Your Darlings" – This has two sides. One is the increasing tendency for search engines, led by Google, to keep track of your preferences and to use them to rank the list of returns from a search you make. As time goes by, you'll only get "hits" that confirm your prejudices. That's why it is a good idea not to search when you are logged in to a Google service such as Blogger, Drive or GMail (or one such as Yahoo Mail if you search using Yahoo). Search as anonymously as possible if you want less biased results. The second side is the tendency of writers to dwell on themes that they love and to give short shrift to others, even if they are trying to discuss "all sides" of an issue. Arthur Quiller-Couch devised the motto, "Murder your darlings", meaning to eliminate the redundant text that inevitably fills your writing about those most-loved themes; pare those sections down to match the less-favored sections.
• Induction, in the chapter "How to Relieve People of Their Millions" – This is a favorite of mine, based on an ancient scam. Someone who is lucky several times in a row may be considered extra favored or blessed, and if you get tricked by your own good luck, it can lead to a feeling of immortality. It also leads to unneeded depression when your luck turns. But it also explains why "financial advisers" invest each client's funds in a different collection of investments (Those who invest all funds equally are called mutual fund managers, and are more likely to have a modicum of honesty!). Here is a key datum: multiply 2 by itself 10 times, and the result is 1,024. Pick a yes/no question, such as "will the market go up or go down?" Send about 1,000 people an e-mail in which you explain why you think the market will go up in the coming week, and send another 1,000 an e-mail in which you explain why you think it will go down. After a week, it has done one or the other. Suppose it went down. Now send an e-mail to just the second group making a new prediction, again of the yes/no variety. The third week, e-mail just the 500 for whom you've been right twice, with a further prediction, and so forth. After five weeks, assuming you actually started with two groups of 1,024, you now have 64 people who have received five accurate predictions. At this point, ask to be paid for further predictions. Let's say they all agree (keep the cost low at first). Now things get complicated. After your next prediction, send an apologetic e-mail to the 32 who saw you "flub", and offer to refund their payment; send a self-congratulatory e-mail to the others, but don't lay it on too thick. You are likely to keep some of the ones who got the apology. Anyway, after a total of 10 predictions, you now have at least 2 people who think you are infallible! You can ask for stratospheric prices for your answers. Investment advisers are not so blatant about it, but by spreading around their clients' funds, they can avoid being wrong too frequently.
• The Black Swan, in the chapter "How to Profit From the Implausible" – The actual fallacy is to think that unlikely events are less likely than they really are. For example, how likely is it that someone could throw a basketball over their house, and have it go through the hoop in the back yard? One in a million, or a billion? Yet there are at least two videos out there showing just this happening. One is shown a couple times a year on America's Funniest Home Videos on ABC. Professional statisticians tend to analyze every distribution as a "normal" distribution, even though very few natural phenomena follow a bell curve. For example, women's height is found to be normally distributed, but household income is not. Also, the daily change in a stock's price is typically analyzed as a normal distribution, but large changes are much, much more likely than such a model predicts. Dobelli claims that unlikely events are getting even more likely, and even more consequential, because our civilization is more strongly affected by events outside "the usual range", like a 100-year flood. Before we began building lots of fragile houses some 10,000 years ago, a 100-year flood simply meant moving camp to higher ground for a week. Now it means high insurance premiums (if you have flood insurance in the first place). Also, those who make big incomes don't work for others. Dobelli's advice is to work in an area where a big break can bring big returns, but to save and invest as though such a big break may never come. If it comes, you can profit from it, and if it doesn't, you will have provided for your future.
• Feature-Positive Effect, in the chapter "Why Checklists Deceive You" – We notice things that are there (but not even all of those!), but it is very hard to notice what isn't there. This is the crux of the Sherlock Holmes story "Silver Blaze", where the important clue was that a watchdog didn't bark. Only Holmes would notice such a fact. Everyone else was busy about evidence they were able to collect, because it existed. Double-talk "explanations" about why something went wrong are solidly based on carefully omitting key facts in the midst of a blizzard of less relevant, but attractive, facts (a related fallacy). Dobelli tells of Beethoven's Ninth Symphony. During its premier many wept. He asks, "Would we be less happy without [it]? Probably not." Had "the Ninth" never been written, we'd never know what we were missing anyway. In the same way, we notice nothing in particular when we are totally well. We really notice any disease or injury.

In a reference section at the end we find 50 pages of bibliographic information, which the author says could easily have become several hundred pages. There is a lot of "thinking about thinking" being written up in the literature. This book is the most accessible of them all in my experience.

Backing up to (electoral) glory

kw: opinion, politics, logical fallacies

The old fallacy of guilt by association has taken a new twist. You know, "So and so has evil friends, so he must be evil also." The argument typically has force only if there are several of these evil associates, particularly if they outnumber the "non-evil" ones.

Former Congressman Foley is no doubt the evil friend to all Republicans these days. However, rather than a few or several evil friend implicating one person, one person is implicating an entire body of about three hundred members. Let's see, is the part-for-the-whole metaphor a metonymy? I forget, but it takes the old fallacy to new levels!

Members of both parties need to recall that Jesus said, "It has been said of old that the fathers have eaten sour grapes, and the children's teeth are set on edge. But I say to you that each shall die for his own sin." It is entirely predictable that desperate members of both parties will get their arguments exactly backwards, flailing for a few votes. Pity.