I reckon there are a few hundred books with subjects similar to the classic How to Lie With Statistics by Darrell Huff. They are really self-help aimed at helping us resist arguments made using flawed, or fraudulent, statistics. Now I find a book aimed at those who might use statistics to make an argument, to avoid fooling themselves: Standard Deviations: Flawed Assumptions, Tortured Data, and Other Ways to Lie With Statistics by Gary Smith.
As I began to read, I remember thinking, "He ought to title it Nonstandard Deviations", but I soon realized that proper statistical thinking is so rare, even among scientific writers, that the deviations the book presents are indeed standard practice. It is trouble enough that cynical marketers and politicos are using statistics fraudulently to deceive us; the larger problem is how many different ways proponents can lie to themselves!
The key chapter is #2: "Garbage In, Gospel Out". Although there are 16 more chapters exposing at least as many errors of statistical logic, and a great summary titled "When to Be Persuaded and When to Be Skeptical", those 16 chapters show all the common ways of using numbers to create nonsense. Several are based on faulty assumptions about trends.
We live in a world with two kinds of time. We are embedded in the cycles of the seasons: days, weeks, months, years, decades and centuries. Every day the sun rises, crosses the sky, and sets (unless you live in the high Arctic or on Antarctica). Every year the seasons come and go in sequence. Our most basic, gut-level experience of time is cyclic. But we also have linear time. Plant a tree and it grows taller every year. Some trees keep that up for a thousand years or more. We see continual population growth in most countries and in the whole world (Germany, France and a few other countries have reducing populations, but we don't think about that much). We have ancestors in the past, going all the way back to Noah or Adam or whatever progenitor we believe in; we also expect to have descendants going pretty much forever into the future, or at least "until Kingdom come".
We are less familiar with linear time, though, and tend to think linear trends can continue without limit. The key to unlocking this quandary is to realize that time itself is linear, but things that happen in time have a beginning and an end, and typically rise and fall in between. An evangelical "young-Earth" Christian believes in a strictly limited span of time, beginning about 6,000 years ago, maybe as much as 10,000 years, and ending within the next hundred or so. A purely agnostic scientist who knows cosmology believes time, or at least the current phase of phenomena in time, began 13.8 billion years ago, but there are a few hundred competing theories about when or whether it will end. Nonetheless, the end of life on Earth is pretty well understood to be a billion years from now, because the Sun is slowly heating up, and the end of the Earth itself will follow 3-4 billion years later, when the planet is crisped and perhaps evaporated by the Sun's red giant phase.
A few billion years is plenty of time enough for some trends to go along and go along for a long, long time. The human population of Earth has been steadily increasing for at least the last 50,000-70,000 years. The hope of many "zero population growth" advocates is that human population will stabilize within the coming 50-100 years, and even begin to shrink. However, if you want to start a business that requires population growth to continue, and you're satisfied with a run of 20-40 years, go for it. It'll take at least that long for growth to slow to the point you'd have a hard time keeping the business going. But the usual business cycle is about 6 years. Plan on some kind of downturn in the next few years. If you survive that into the next cycle, you just might keep that business going until your kids are grown.
The author exhorts us, again and again, to think. The motto of IBM used to be "THINK". Statistical reasoning doesn't come naturally, even for statisticians. He uses humorous stories of "experts" who ran afoul of their own wishful thinking. It takes a lot of data to prove a statistical inference. A key concept of statistics is "significance". Scientific journals are filled with articles that employ statistical tests and declare that some finding is "significant to the x% level". That "x%" is typically 95%, which is frequently stated as 0.95. That means that there is at least a 95% chance that the "significant" finding is true. But there's a 5% chance that it is not true.
Let's suppose that every scientific experiment resulted in a publication telling the results. Further, let's suppose that only one in ten reported "significant" results. Think a minute: why do scientists use statistics? It is because they don't get a clear-cut result. If using widget A was always lots better than using widget B, statistics would not be needed. The article could be very short: "In 100 trials, widget A always did a better job than widget B". Then you'd question whether the scientist were sane: after about 10 trials, you can stop already! That depends on just how much A was better.
More typically, there is overlap. Suppose that some scoring method showed that A is better 64% of the time. If that was 64 out of 100, it is probably a significant result, but if it was 16 out of 25, you could be in trouble with the law of small numbers. This is analogous to flipping a coin 25 times to see if it is a fair coin. You get 16 heads. How likely is that? Many people think there ought to be a nearly exact even split, either 12 or 13 heads. Here is how to analyze it:
- For 25 coin flips, there are 33,554,432 possible outcomes, from all heads to all tails, but in 33,554,430 out of 33,554,432 cases, it'll be some mix.
- An outcome of 12 heads occurs 5,200,300 different ways, as does an outcome of 13 heads. Together they total 30.1% of all outcomes. That is, intuition is correct less than 1/3 of the time!
- An outcome of exactly 16 heads occurs 2,042,875 different ways. Thus, the chance you'll get 16 heads is 6.1%.
- There is thus a 6.1% probability that this outcome indicates there is no difference between the two widgets. The result is not sufficiently "significant".
This analysis was done using Pascal's Triangle, and there is plenty of software out there that can do such an analysis. You just have to know enough to set it up. By the way, if this were the result of 50 trials, with 32 heads, you'd have a different conclusion. Firstly, getting exactly 32 heads in 50 throws occurs 1.6% of the time. You could also say that getting at least 64% occurs 3.2% of the time by chance alone. Thus, the "significance level" is 96.8%, which is better than 95%, so there is support to say that widget A is actually better than widget B.
This is not a lock. Remember, I posited a world in which every result is published, whether favorable or unfavorable to the initial conjecture. Do you think negative results are published? Nearly never!! So in a world of "publish everything", if 1/10th report "significant" results, some of those are likely to be due to chance alone. Perhaps one in 20, or 2 of the original 100 articles. But in the real world, the proportion may be quite a bit higher. It is certain to be at least 1 in 20.
OK, that's a long-winded excursion into just one item that struck my fancy. As in most endeavors, there is a very short list of ways to do it right, and a near-infinite number of ways to go wrong. That's why we need to expose our ideas to a great variety of folks with different backgrounds and viewpoints. Many times, though, the proponent(s) of an idea will circulate only among those who think alike.
It is also shown that wanting a certain result is the most powerful enemy of truth. I recall an old story of someone seeking a simple answer, because he didn't know how to figure it for himself. He got a variety of answers from people he knew, until he asked a political lobbyist, who responded, "What do you want it to be?" Well, that joke may be more political than statistical, but it is sobering. No matter how much we may want this or that to be true, the actual case is the actual case, the truth is the truth, and will outlive you and your most heartfelt desire.
No comments:
Post a Comment