We can't lie to Google

kw: book reviews, nonfiction, data mining, data analysis, big data

There is a very interesting and useful tool available from Google, Google Trends. It provides a taste of Big Data (or Data Mining) to all of us. It tracks search terms (all of them) that have been entered in the Google search box for the last 14 years. It charts their relative popularity. Here is an example:

This is just the top of a page of charts related to these searches. There are breakouts by region and certain subtopics or related searches. Just from this chart we can see three things:

1. While on average the Grand Canyon is the most popular of the three, the relative popularity of these three National Parks varies over time (this chart shows 5 years).
2. All three parks' popularity varies with the season. I presume that in springtime and early summer, people are planning vacations.
3. Each park had a spike of interest in a narrow time frame. I'll leave it to you to figure out which event triggered which spike. (Hint: high wire; seismicity; sesquicentennial)

Here is a clue to the value of meta-analysis made available by a tool like this: people search for what interest or concerns them, and there is nothing to be gained by lying about it. They really want to know. Thus, a result like this is more telling:

Here, seasonality of weight loss searches is quite evident. My impulsive analysis: a big spike in January is related to this most common New Year's Resolution, followed by sustained interest until mid-summer, then a slacking off as people enter first vacation season and then the holiday season, and have other things on their minds. Also, it is clear that many more people are worried about their weight than about smoking (and I may not have used the most common search term about this; GTrends is very literal and there are lots of ways people ask questions. But I could have searched for Topics instead).

Just out of curiosity I removed the weight loss portion. This amplified the remaining "stop smoking" result, and it is had no seasonality.

The little arrow at center-right in the diagrams shows that you can download the data (I haven't tried it) for analysis using Excel. This allows you to compare a great many search terms or topics, to winkle out all the variations, and sum them up for a general picture, or parse them by country or region or whatever. Note that if you mix topics and search terms, they are measured on different scales, because a topic gathers many search terms.

I'll refrain from going on and on, for this is a book review, after all. Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are, by Seth Stephens-Davidowitz, introduces the new things that are being done because we have enormous data sets available, and even the more because certain of them (such as Google searches) give more truthful information than "traditional" methods of social study such as surveys.

Have you ever lied to a survey taker, or on an online survey? Many folks do. Voter polls are notorious for inaccuracies, particularly during hotly-contested elections, such as the recent circus featuring Mrs. Clinton and Mr. Trump. Nearly every poll was dramatically wrong, nearly every time. Why? This is more than just bad poll question design (which is all too frequently done on purpose), this is a case that many of those who were going to vote one way either said, "I don't know for sure" or flat-out lied to the pollster. Consider someone who answers the telephone at work, and it is a poll taker. They aren't supposed to call workplaces, but they frequently do anyway (where else you gonna find people at midday? And with so many folks declining to even answer a phone at dinner time...). The office worker agrees to a "quick poll", and soon enough the big question comes, "Who do you prefer in the Presidential election?" This person is a closet Republican in New York City or San Francisco, and knows she'll be overheard by co-workers, with whom she never, ever talks politics. How do you expect her to answer? In those cities, you can lose your job over your Presidential preference.

The author of Everybody Lies shows a number of ways that he and others gather data from Google searches (he calls them his favorite data set), from Facebook likes, and a from number of other data collections about our behavior online. He also places emphasis on what he calls New Data, data we didn't have available before. Online behavior traces are one kind of New Data, but the concept is not new. For example, in 1854 Doctor John Snow mapped every death from cholera for the year in the Soho area of London, hundreds of them. That counts as Big Data for the 1850's! His dot map, shown here, helped him pinpoint a specific water pump as the source of the cholera epidemic; it was a pump on Broad Street, just above and right of center in this map (the dark bar there shows nearly 20 deaths in a single multi-family house). This was both Big Data and New Data for the time.

Getting back to November 2017: People who didn't get nearly as much press as the daily pollsters, but who were looking at "sentiment analysis" of internet searches, were saying, "Not so fast..." but were generally ignored. But it was they who were right. That is because, while we can lie to Google,
we don't because what is the point? We search for what we really want to know. And if someone wanted to mess with Google's search results database, and skew what it reports, it would take a huge effort, even by spambot standards. Even bot-masters have better things to do with their time!

There are some arenas in which we do lie, even to our computers. Facebook is an example. We post mostly positive stuff. We also primarily like to see positive stuff. In fact, when we see too much negative material in our newsfeeds, many of us either block or unfriend the purveyor thereof. So we tell our Facebook "friends" about what is going well with us, and we ask Google about what bothers us.

A number of fascinating and unexpected results show up. For example, would you expect a larger proportion of sports stars to come from comfortable, middle-class backgrounds, or from poor or working-poor backgrounds? Many people think that desperation to escape the ghetto drives talented kids to excel so they can actually get out. Analysis of the history of every professional sports figure counters that expectation. Most successful professionals in baseball, basketball, etc. come from middle-class backgrounds. One major factor: There are a whole lot more middle-class people than poor people in America. But even in proportion, the percentage of middle-class youngsters who excel in sports is greater than the percentage of poorer kids. We could go into a lot of possible reasons why, but it's a side point here. Truly great players like LeBron James, who was raised by a poor single mother, are the exception.

Late in the book the ethics of data mining arise. Many people are a little creeped out to learn that a trend analysis can reveal so much about them. Meanwhile, CEO's of retail companies are drooling over the same data, trying to figure out how best to induce us to part with our cash. One aspect is A/B testing. It isn't hard for the programmers at Google or Facebook to write code that, for example, puts a bluer "click here" button on an ad sent to everyone from a certain set of web servers, and a purpler button to everyone else. Which one gets a greater proportion of clicks? Do the words "click here" or "try it" or "join up" work best? When billions of ads are being shown, a 1% advantage comes to thousands or millions of potential sales. Ethical or not, that cat is out of the bag. There is no putting it back. We simply need to stiffen our own backbone of ad resistance if we wish to avoid being manipulated.

A fun side note. I am learning to use Incognito Mode and Duck Duck Go searching when I want to research something to buy. Otherwise Google and Facebook and just about everyone else spends the next month or three sending me ads related to something I already bought (or decided not to buy). When they get better connected to the retailers, so they know I bought it and probably won't buy another, and when they get better connected to Amazon's database of "people who bought this also bought these", it is likely that my ad resistance will become a lot harder to maintain! They'll have a pretty good idea what I want next, before I know I want it.

That is the world we live in. Teach your children well. Maybe they'll become more canny and cautious Web users than we have been.

Paranoia is more justified than ever

kw: book reviews, nonfiction, technology, privacy, data mining, cyber crime, surveillance

Whom do you fear the most? The criminals or the police? For most people it will depend on where they live. For me, living on the liberal coastlands of America, criminals are the bigger threat. However, that may not always be true.

George Orwell's 1984, which he thought was set comfortably some 36 years in his future, is now set 30 years into our past, and will be set 36 years into the past in 2020. By then its forecasts may prove all too true. The newest models of "smart televisions" and some set-top cable boxes have cameras and microphones. Many owners never turn on the devices, and most forget they are there. But these are only the tip of the iceberg of the surveillance society in which we live.

Prior to about 40 years ago, the most skilled safecracker or most wily train robber could snatch a payroll in cash. A few skilled embezzlers working at banks used "bologna slicing" methods to skim a few cents here and there from thousands of bank accounts. But as recent data-theft cases from companies such as Target show, the ubiquitous Network ("The Net", formerly called "the Internet") facilitates much broader-scale thievery, of the identities and buying (or other) habits of tens of millions at a time.

Suppose that you could be assured you'd never lose a penny to fraud or theft of any kind, yet your every move would be known and your every word recorded, even in your most intimate moments? Growing numbers of people are paying $100 or more yearly for "ID theft protection" services such as LifeLock. What would you pay to be sure neither Big Brother nor the neighborhood felon, nor even your brother-in-law, could ever see into your toilet, bedroom, or clothes closet?

It's funny. Privacy in America is a big issue in some areas, such as Abortion ("privacy" is THE core of the Roe vs Wade decision), yet is no issue at all in others, such as what you eat, whom you love, what your doctor says, or where you go on vacation. The countries of Europe have strong constitutional or legislative protections for privacy, usually giving you ownership of everything that can be recorded about you. Not so in the good ol' USA.

Privacy! Do you forget the smart phone in your pocket, with its camera and microphone that can be controlled remotely? The cameras, not only the ones in the store with all those little black hemispheres in the ceiling (or simply sitting in the open), nor the ones at red-light-violation intersections, but in places you might never think to look. All are recording 24/7, attached to near-infinite data storage devices. When was the last time you saw a news report about some crime or disaster somewhere that didn't show a few seconds of surveillance footage? And just the other day on a walk my wife and I saw one of our neighbors learning how to use a quad copter in the park down the street. Today, he is satisfied to zoom it around and do acrobatics. Does it have a camera? Probably. How long will it be before I hear it humming overhead as I work in the yard? or outside my window?

A note to the handy: a search for "homemade herf" yielded 100,000 hits just now. Might be worth a look. The device will be the size of a trash can, but can zap a drone out of the sky. Careful, though, a side lobe can zap your PC in the house and the phone in your pocket.

All this is leading up to a very handy reference book: Privacy in the Age of Big Data: Recognizing Threats, Defending Your Rights, and Protecting Your Family by Theresa M. Payton and Theodore Claypoole. Their premise is simple: technology allows ever more intrusion into our private lives, and we are doing it to ourselves. Particularly in the wake of the 9/11 attacks, now 13 years ago, at least half of Americans feel trading away some of their rights is worth getting increased protection. The trouble is, we are getting a pig in a poke. We don't know if we are getting what we pay for, and the pig is accompanied by a wolf: what the government can see can also be seen by criminals.

I was present at some of the meetings in 1980 between Control Data Corp. and data managers for the Mormon Church, who were planning to install a 1TB data store for genealogy records. Its projected cost was $1.2 million. Considering that a big disk in those days was 500 MB, and the drive cabinet was half the size of a washing machine, the building to house the 2,000 disk drives and their A/C equipment was going to cost another quarter million. I recently got a backup disk, 2 TB for $99. The current million-dollar system would be around a tenth of an exabyte, or 100 PB (petabytes). An exabyte is a million TB.

Do you know just how big a Terabyte is? One minute of good quality MP3 audio produces about a 1MB file. 1,000 minutes (16.7 hours) takes up 1GB, and 16,700 hours of audio would fill 1TB. But lower quality audio, such as a phone call recording, can be 1/3 the size, or smaller, so a 1TB disk could hold more than 50,000 hours of telephone conversation, or 3 million minutes. That is 5.7 years of non-stop talk. Even a dedicated motormouth would be hard put to use the phone more than 2,000 minutes per month, so the disk could hold 125 years of phone use for that one person. On the other hand, uncompressed text is quite compact, and compressed text is even smaller. If you type at a "standard" rate of 40 wpm, that's 14.4 KB per hour or about 1/70 of a MB. 1TB is a million MB. That's why it is so easy to copy and store billions of e-mail messages.

In 1980, at one of the meetings mentioned above, someone from CDC joked, "If we invent a disk with infinite capacity—call it the God Disk—the Government will buy two of them." One of the Mormons deadpanned, "So would we." But a big disk is nothing without software to sort and collate it. Now that any $500 laptop can sift through huge amounts of data in a few seconds, all kinds of records become useful.

"All kinds" is amazingly comprehensive to Payton and Claypoole. Nearly everything we buy, from groceries to clothes to knickknacks to more personal items, is recorded, usually along with our identity. To keep from having a purchase tagged to you, you'd need to pay cash at the point of sale, and be among the lucky ones to buy from a store that doesn't have face recognition software running on its security cameras. Retailers love data about buying habits. So do criminals; they want to know who just bought anything expensive and comparatively portable. Government usually doesn't care, except for certain kinds of purchases. I suspect an order for 500 decks of cards, not from a casino, might get scrutiny for two reasons. Firstly, the possibility of an underground high-stakes card game. Secondly, the plastic in certain card brands can be chemically altered into a cheap plastic explosive.

How soon will it be that the store's computer system knows who you are, and your tastes, by the time you walk in the door? Maybe it knows who you are from your phone, or maybe from cameras. The rest is online and they're already tied in to it. Think you can fool the cameras with a wig or hat and dark glasses? They way you walk and other bodily motions you habitually make are harder to disguise (a one-time fakeout can be had by putting some duct tape across your back or a pebble in your shoe).

Planning to travel abroad? The authors advise, most particularly for repressive countries, that you leave your usual phone at home and take a cheap throwaway phone. Also, make no purchases using the phone! Some global companies have their expat executives take "vacations" to safer climes when they need to arrange sensitive deals or discuss important matters with HQ. If you need to carry your phone, keep it in a metal case, and turned off, when not in use, along with those credit cards and their convenient microchip. An easy alternative is an aluminized Mylar balloon, deflated and just wrapped around your electronics. Then the phone can't communicate with cell towers and the chips can't be read by someone walking by using a RFID reader. In one chilling section, though, the authors report on gadgets to read brain waves from a modest distance, say a foot or so. Will I need a foil-lined hat? Better yet would be a phony brain wave generator in my hat, a kind of jammer.

How do we use social media? I never tell my FB "friends" where I am going, only where I have been…usually! Many online retailers want to use us for free advertisements, and urge us to "share" on FB or somewhere. Some have six or more "share" options! Even if you don't share, you'll start seeing more ads for the thing you just bought, or only researched. If you do share, those ads will multiply. That's your "payment" for your "help".

Things will only progress. There is no going back. The authors close with a chapter on legislation. Here we learn that America, where "privacy" was so important to winning the abortion decision, does the least to protect privacy among so-called Democratic nations. Levels of black-market activity are lower in America than nearly anywhere else, which indicates more trust in government here. But I am not worried so much about the government of today in America. I am worried about future governments here and around the world, with a more and more nannyish attitude, or worse yet, with Sharia laws and the technology to enforce them in absolutely every case.

The trusting American people are embracing cars that know more and more about them, and report on them to the car dealer and insurance company; electronic medical records with poor security; smarter and smarter pocket computers that have phones attached, along with multiple cameras and microphones and geolocators and accelerometers and whatever is next; and cameras, cameras everywhere. The members of the freest society ever to arise are forging their own chains.