A Programmer’s Guide to Data Mining – The Ancient Art of the Numerati

Intro to data mining & how to use this book

Imagine life in a small American town 150 years ago. Everyone knows one another. A crate of fabric arrives at the general store. The clerk notices that the pattern of a particular bolt would highly appeal to Mrs. Clancey because he knows that she likes bright floral patterns and makes a mental note to show it to her next time she comes to the store. Chow Winkler mentions to Mr. Wilson, the saloon keeper, that he is thinking of selling his spare Remington rifle. Mr. Wilson mentions that information to Bud Barclay, who he knows is looking for a quality rifle. Sheriff Valquez and his deputies know that Lee Pye is someone to keep an eye on as he likes to drink, has a short temper, and is strong. Life in a small town 100 years ago was all about connections.

People knew your likes and dislikes, your health, the state of your marriage. For better or worse, it was a personalized experience. And this highly personalized life in the community was true throughout most of the world.

Let’s jump ahead one hundred years to the 1960s. Personalized interactions are less likely but they are still present. A regular coming into a local bookstore might be greeted with “The new James Michener is in”– the clerk knowing that the regular loves James Michener books. Or the clerk might recommend to the regular The Conscience of a Conservative by Barry Goldwater, because the clerk knows the regular is a staunch conservative. A regular customer comes into a diner and the waitress says “The usual?”

Even today there are pockets of personalization. I go to my local coffee shop in Mesilla and the barista says “A venti latte with an extra shot?” knowing that is what I get every morning. I take my standard poodle to the groomers and the groomer doesn’t need to ask what style of clip I want. She knows I like the no frills sports clip with the German style ears.

But things have changed since the small towns of 100 years ago. Large grocery stores and big box stores replaced neighborhood grocers and other merchants At the start of this change choices were limited. Henry Ford once said “Any customer can have a car painted any color that he wants so long as it is black.” The record store carried a limited number of records; the bookstore carried a limited number of books. Want ice cream? The choices were vanilla, chocolate, and maybe strawberry. Want a washing machine? In 1950 you had two choices at the local Sears: the standard model for $55 or the deluxe for $95.

Welcome to the 21st century

In the 21st century those limited choices are a thing of the past. I want to buy some music? iTunes has some 11 million tracks to choose from. 11 million! They have sold 16 billion tracks as of October 2011. I need more choices? I can go to Spotify which has over 15 million songs.

• I want to buy a book? Amazon has over 2 million titles to chose from.
• I want to watch a video? There are plenty of choices:
• I want to buy a laptop? When I type in laptop into the Amazon search box I get 3,811 results
• I type in rice cooker and get over 1,000 possibilities.

In the near future there will be even more choice—billions of music tracks online—a wide variety of video—products that can be customized with 3D printing.