This is an actual pic I took from the deck at our vacation rental in Ashtabula, Ohio this weekend. No Photoshop! Just fortuitous timing and cloud cover.
There’s no real profound content here today, but the pic is too good not to share. Enjoy!
Or, Data Normalization Using Indices for Fun and Profit
One time a big outdoor sports retailer asked me to analyze their online and catalog sales data in one region of the country, and then use that analysis to try to estimate sales in future brick-and-mortar locations in the same region.
So I did the analysis. Based on my results, I confidently figured they’d probably sell more stuff in Atlanta than they would in Siler City, North Carolina. Wow, what a blinding glimpse of the obvious. The right thing to do would be to compare data between the two places on more of an even footing. There are many ways to do that, but one of my favorites is to use indices. Indexing is meaningful, easy to understand, easy to calculate, and fun. So naturally, I charge a fortune for it (just kidding).
To demonstrate, I took two of my favorite things, 1) anything to do with Ohio and 2) cheeseburgers. Then I calculated indices on based on some real retail sales data from last year.
Fact 1: People in Ohio Love Cheeseburgers. Or Big Macs, or Frosties, or Whoppers. Ohioans consumed 68% more of these types of fast foods, on average, than US consumers in general.
Fact 2: People in Warren have a higher Munch-a-Burger Index than do people in Youngstown. Folks in both cities spend more than the US average on this food, but less than others in Ohio (note the Bonus Facts below).
Fact 3: 26% of people nationally spent some amount > $0 at one of these restaurants in the past 6 months. But that’s true of 34% of Ohioans! It’s 30% in Youngstown, and 29% in Warren. Now there’s an interesting little data point. About the same percentage of the population frequent one of those restaurants between Youngstown and Warren, yet those hungry Warrenites spend oh so much more: $102.28 in the past 6 months versus $74.31 among Youngstowners.
Bonus Fact: Among largish Ohio cities, here are the indices of the Top 5 towns at these restaurants relative to others in the state:
5. West Chester 149
4. Hamilton 163
3. Loveland 187
2. Hilliard 198
1. Grove City 204 (!!!)
So people in Grove City spend twice as much as other Ohioans at these restaurants.
Bonus Fact #2: Here are the cheapest—er, lowest spending—cities in the state on this food:
5. Mentor 79
4. Massillon 76
3. Youngstown 73
2. Canton 64
1. Mansfield 49
People in Mansfield spend about half as much as other Ohioans.
If you’re a data scientist, and you happen to enjoy baseball, then MLB DFS (daily fantasy sports) is the game for you. If you like parting ways with your money, then it’s doubly enjoyable.
There is no shortage of advice and access to statistics to help you in your never ending quest for the perfect lineup. You can absorb knowledge about stacks, chalkiness, fading, and other esoteric terminology. Experts will try to convince you that you must build lineups differently for cash games versus GPP tournaments. Silly me, I just thought I had to build a lineup that’s better than everyone else’s, and then beat them all. Kind of like real baseball. I wonder if Tito Francona fields a different lineup if he thinks he’s in a cash game or a GPP? I kind of doubt it.
Some of the most entertaining DFS advice can be found on Rotogrinders (www.rotogrinders.com). In addition to statistical models, statistical probabilities, and statistical statistics, there are podcasts and–on today’s topic—videos featuring past big winners sharing their strategies for the perfect lineup.
One such video, appearing a couple of times a week, is Beer’s MLB Six-Pack. Chris “Beermakersfan” Prince runs through tonight’s slate of ball games, choosing 5 players who, according to past performance and gut instinct, should rack up those fantasy points tonight. Then a sixth player is highlighted, the so-called “Beast of the Night.” This player, saved for the grand finale of the video, is the equivalent of the “best bet” of all the players in all the lineups in all the games. Chris’s recommendations are based on sound, empirical logic (what some of us still quaintly refer to as “sabermetrics” because it makes us sound profoundly smart) using the most recent statistics about every single aspect of a plate appearance.
So how reliable is Beer’s Six-Pack when it comes to building your lineup? Well, like everything else in DFS, reliability can be quantified. For the non-technical reader, that means “counting.” Let’s see how Chris does.
I conducted some extensive research (i.e. searched YouTube) for this season’s Beer videos. I had, of course, seen most of them before, coincidentally on the same day I was building my MLB lineup. Imagine that. I analyzed 19 days’ worth of videos, 114 separate recommendations (19 of whom were Beasts), across 5,060 players who had at least one plate appearance on those 19 days between April 3 and June 20.
Each day was analyzed separately. The statistic of interest was this: how many FanDuel fantasy points did the Beer recommendation players produce relative to the rest of the league? Did Chris choose the best 6 producers that night?
My methodology was to produce an ordinal rank each of the 6 recommendations relative to all of the other players in MLB lineups that day. Since the number of available players differed based on the number of scheduled games, weather, and other occasional weirdness, I normalized the rankings be converting them to a percentile.
So, for example, Chris recommended the Red Sox’s left fielder Andrew Benintendi on April 3, and he produced 25.7 FanDuel points. He went 1 for 4, that one hit being a home run that yielded 3 RBI. He also struck out once.
Benintendi’s 35.7 points put him in the 89th percentile of all 210 batters that night. Not bad! Yasmani Grandal, the Dodgers’ catcher, delivered a whopping 40.9 points, putting him in the 97th percentile. Here’s the analysis for April 3:
All in all, a good night for Beer (aren’t they all?). But like everything else in baseball, the true measure of greatness is the full body of work. Here are the results of all 19 games. The center line represents the median score for that day for all players, and the blue bars show the results of Chris’s selections that outperformed the median. The red bars show underperformers. The BEAST scores are on the far right.
Overall, Chris’s picks outperformed the league average 51% of the time. The 19 Beasts of the Night did marginally better, performing above average 57% of the time.
As a group, the Beasts scored about the same number of points as the league average, while the other 5 picks scored below that mark. But this is one of those cases where averages can be misleading. For instance, say you’re standing in a bucket of ice water with your head sticking in a hot oven. On average, you feel fine. That’s a data scientist joke.
Now is the time you ask, “so what?” What do you do with this precious information? Here’s my completely non-scientific hypothesis:
- Chris “Beermakersfan” Prince probably knows more than you when it comes to building DFS lineups
- You don’t even know what wOBA stands for, let alone wRC+.
- You have a sneaky suspicion you want to put Edwin Encarnacion in your lineup tonight, and you watch the Six-Pack show, and sure enough, he’s the Beast of the Night! You came upon that roster spot independently! And Chris agrees with you! What an intuition! What a perfect lineup! You’re a genius!
- The Indians are rained out tonight. Thanks for playin’.
Growing up in Northeast Ohio, I do not recall ever seeing, let alone actually kicking, a soccer ball. In those times and in that place the term “football” meant something entirely different. It meant an oblong, leather-clad, brown inflated ball. It meant glorious Friday nights at Mollenkopf Stadium. It meant watching the Ohio State Buckeyes stomp on the University of the Sisters of the Poor every Saturday afternoon. And it meant exploring new and exciting ways to express one’s displeasure and disgust at the Cleveland Browns every Sunday. So naturally I wondered how the 2014 FIFA World Cup was playing in this nether world of chauvinistic American sport.
Through the magic of the Twitter API, R code, and a few extra moments of time on my hands, I set forth on the journey to find out.
The R scripts for this little project can be found here. https://github.com/dino-fire/worldcup
The Twitter REST API enables users to set a geographic parameter to limit searches to a specific geographical area. The search terms were limited to #WorldCup, #worldcup2014, or #Brazil. These terms were subsequently eliminated from the analyses, because we’re interested in what people are saying about those terms, not about counts of the terms themselves.
I started with the latitude and longitude of Columbus, Ohio, and specified a 200-mile radius. A word cloud, of course, yields larger, more prominent displays of words with higher frequencies. The basic word cloud of Ohioans’ tweets demonstrate some interest in the Spanish and Croatian futbol teams. Speaker of the House John Boehner garnered a few honorable mentions as well. What that has to do with the World Cup, I do not know.
Next I made a little side excursion that explored the tweets from the Youngstown/Warren area with those of residents of Youngstown’s sister city, Salerno, Italy. The outcomes were predictable but nuanced. The Youngstown and Warren folks tweeted about the generic USA. Could’ve been the soccer team, could’ve been native cuisine, like hot dogs, and could’ve been anything. Not so with the Italians, though; the national football club was front and center.
Ohioans are people of few words, at least as far as tweeting about the World Cup is concerned. The vast majority of Ohioans’ tweets comprised 8 or 10 unique words. The base R program provides a nice histogram.
Before we get into the deeper statistical analysis, I should point out that THE BIG BUZZ at the time was about England getting unceremoniously booted from the tournament in the opening round.
What’s the difference between England and a teabag? The teabag stays in the Cup longer.
A hierarchical cluster analysis of Ohioans’ tweets is intended to depict how words tend to cluster together in Euclidean space. It’s a fancy way of seeing how words correlate. And here are the results.
One group of tweets centered on England’s demise, and another seemed to be about who was showing up in Rio de Janeiro. Yet another group of words dealt with the Italy – Costa Rica match, while a fourth cluster seemed to inquire about who was supporting US soccer.
Disregarding the clustering of words, we can review the correlations themselves. I’m proud to say that Ohioans are expert analysts of English soccer.
Despite a seemingly infinite number of startups claiming to do better social media mining better than anyone else, sentiment analysis is an iffy proposition at best. For those who aren’t blessed with 50 unsolicited emails a day from social media mining companies, sentiment analysis refers to an evaluation of a tweet from a subjective, qualitative standpoint. The analysis tries to classify tweets or other textual content “scraped” from various websites into “good” or “bad,” “happy” or “sad,” or other such bipolar sentiments. But often that’s where the problem arises. For example, the following tweet would be classified as “good:”
Well, England, that was a good effort.
But unfortunately, so would this one:
Well, England, THAT was a good effort.
He or she whom invents a sentiment algorithm that can accurately interpret sarcasm wins the prize. Yeah, THAT will happen. Scrape THAT, you bums.
Nevertheless, I’ll hop upon the sentiment analysis e bandwagon and see how Ohioans feel about the World Cup so far. First of all, we see that there is no transformation of the sentiment-scored data required. The results reflect a very normal distribution, not skewing one way or another too badly.
We see that the sentiment scores are more positive than not, but as of this writing, the USA team is 1 – 0. Those scores are subject to shift later, to be sure.
In this case, the sentiment scoring algorithm freely admits that it is clueless about the context of many of the words it encountered. Still, it seemed predisposed to find and tag joyful comments.
The sentiment scoring algorithm output a nice comparison word cloud, which visually demonstrates the words and their respective classifications based on frequency. Yes, I always associate the term “snapshot” with “disgust.” Interestingly, “Redskins” got lumped into that classification as well.
So are Ohioan’s beliefs about the World Cup different from other, surrounding, and, some would believe, inferior types of people (based on their state of residence)? Well, let’s see.
Sentiment scores in Ohio, Michigan, West Virginia, Pennsylvania, and Indiana lean uniformly positive. But a careful look at the boxplots show that Ohioans and Indianans opinions tend to cluster in the middle: not too positive, and not too negative. That’s not the case among Michiganders, who tend to be extremely more positive or extremely more negative. Those Michigan folks represent very nicely the dangerous reality about averages: You can be standing with your feet in a bucket of ice water and your head in a roasting hot oven. But on average you feel just fine.
A comparison cloud shows just how different the tweets from these separate states really are. Michiganders seem obsessed with the Italy – Costa Rica match. Indianans seem strangely interested in the Forza Italia political movement. Pennsylvanians are engrossed in a game of “where’s Ronaldo?” Ohioans are losing interest, and starting to turn their attention toward Wimbledon. And West Virginians don’t seem to care much about the World Cup at all.
The neighborhood around Cooperstown, New York was upgraded considerably in 2007 when two particular men, Tony Gwynn and Cal Ripken, were enshrined into the Baseball Hall of Fame. To say that these guys represented the “Class” of 2007 is true in any number of dimensions.
I moved to the Baltimore/Washington area in 1983, which is right about the time both Ripken and Gwynn came on the MLB scene. Ripken of course, soon became a minor deity in Baltimore. Growing up in the Midwest and later living on the east coast, I did not much follow the San Diego Padres. They might as well have been the Pluto Padres as far as I was concerned. What I did know about them, however, was codified in the persona of Tony Gwynn.
No tribute to Tony Gwynn is complete without that hilarious commercial he and Bip Roberts did for MLB about twenty years ago. Bip mistook the value of Robin Roberts’ rookie card for his own, until Gwynn corrected him. As a baseball card collector, I could not stop laughing. Besides, I had a Bip Roberts rookie card, but not a 1949 Bowman Robin Roberts one, unfortunately.
Watch, remember, and enjoy. RIP Tony.
Back by popular demand…derived importance.
A great deal of research is designed to measure the relative impact of specific features of products or services on customers’ satisfaction with those products or services.
Sometimes, surveys are designed to measure importance of those features explicitly and in isolation—no further analysis is necessary than an understanding of which features are more important to customers than others.
In other cases, the importance metrics will be used to determine what, if anything, could or should be changed to improve the product. That’s where key drivers analysis comes in, but more about that later.
Measuring importance through traditional Likert scales, while certainly frequently done, is not the method FGI recommends to measure importance. There are 2 fundamental reasons for this.
First, importance scales often do not provide adequate discrimination and differentiation between product features, especially when viewed in aggregate.
Q: How important is price?
A: Oh, that’s very important.
Q: How important is product availability?
A: Oh, that’s very important.
Q: How important are helpful store employees?
A: Oh, that’s very important too.
Second, people use scales differently (and this problem is not limited to importance scales). Respondents tend to calibrate their responses to previous scores. For example, here’s Respondent #1, rating the 3 attributes in our survey.
Q: How important is price?
A: Let’s give it a 9.
Q: Now, how important is product availability?
A: Well, not as important as price, so let’s say 8.
Q: How important are helpful store employees?
A: Less important than price, but more important than availability. 8 for that one too.
But Respondent #2 may follow precisely the same response pattern—9 / 8 / 8—but start their ratings at 6 instead, yielding 6 / 5 / 5. Should we view these three features as more important for Respondent #1 than for Respondent #2? No. Do any of Respondent #2’s answers qualify for top-2 box summaries?
No. One’s person’s 9 rating may be another person’s 6 rating. The very nature of scales—that the values are relative, not absolute—can cause misinterpretation of the results.
There are occasions where stated importance is appropriate and useful. If this is the case, there are far better ways than Likert scales to measure it, but that’s a subject for another day.
Measuring derived importance
Key drivers analysis yields importance in a derived manner, by measuring the relative impact of product features on critical performance metrics like overall satisfaction, likelihood to purchase again, likelihood to recommend, or some combination of those. The structure of a key drivers questionnaire looks like this:
Q. This next question is about your satisfaction with XX in general. Please rate the store on how satisfied you are with them overall. 10 means you are “Completely Satisfied” and 0 means you were “Not At All Satisfied.”
This question is treated as the dependent variable for our analysis.
Q. Now, consider these specific statements. Using the same scale, how satisfied are you with XX on…
- Variety of products and services
- Professional appearance of staff
- Length of wait time
- Ease of finding things in store
- Length of transaction time
- Convenient parking
- Convenient store location
We can then do some analysis to determine to what extent each of these independent—aka predictor—variables influence overall satisfaction. This is done through something called Pearson’s R Correlations.
In correlations, we get a statistic called R^2 (R squared), which is a measure of the strength of the score of one item to another. In the case of Pearson R, 1.0 means a perfect, positive correlation and -1.0 reflects a perfect, negative correlation. An R^2 value of 0.0 means no correlation at all.
In a key drivers analysis, the higher the correlation between each of the specific attributes and overall satisfaction, the more influence that attribute has on satisfaction, thus the more important it is. Notice that we never have to ask the question “how important is…” since the derived importance tells us everything we need to know. But that’s only half of the equation.
As a result of the question structure, we get explicit satisfaction metrics on each of the individual attributes as well. This data tells us how well we perform on each of the attributes. The resulting output looks something like this:
In our example, “helpful staff,” “coupon policy,” and “items in stock” are the most important attributes; they have the highest correlations to overall satisfaction.
Now compare those attributes to “store location.” The correlation is still positive, but not nearly as powerful as the first two examples. Remember, derived importance measures importance of individual attributes in relative, not absolute, terms.
The second part of our analysis shows that our store’s employees are helpful. In fact, it’s the highest performing attribute of all (while importance is viewed on the X, or horizontal, axis, performance is viewed on the Y, or vertical, axis).
This means that our store does well on this important attribute, and is considered a core strength. This is not the case with the other important attribute, like having items in stock, however. Our store gets the lowest performance rating on that very important feature.
From our survey results, management can quickly see that resources should be directed toward reducing wait times (more cashiers), improving their coupon policy if they can, and especially keeping popular items in stock.
We’ve precisely identified the few items that need to be prioritized, as improvement in satisfaction with these things will have a direct and measurable impact on overall satisfaction.
The MLB All-Star game is coming up soon, so I thought I’d toss a few random analyses your way to commemorate the occasion. Here’s one…
So you want to be an All-Star, do ya? Then change your name to Rodriguez or Robinson. Here are the surnames of the top 250 All-Stars, by number of All-Star game selections, going back to the dawn of the All-Star game, in 1933 Chicago. Unfortunately, notable baseball fan Al Capone was probably not in attendance, since he had other commitments at the time in the Big House. But I digress. The bigger and bolder the name, the more someone with that name appeared in an All-Star uniform. This fine graphic represents the intersection of baseball and big data. For example, Robinson refers to the Orioles’ immortal third baseman, Brooks Robinson (18 career All-Star games), Frank Robinson, player-manager for my beloved Tribe despite those gawd-awful red uniforms (14 career selections), Eddie Robinson, who represented the White Sox and Twins in 4 contests, and of course Jackie Robinson, with 6 games as a Brooklyn Dodger. Frank Robinson was an All-Star selection for 3 of the 4 teams he played for in his career–Cincinnati, Baltimore, and LA. He never made it to the All-Star game as an Indian. Of course. All of the Robinsons on this list are in Cooperstown. Rodriguez is attached to Alex, Ivan, Ellie, Francisco, and Henry.
The word cloud was created using the R wordcloud, tm, and rColorBrewer packages. The simple R script and data file can be found at https://github.com/dino-fire/allstar-analysis.
Like all of this and my upcoming All-Star analysis, a huge shout-out goes to the data geniuses at Baseball Reference. More baseball statistics than are fit for human consumption. This blog has been cross-posted to the most excellent R-bloggers site as well.
I have a proposition for you. It’s like one of my heroes said, it’s an offer you can’t refuse.
You stand next to a 17-inch wide rubber platter, holding a 36-inch long cylinder of ash or maple in your hands. Another, much more athletic individual stands precisely 60 feet and 6 inches away from you, and throws a baseball in your general direction as hard as he can. When it crosses that 17” plate, the baseball will be traveling between 85 and 100 miles per hour. The ball may or may not hit you. You will almost certainly not hit the ball. (A major league hitter can connect with it one time out of three if he’s very good; what makes you think you can do better than that?) Don’t worry, it will be over in about 2.3 seconds.
For your troubles, I will now award you $11,628. Before taxes, of course. What a deal! You get 11-large to stand there and get a baseball thrown at you. Once. And for every additional time you stand there and have that ball thrown at you, I’ll give you another $11,628. How long would you stand there? How many pitches would you confront for that money?
Well, if your name happens to be Miguel Cabrera, and my name happens to be the Detroit Tigers, you will come to the plate 675 times, and stand there look
ing at 2,500 pitches between now and October. At $11,628 per pitch, I will give you $29 million dollars. And to sweeten the deal, we will do this, you and I, for the next 10 years. Deal?
You’d smile too.
This is not a fantasy, except for the part that you are not the one collecting that cash, and I am not the one doling it out. This is simply one way to view the $292,000,000, 10-year contract the Tigers “inflicted” on Miguel Cabrera this week. Here are a few other fun ways to look at this princely sum.
- A typical ball game lasts 3 hours. Miggy shows up to work: he has played in an average of 157 of the 162 games in a season since he came to the American League from the Miami (nee Florida) Marlins in 2008. So he makes $61,571.13 per hour. The federal minimum wage is $7.25 per hour. You, on the other hand, would have to work 8,500 hours to earn what Miggy does in a single hour. That’s 4 years, chum.
- Cabrera is a third-baseman, mostly. He averages having 1,026 balls hit at him every year. Do the math: that’s $28,265 per fielding chance. He also makes 8 errors per year. He would probably refund the Tigers the $226,000 he got for those particular misplayed chances, but the union contract won’t let him. So you places yer bets and you takes yer chances.
- The Detroit Tigers drew 3,083,397 fans to Comerica Park in 2013 to see their Central Division winning team. And, Cabrera’s 2014 salary of $29,000,000 represents an $8,000,000 raise from 2013. So each fan needs to pony up another $2.59 to cover the additional labor cost. Put another way, the Tigers need to sell a million more $8.00 beers. Shouldn’t be too hard when you think of it that way.
“Professional baseball is on the wane. Salaries must come down or the interest
The more things change, the more they say the same.
I got a Fitbit device a few weeks ago (stop laughing). If you’re not familiar, these things keep track of your exercise (such that it is), and help you stay motivated to get in shape. It does so by tracking the number of steps you take each day, among other things.
One thing it does not do is track your heart rate. Which is a good thing, because it would have exploded last night in the 9th inning of the Indians game against Kansas City. Chris Perez, we all have to die of something, and my demise will be because of YOU.
Perez has the lofty vocation of “closer,” meaning he has one job–admittedly a difficult one–but it is one job. Get three outs, Specifically, the last 3 outs of a ball game, but only those games in which your team is already winning. He did so successfully 39 times last year, but has only 23 saves thus far in 2013. Interestingly, he actually has 5 wins this year, and 3 losses (he was 0-4 in 2012). How does a closer acquire 5 wins, you ask? By blowing saves, that’s how, at which point the Indians’ offense comes in and bails you out in extra innings. It’s not something to be proud of, CP. These are the facts, and they are not in dispute.
Perez blew 4 saves in 2012, and has blown 5 thus far in 2013. Not a big difference statistically, but boy did they seem to hurt this year. Most painful was the 6-5 loss to Boston at the end of May, the same game where he added injury to insult. Perez gave up a three-run 9th-inning lead in that game. Suspecting an injury, he was asked to throw a warm-up pitch, which he promptly launched into the general area where Kevin Costner and James Earl Jones sat in the Fenway Park scene in Field of Dreams. I couldn’t help but think it was intentional: “See, look how hurt I am! It wasn’t my fault!”
And what does Mr. Perez have to say for himself about that Boston game, last night’s Royals game, or any other game, for that matter? Nothing. Nada. Silencio. You see, CP has declared that he would not be talking to the media whatsoever this year, because “it is too much of a distraction to the team.” How convenient. Why did you walk a number 9 hitter batting 0.167 on four straight pitches? Have the consequences of having a bunch of weed shipped to your dog affected your performance? Or is it due to the substance itself? Crickets.
Talking to the media–and vicariously, to the fans that make your mediocrity possible–won’t necessarily make you Mariano Rivera overnight. But hiding behind a “no talk” clause is a poor way to absolve yourself of accountability. In case you haven’t noticed, CP, some people think the Indians actually have a chance at some October baseball this year. Last year, CP castigated fans for not coming out to Progressive Field in larger numbers, This year, you might be well advised to answer that question, at least in part, by looking in a mirror.