Looking at Lots of Letters
It's been a busy couple of months.
I'm working in a software engineer-ier capacity these days, building out a Sphinx-based documentation platform, standing up a Package Index for library code we've been brewing, and doing all kinds of internal writing about standards and best practices. Job title still says "Senior Data Analyst," but I've been signing emails as developer advocate-- an excellent characterization of what I've been up to these past couple years. It's been slow-going and frankly a bit taxing, but I'm optimistic that it'll translate into my doing some quality data science once things are platformed a bit better.
Of course, since I'm learning a whole lot during the day, my constructive extracurricular work has fallen to the wayside. And in turn my decompress hobbies have picked up in a big way. Since the middle of June, I've burned through:
- Mario Rabbids for the Switch: Weird and fantastic
- All 100-something Marvel Civil War comics and many others: Turns out comics are pretty cool. Who knew?
- Bloodborne on the PS4: It took three, multiple-month quits before I finally got gud.
- A good deal of skin: I've still got signs of the sunburn I got beginning of the month, lol
And last, but not least, my good friend Will recommended that I check out Daniel Kahneman's Thinking Fast and Slow. We've compared notes over the past few years on all things mental health, imposter syndrome, and Internet garbage. He was instrumental in helping me get a standup routine together that seemed to go over well-enough, and we're often thinking on the same wavelength. So when he made a book recommendation, I was certain that it would be a quality one.
Nevertheless, I didn't even make it through the introduction before I found myself itching to do some tinkering over a couple sentences that I read, lol
The Question¶
The book itself explores thinking and decision making-- establishing some simple constructions that help you conceptualize the way your brain works. It also provides insight into how that may occasionally work against us in the form of biases, and for us via heuristics (or roughly, rules of thumb).
And so while explaining the availability heuristic, or our tendency to form opinions based on the examples we can easily summon, he says the following:
Consider the letter K.
Is K more likely more likely to appear as the first letter in a word OR as the third letter?
And I stopped and went through the mental exercise.
cake, lick, kangaroo, clock, acknowledge, knowledge, know, knack, kick, ...
It seemed about even, but I was very aware that for every first-letter-k I was coming up with, I was deliberately suppressing another one while I thought even harder for a third-letter-k, in some clumsy balancing strategy. Then all of a sudden, I was in my head about it, and the reflexive, "ease of access" point he was trying to impart was wasted on me.
Be that as it may, what was the answer?
My Approach¶
Basically, the mental-model I was starting to think up looked like the following:
I was going to have a big old list of every letter, where each letter had its own corresponding (empty at first) list.
letters = make_letter_dict()
print(letters)
And I'd be able to take a word, and see where each letter fell (first, second, etc)
for idx, letter in enumerate('kickback'):
print(idx, letter)
Then I'd be able to populate those sub-lists, per letter, with a count of where letters have occurred.
For example, here, k occurs once in the 0th, 3rd, and 7th places.
print(get_letter_dists(['kickback'])['k'])
And so as I layered in more words, I'd be able to get a running total of counts, per letter, of where they appeared in my list of words.
print(get_letter_dists(['kickback', 'knock', 'knuckle'])['k'])
But to extend this idea to the point where I can make real inference, I was going to need a ridiculous amount of data.
A Ridiculous Amount of Data¶
So I found myself borrowing from the popular.txt by GitHub user Dolph, which is a combination of the very-famous Enable 1 dataset, cross-referenced against Wiktionary's word frequency lists generated from a comprehensive look at scripts from English-speaking TV shows and movies.
words = load_all_words()
It's over 25,000 words long.
len(words)
It includes words spoken by the everyman, you and me.
print('dog' in words)
print('beer' in words)
print('data' in words)
print('science' in words)
But doesn't include $2 words of high-society.
print('tophat' in words)
print('wainscoting' in words)
print('bourgeoisie' in words)
print('universalhealthcare' in words)
As well as just about all of the good swear words (left as an exercise to the reader).
Running It¶
I ran the same "letters by count in words" across this whole dataset, this time, opting to store it in a more-easily-readible tabular format.
df = get_letter_dists(words, asFrame=True)
df
Idea to execution, procuring and wrangling this dataset was actually much easier than I'd originally thought.
The real fun was trying to figure out compelling visualizations.
Visualizing It¶
Admittedly, my matplotlib chops have always been "good enough to get the job done" but have always taken a back seat to some of the meatier Python topics. (And we're not going to talk about the D3 book collecting dust on my book shelf :( )
I had a lot of thoughts about what some nice visualizations might look like, doodling and shopping ideas from friends who are better at this stuff than me (Here's where I'd love to link the blog you should write. You know who you are!). But as I started hurling spaghetti code at my interpreter, it soon became clear that I'd be doing myself a lot of favors if I started peeling back the curtain on matplotlib a little bit. It wound up being a good an excuse as any to start padding out the Data Viz section of my notes-- explaining things I've probably Googled dozens of times.
Ultimately, I had a lot of fun deciding what a visualization was going to look like, then figuring out how to leverage the expressive API to make that happen.
Here are some of my favorites:
The Meat of the Post¶
First, I tried taking a look at how these counts measured up, per letter, as a percentage of that letter's overall counts. It's clear here that k shows up as the 4th far more than the first or third spot (sorry, folks not used to 0-indexing!).
I wound up leveraging the seaborn library to organize the plot, and in so doing stumbled across a fascinating talk on how the devs picked this color gradient.
letter_dist_heatmap(df)
But this was almost too close to tell, depending on how well you can interpret differences in hue.
Instead, I went with a more traditional approach and did a bunch of bar charts, and it helps you get a better idea of the how the letters are distributed. k still looks pretty close.
plot_letter_dists(df)
But a closer inspection of the k chart reveals that the first letter does, indeed, occur more often than the third.
plot_just_k(df)
Look out for my unofficial sequel, Thinking Fast and then Tinkering at Your Computer for a Couple Nights
Bonus Scrabble Round¶
Before I closed the lid on my tinkering, I was struck by some of the distributions of the letters that I'd found. j was overwhelmingly the first letter. o and u the second. Many others followed an almost-normal distribution.
So I did what you'd expect of any data scientist who'd gotten destroyed in Scrabble the night before, and I reworked the visualization to also include point values of each letter. Study your losses, right?
plot_letter_dists_scrabble(df)
I thought it was interesting to see that vowels earn their one-point-edness and by having a pretty wide distribution, making them flexible tiles. But you might also notice that all of the two and three point consonants share a pretty handy availability as more-often the first letter in a word, making it much easier to play onto an uncrowded board. Conversely, you'll have to scratch your head a bit to make a quality play with z.
Alternatively, you might also just look at how prevalent these letters were across our text and find a much simpler interpretation.
plot_letter_volume_scrabble(df)
Conclusion¶
Thanks for reading! Per usual, the code I used to generate all of the above can be found here. I'm gonna go get back to that book.
Cheers,
-Nick
Update¶
Had an interesting suggestion come my way via my post to Reddit, where someone was curious how this distribution shook out when you examined when a letter was the last letter in a word.
Went back and reworked it. Check out d, g, s, and y!
from IPython.display import Image
Image('post_reddit_questions_files/post_reddit_questions_31_1.png')
