Ruff Style Transfer
The last couple months have been blog-dark as I worked through the Deep Learning Specialization on Coursera. After the first couple courses of fundamentals, we brought some practitioners on site at work to give us a two-day-long bootcamp on a whole range of topics. I've since worked my way through the next couple of courses regarding project structures and computer-vision applications; I also got my computer doing all of my model-training on my GPU (which equals dramatically-faster code!).
Getting handy with Deep Learning techniques has been a goal of mine for some time now and so I'm excited to be able to start sharing some of the stuff I'm figuring out.
Overview
This is an alarmingly-reductive representation of what's happening under the hood, but if you're not familiar with Neural Networks/Deep Learning, the idea is that you feed some data in on the left, it comes out on the right with some sort of prediction. Along the way, Data Science™ happens as a result of our model iteratively figuring out the equations to affix to the black lines and blue circles. Provided that we give the program enough data and let it run for a long time, it can "learn" some really complex behavior.
An interesting application of these techniques is called Neural Style Transfer where basically, we crack open one of these structures, show it two pictures, and let it figure out how to splice the two of them together by borrowing the Content from one and Style from the other. Some of the more famous examples look like the following:
But this isn't intended to be a tutorial post (that's been expertly done here and here). Instead let's look at some of the more ridiculous applications I've stumbled into!
A Dog Day Afternoon
Instead of visiting with friends and playing in the Alumni Marching Band at Michgan last weekend, I spent my Saturday at home sick as hell and tinkering with the keras deep learning framework. It took awhile to get everything up and running, but once I was set up, I was able to enjoy a "fire-and-forget" model training workflow where I could provide a couple images, then let the program do its thing while I went and napped.
And hey, check it out, it works!
I took a picture of the castle my best friend got married in and weighed that against a really cool painting I found on /r/imaginarylandscapes
And while the result didn't really mesh whatsoever...
... it was obvious that it was figuring out some sort of way to combine the pink, streaky elements of the second picture, while still preserving the content of the first.
So now that I was capable of using this technique, the question became
"What's the single-most impactful use of this new skill in my toolkit that I can employ today?"
Anyways, Look at My Dog
This is Daisy. She's a goof.
I initally set out to see if I could use Style Transfer to punch up a picture this old girl.
I tried for bombastic.
Then stately.
Finally, I let the algorithm get carried away Miazaki-fying her.
A More Earnest Attempt
Perhaps it had less to do with style transfer, and more to do with starting with a flattering picture.
Then apply Style Transfer, we do.
Or this one.
What if we decide that she should look more like a snow leopard?
And so this was certainly amusing for awhile.
But what if it was weirder?
Unleash THE_PACK
Before we launch into this next section, I feel it bears a quick bit of explanation for friends out there that spend less time indoors than I do.
Rules of the Road
/r/THE_PACK (NSFW) is a funny corner of Reddit where nearly 100k subscribers are all rallied around the same, ironic in-joke.
At its core, the entire sub is sarcastically derivative of the kind of machismo, curse-word laden, skeleton-bearing images like you might see on shirts like this
But often lousy with Early 2000s-era Microsoft Office WordArt...
...about hysterically-specific things...
...written from the persona of an edgy middle-schooler who's just getting the hang of computers and swear words...
...and living out power fantasies, sharing the thoughts that they can't in real life.
They're occasionally pretty wholesome...
...and almost always steeped in irony.
But that's the long-and-short of it. It's genuinely one of my favorite subreddits. If this isn't the funniest thing you've seen all day, I don't know what to tell you.
An Unholy Marriage
So naturally, I was going to Style Transfer THE_PACK. I took to /r/aww to find some prime, dog-shaped candidates.
First off, I spied this good boy on a shopping trip and elected to pair him against a needlessly-anthropomorphic dragon with a dash of spelling error.
I think my favorite thing about how this turned out is that the font on the cart handle adopted the gross, block lettering from its counterpart.
I was much more excited to run this next one if for the diversity in color/letter style alone.
And boy did it deliver, lol
Finally, I paired this cute pup in surgery recovery with, perhaps, my favorite post on the whole subreddit.
I set it up to run 1000 iterations and then went to bed.
And woke up to a beautiful disaster.
And that was the brunt of what I intended to share.
Bonus: Simpsons Couch Gags
But then I was outlining the premise of this post to someone earlier and after the initial wave of confusion, they offered that it might be interesting to try and generate some variations on the classic couch gag.
Why, yes it would.
Doing It
So we start with a base image
And we spend our Work From Home Day doing actual work on the laptop, while periodically firing off
python transfer.py images/simpsons.png images/some_garbage.png --iter=100
and returning to results of... mixed quality.
The Good
I started off with that Banksy piece that's been in the news this week and it turned out pretty great.
I'd seen other projects have a lot of success transferring psychedelic-style art, so it came as little surprise when it worked well here.
So did a still from the Take On Me music video.
And the design document for my tattoo, courtesy of Johnny Andres
But I think the house from Up wound up being my favorite result.
The Bad
I think the DEEAAAAAAAD looked similar-enough to Marge's necklace next to the top of Lisa and Maggies' hair that it caused their faces to bleed together.
Dali blurred everything. Go figure.
Apparently South Park is a show about everything melting.
And the essence of Spongebob Squarepants is a downright-Lovecraftian number of eyeballs.
...the Hell is That??
Finally, I fed it this picture of Steve Buscemi for about an hour, because why not?
And if you stare at this like a Magic Eye puzzle, legend has it that he actually enters your soul.
This was as fun to write as it was weird to read. If you've got the hardware that can swing it, putting two images together is basically a one-liner from the command line. Source code is here-- hit me up on Twitter if you come up with anything particularly bad!
Cheers,
-Nick