Crude Sprite Prosthesis
Motivation¶
So I stumbled across a YouTube video a good while ago that had me transfixed.
YouTubeVideo('U1KiC0AXhHg')
The idea was so simple and so amusing that I knew it'd be an excellent candidate for a quick hacking session. However, when I sat down to actually make it happen, it turned out to be deceptively tricky.
Here's what I figured out:
Setting Up the Problem¶
Napkin Mathing¶
So for starters, I knew I had to be able to express an image in terms of "punched areas" and "ignored areas." I replayed the video and screen-grabbed finished product before it panned over to the sprite.
Image('images/washington_punched.png')
Then I opened up my favorite, free image-editing software and took a look at the size of these boxes.
Image('images/washington_total_height.PNG')
If you look at the bottom-left corner, my dashed-line selection was about 290 pixels. Measuring one of the boxes, I found them to be about 10x10 pixels each, so now I just had to figure out how to express the blank space relative to the box-size.
Well, 11 boxes in my selection means 10 blank spaces in between. So a bit of napkin math yields
(290 - (10 * 11)) / 10
18 pixels. We'll round to 20 for neatness' sake.
Functional Programming Magic™¶
In sketching out the size of "keep" and "skip" spaces, it was clear that I could generalize the pixel selection to "skip a few, keep a few, skip a few, keep a few..." until I covered the height of my image. Then repeating the same process for the width of the image, I'd arrive at the same hole-punched result that the video had.
I've been playing with the fantastic itertools library a lot this week, so when I saw "infinitely repeat the same pattern," I knew I'd want to use something like the inifinite functions in itertools (my notes here).
Concretely, I wanted to be able to specify a num_skips and num_keeps, then devise some endless loop that would serve the value 0 for num_skips sequential steps, then a 1 for num_keeps sequential steps.
def spaced_zero_one_generator(num_skips, num_keeps):
while True:
for _ in range(num_skips):
yield 0
for _ in range(num_keeps):
yield 1
For anyone not in the know about Functional Programming-- specifically as it relates to Python iterators-- I'm pretty pleased with how I explained it here.
But let's unpack what this code's doing.
For starters, we can define this iterator once for use throughout our program. We'll use the values 20 and 10 that we worked out above.
our_iter = spaced_zero_one_generator(num_skips=20, num_keeps=10)
Looking at the code, the first thing that should be immediately obvious is that all of the actual logic is contained in a while True: loop and will thus run forever.
In the next couple of lines we basically say "I want to produce the value 0, num_skips (20) times". The underscore here is simply good code style and effectively says "we're not interested in using the values 0, 1, 2, ... that range() provides.
So when we look at the first 10 values that this generates, they should all provide 0
for _ in range(10):
print(next(our_iter), end=' ')
And again
for _ in range(10):
print(next(our_iter), end=' ')
Then, crucially, the next 10 should provide us 1s
for _ in range(10):
print(next(our_iter), end=' ')
And then switch back to 0s
for _ in range(10):
print(next(our_iter), end=' ')
Finally, if this is able to produce results infinitely, we can just run it once over the height of the image, then again over the width.
for _ in range(1000):
print(next(our_iter), end=' ')
Looks good to me.
Executing¶
So now that we've got a rough approximation of how we'll skim over the image height and width, what do we do next? Let's talk about mask really quick.
Masks¶
Essentially, a mask is a simple filtering criteria that matches the dimension of whatever you're filtering over. If it's True, you keep the value, if it's False you drop it.
So as a contrived example, we've got 6 letters and we define todays_vowels to be 5 sequential Trues and one False
candidate_values = np.array(['a', 'e', 'i', 'o', 'u', 'y'])
todays_vowels = np.array([True, True, True, True, True, False])
And when we select elements from candidate_values using the todays_vowels mask, you see that the last value drops out because its corresponding todays_vowels element was listed as False
print(candidate_values[todays_vowels])
Similarly, if we set that last value to True, we see that y is indeed sometimes a vowel.
candidate_values = np.array(['a', 'e', 'i', 'o', 'u', 'y'])
todays_vowels = np.array([True, True, True, True, True, True])
print(candidate_values[todays_vowels])
So in case it wasn't immediately obvious why we cycled through values of 0 and 1 in the last section, it's becasue we'll be assembling a mask over our image, because in Python 0 is the same as False
0 == False
And any non-zero value (we'll use 1) evaluates to True
1 == True
Building the Mask¶
So let's assume that we've got a 500 by 500 image.
im_height, im_width = (500, 500)
We want to iterate over the height of the image and generate the appropriate sequence of 0s and 1s to determine our skip/keep scheme
our_iter = spaced_zero_one_generator(num_skips=20, num_keeps=10)
vertical_mask = []
for _ in range(im_height):
vertical_mask.append(next(our_iter))
vertical_mask = np.array(vertical_mask)
vertical_mask
And we'll do the same to create the horizontal_mask
our_iter = spaced_zero_one_generator(num_skips=20, num_keeps=10)
horizontal_mask = []
for _ in range(im_height):
horizontal_mask.append(next(our_iter))
horizontal_mask = np.array(vertical_mask)
However we've got a small hiccup here-- Our original image size was 500 x 500. But here, we've only got two arrays of length 500.
print(vertical_mask.shape)
print(horizontal_mask.shape)
Thankfully, the numpy library comes chock full of helpful functions and documentation to manipulate arrays and matricies. For this problem, we'll employ np.repeat(), which essentially takes our N x 1 array and stacks arbitrarily-many copies on top of the other to build a consistent matrix, like so.
vertical_repeated = np.repeat(vertical_mask, repeats=500).reshape(500, 500)
vertical_repeated.shape
horizontal_repeated = (np.repeat(horizontal_mask, repeats=500)
.reshape(500, 500).T)
horizontal_repeated.shape
So now we've got the correct shape for our masks, but what's really valueable about this is what we've got underneath.
plt.imshow(vertical_repeated);
plt.imshow(horizontal_repeated);
If both of these images represent a 500 x 500 matrix of values, then every purple pixel represents a 0 and every yellow a 1. Finally, we can take advantage of some boolean identities and mash these two together using the and operator.
print(False & False) # 0 and 0
print(False & True) # 0 and 1
print(True & False) # 1 and 0
print(True & True) # 1 and 1
So as you can see, we should only see values that evaluate to True or 1 when both of the corresponding values are True or 1. And that's precisely what we get!
final_mask = horizontal_repeated & vertical_repeated
plt.imshow(final_mask);
Nice!
Filtering¶
Alright, so it looks like we're home free. Our mask looks like it follows the same general "punch out regularly-spaced" holes that the original Washington image did above. Let's use it!
But first, we'll cast our final_mask to be explicitly of type bool, so numpy knows that we intend to use it as a mask.
final_mask = final_mask.astype(bool)
Then we'll load a candidate image
im = cv2.imread('images/hannibal.png')
im = cv2.cvtColor(im, cv2.COLOR_BGR2RGB)
plt.imshow(im);
And filter based on our Mask.
3...2...1...Voila!
sprite = im[final_mask]
plt.imshow(sprite);
Wait, hang on, that is kind of Wack.
What happened?
Reassembling¶
Taking a closer look at sprite, our biggest tip-off is the shape of the matrix
sprite.shape
For comparison, our original image had shape
im.shape
That last 3 is reserved for the (Red, Blue, Green) values. Both 500s are for height and width, respectively.
Therefore, filtering im on final_mask simply consolidated all of the pixels into one long array.
So getting to our intended sprite image should be as easy as figuring out the new N x N dimension and reshaping this long array to meet that shape. Luckily, our starting image was a square, and our traversal was identical vertically and horizontally, so the output should be a square as well, which makes our last hurdle a trivial arithmetic problem.
edge_len = np.sqrt(sprite.shape[0])
print(edge_len)
At long last, Hannibal Buress goes from "Wack" to "V, half of a C" as we intended.
plt.imshow(sprite.reshape(160, 160, 3));
Conclusion¶
I figured most of this stuff out on stream (give me a follow!), but ultimately went on to add:
- Support for images that aren't perfect squares
- Custom
skip_n,take_nvalues - Smart defaults for when you don't want to supply either
And I packaged it into a tidy little script that you can download and use for your own images. If you wind up making anything cool with it, I'd love to hear about it :)
from spritify import im_to_sprite
Anyhow, here's some of my favorites
Blur¶
Using a small enough skip_n, take_n leads to basically a "Cops blur" effect
coffee = cv2.imread('images/coffee.jpg')
coffee = cv2.cvtColor(coffee, cv2.COLOR_BGR2RGB)
plt.imshow(coffee);
plt.imshow(im_to_sprite('images/coffee.jpg', skip_n=7, take_n=1));
20XX¶
The implications that this has on retro-fitting Smash Bros Ultimate to Nintento 64 graphics are still undetermined
fox = cv2.imread('images/fox.png')
fox = cv2.cvtColor(fox, cv2.COLOR_BGR2RGB)
plt.imshow(fox);
plt.imshow(im_to_sprite('images/fox.png', skip_n=7, take_n=10))
f(f(x))¶
And speaking of video games, downsampling the pixels of sufficiently-large pixel art is basically negligible, haha
mario = cv2.imread('images/mario.png')
mario = cv2.cvtColor(mario, cv2.COLOR_BGR2RGB)
mario.shape
plt.imshow(mario);
plt.imshow(im_to_sprite('images/mario.png'));
