Spout Effects

This program is a Windows app that applies effects to video streams and re-outputs them as a Spout2 source. This works great with the OBS Spout2 plugin for use as a source.

History

I initially made a simple ASCII converter in Python for fun. This looked great, but had several downsides. Being a command-line application, I was stuck with a lot of undesirable features. Adding color would require printing 4x the data - and the application was already struggling to reach 30 fps. Changing text size and the font was incredibly finicky and made it difficult to quickly prototype different styles. And most importantly: no hardware acceleration.

It's hard to see due to compression, but you can see it in action below:

So naturally, in order to fix these small problems, I began a totally new project from scratch and decided to learn OpenGL, which I knew nothing about, while I was at it.

The result was Spout Effects - made with C++/OpenGL, it's the result of a lot of trial and error. I went in blind and looked up as little help as possible. Initially rendering each character with a separate draw call, I approached more advanced concepts like batch rendering naturally.

I reimplemented my original Python program and optimized it, carrying out the entire process on the GPU. However, I still wasn't quite happy with the results. A big problem with ASCII filters is the loss of detail - you're essentially downscaling the image by 8 times, or whatever the size of your characters is. Because of this, busier scenes can make it difficult to decipher what's happening on the screen. Lucky for me, Acerola, a very talented graphics programmer, noticed the same problem around the same time and created a very unique solution. I implemented a similar approach, which I'll describe below.

Essentially, by using straight-line characters (_, |, /, ) at the edges of objects, perceived fidelity can be greatly improved, even while keeping the same lower resolution. We have to calculate where the edges are in order to know where to use these characters, but we also have to know which one of these directional characters to use. For that, we use the Sobel filter to detect the direction of change in the image. This is illustrated in the figures below:

The colors in the processed image represent the direction of the edge. The actual angles are continuous over [0, 1], but these colors show the results after I've already partitioned the range into 4 discrete buckets for the 4 edge characters.

Something else you might notice is the large amount of noise in the image. This is addressed with the next step: a compute shader, used for downscaling the image. Remember, each character is essentially 1 "pixel", yet takes up 8x8 actual pixels. So using this compute shader, we do a simple downscale and take the opportunity to calculate all the data we'll need for our final step. For each character, we dispatch 8x8 local groups (read: threads) in the compute shader. Each thread samples 1 pixel and determines which type of edge it is, adding it to a shared array. After each thread is done, the first thread adds up the amounts for each edge type, selecting the most common type. This means that the noisy areas in the Sobel filter image will be narrowed down to whichever color was most common. We also have each thread store its luminance in a shared array. The first thread gets the average luminance and stores it in the output texture for future use when deciding which ASCII character to draw.

That's a pretty good result. There's definitely still some noise, and this is a fairly simple image - busier scenes have worse noise issues. However, these are fairly simple problems to fix. Using a Difference of Gaussians filter to clean up the image actually do smooth edge detection would cause the Sobel filter to introduce significantly less noise. That's currently still planned as future work - I am still a beginner in OpenGL and am learning as I go, so there's a good chance that problems arise and this takes longer than it should.