Tons of comic books have come to life as movies, but researchers at the Warsaw University of Technology have developed a system for turning live-action video into a comic. Comixify picks out the best frames from any video you feed it and turns it into something that looks like it belongs in the pages of a comic.
Comixify: Transform video into a comics. They used a 2-stage approach: (a) frame selection and (b) style transfer. The results look pretty cool!
web demo: https://t.co/QZq41b2TnN pic.twitter.com/FGcPfsN9u2
— hardmaru (@hardmaru) December 12, 2018
According to the researchers, the project uses a neural style algorithm based on Generative Adversarial Networks (GANs). The machine learning technique by using two different networks: a generator and a discriminator. The algorithm is fed tons of information, which the generator uses to create new data instances. The discriminator is then tasked with determining if the instance is from the dataset it was trained on or generated by the other network. The process continues until the generator is producing work so similar to the training set that the discriminator can no longer tell the difference.
Because comics only have so many frames, the system has to determine the best, most representative moments from a video to create the comic. Once the algorithm has picked its frames, it then applies the comic style to them. The end result is a pretty reliably solid reimagining of any given video as a page from a comic book.