NVIDIA's AI fixes photos by recognizing what's missing
The results are uncannily accurate.
Most image editing tools aren't terribly bright when you ask them to fix a photo. They'll borrow content from adjacent pixels (such as Adobe's recently demonstrated context-aware AI fill), but they can't determine what should have been there -- and that's no good if you're trying to restore a decades-old photo where you know what's absent. NVIDIA might have a solution. It developed a deep learning system that restores photos by determining what should be present in blank or corrupted spaces. If there's a missing eye in a portrait, for instance, it knows to insert one even if the eye area is largely obscured.
There have been previous attempts at AI-guided fills, but they've typically been limited to rectangular sections, have focused on gaps near the middle of the picture and haven't scaled well to missing photo data of different sizes. NVIDIA's "partial convolution" approach, which guarantees that the output for missing pixels doesn't rely on the input values, can work with holes of any shape, size or location. That, in turn, produces uncannily realistic results in many cases -- even if the AI doesn't know exactly what's missing, the result usually looks like it fits. Previous methods tended to produce obvious glitches.
NVIDIA trained its system by generating tens of thousands of hole variations and making the AI learn how to reconstruct photos. It tested using a different set of holes to ensure the AI genuinely understood how to restore photos on its own.
The results aren't always flawless. You may see a facial feature clearly borrowed from someone else, and it's bound to struggle if the hole is so large that there isn't enough information to create a plausible reconstruction. But what's here could still be incredibly useful. You could repair seemingly hopeless images without hours of painstaking reconstruction. The scientists also envision the AI helping to upscale images without losing sharpness. In effect, you'd only ever have to worry about touching up minor details -- the days of recreating whole segments from scratch might soon be over.
