Samsung Mobile's head of camera R&D wants your phone to 'personalize' your photos

How much is too much?

Apart from image quality this year, the best thing about the Galaxy S21's cameras is the flexibility they offer — the 12-megapixel main shooter is a solid performer, while the ultra-wide and telephoto cameras do a great job adding and subtracting space between you and your subject. Naturally, the Ultra amps up the excess a little further by packing two distinct telephoto cameras, one with a 3x optical zoom range and another with a 10x optical range. And then there's Space Zoom, which — like last year — allows you to zoom in as far as 30x or 100x depending on which phone you're using.

The feature mostly felt like a technical flex last year, but improvements to the way these phones stabilize these super-long-range photos are starting to make Space Zoom feel legitimately usable. But at what cost? Yes, being able to shoot reasonably detailed images of the moon at 100x is undeniably cool — but what about the ability to watch people from extreme distances? How does a company that accounts for roughly one-fifth of all the smartphones in the world balance the usefulness of this technology with some obvious ethical implications?

Honestly, I was hoping for some kind of thorough, reasoned argument, but Samsung didn't have one for me.

"I understand your concerns, but whenever we launch a product we receive thorough legal advice at a global level, and we only launch products we can be proud of," Cho said. "I think you can be assured that we always follow through all of the legal processes and procedures thoroughly."

Pride in a technical achievement is perfectly valid, and Cho is also right about legality. In the United States, at least, it isn't illegal to walk around with a camera and a super-long zoom lens and take photos of people in public spaces. Creepy, sure, but not illegal. But let's not forget about privacy.

Frankly, it would be hard to forget about it. Samsung takes every opportunity during its new smartphone launches to tout its Knox on-device security, which features multiple layers of protection designed to shield your personal data from external threats. This year, the Galaxy S21s will also ship with a feature called Private Share that lets you control access to the documents and images you send to other Galaxy users, and even strip location data from photos you may want to pass around.

Clearly, your privacy is a major concern for Samsung, but our conversation didn't leave me with the impression that the company was overly concerned with other people's privacy when building out this feature. For what it's worth, the company provided a statement on Joshua’s behalf a few days after our conversation:

"Our mission is to bring the best mobile experience to our consumers and make their lives more convenient and enriched. In order to do so, we’ve relentlessly pushed innovation to provide Space Zoom capabilities in our smartphones. With Space Zoom, people can capture more than ever before, even from a distance. So far, our innovation has been beneficial to many and we remain committed to democratizing the latest technologies. As with all our innovations, we hope and ask for consumers to use their technology responsibly and respectively."

Samsung’s next steps

As we've already established, the Galaxy S21s come with a lot of cameras — arguably enough for most situations you'd feasibly find yourself in. But now that all those bases seem adequately covered, where does Samsung go from here?

One potential approach is obvious: just add more cameras.  Cho believes there's no upper limit to how many cameras a smartphone should have, so long as they all jointly meet Samsung's core criteria: producing eye-catching images, no matter the conditions. But the flip side is also true. There's nothing stopping Samsung, or any other company for that matter, from moving toward one single camera that — thanks to some help from AI — can perform multiple roles. With the right kind of training, for instance, machine learning could help a single sensor pull double duty as a macro or telephoto shooter. (To be clear, that's my example, not Cho’s.)

"When there are ten people taking a picture of the same object, I want the camera to provide ten different pictures for each individual based on their preference."

While he wouldn't confirm whether Samsung actively planned to embrace a one-size-fits-all camera empowered by AI, it's an area of obvious interest to the company's imaging software chief — he mentioned the possibility himself in a piece published to the Samsung’s media site. 

"Our AI technology is continuously improving thanks to the image training technology we have developed at Samsung," he said in November. "At the same time, we are also working to deliver the best camera hardware for our devices. If we are able to achieve the same results from multiple cameras with just a single AI-powered camera, then we will merge the technologies."

For now, he says, the limiting factor is silicon — or more specifically, the neural processing units currently available in chipsets like Qualcomm's Snapdragon 888 or Samsung’s own Exynos 2100.

"At the moment, we are in year three of the NPU," Cho said. "I would call that still an initial stage. You have to reach at least year four or five from the initial launch of the NPU in order to see it replace servers. And when that time comes, I think that one of the players in the field — it doesn't have to be Samsung — could come up with a single AI camera."

Given the weight the company places on its global survey, and its keen desire to satisfy as broad a swath of its audience as possible, it's perhaps no surprise that Cho seems more enamored by the idea of tuning Galaxy cameras to individual tastes. In other words, in the near future, the photos your Galaxy phone takes may look nothing like the ones mine take.

"My goal is to provide a camera that can satisfy everybody 100 percent through personalization," Cho said. "When there are ten people taking a picture of the same object, I want the camera to provide ten different pictures for each individual based on their preference of the brightness, the color tone, the detail enhancing, etcetera."

The underlying mechanisms to make this on-device learning happen aren't quite ready, again because neural processing units are still essentially in their infancy. When asked how long it would take before a smartphone company could make a truly "personalized" camera, Cho admitted that "no one can answer that at the moment." Even so, he has a firm sense of where he and his team could start once the silicon starts catching up to his ambitions. 

"You could look at the user's album to find out what pictures they have saved over the past years as opposed to deleting," he noted. "Also, you could look at what kind of editing they do, what filters they used the most. Those are some of the things that we could look at in order to ensure the system learns about the user."

While the future may bring ultra-personalized photos that shine for you and you alone, expect Samsung to stick to its tried-and-true formula in the meantime. That means developing more new camera sensors, cooking up kooky features polling its users to capture the visual spirit of the moment, dutifully tweaking its image processing year after year.. And if you don't like the vibe Samsung landed on? Just tweak the photos yourself — maybe the company will pick your brain this year.