Samsung Mobile's head of camera R&D wants your phone to 'personalize' your photos
Exclusive: Samsung's Cho digs into AI, Space Zoom and what may come next.
Samsung announced its first Galaxy S smartphone in the heady days of 2010, and at the time people were too jazzed by its 4-inch Super AMOLED screen and 1GHz processor to fret much about its cameras. The same could be said of Samsung itself — the company's original US press release mentioned them a grand total of zero times outside of the spec sheet.
Eleven years and millions of Galaxy phones later, cameras have become a crucial part of Samsung's smartphone identity. If you needed any proof, just look at the company's new flagship devices, which go on sale today. The Galaxy S21 and S21 Plus pack four cameras apiece, while the high-end S21 Ultra sports five (and generally touts a greater emphasis on telephoto shooting). And while pundits and reviewers tend to go back and forth on the merits of Samsung's approach to cameras, most of them (myself included) were impressed with what the company pulled off this year.
That warm reception was music to Joshua Sungdae Cho's ears. After picking up two post-graduate degrees at Rensselaer Polytechnic Institute in upstate New York, Cho signed on with Samsung in 2004 and rose through the ranks to become vice president and head of Samsung Mobile’s visual software R&D. In an exclusive conversation with Engadget, Cho — flanked by PR people and an interpreter in a small conference room in Korea — pulled back the curtain on how the company lands on a vibe for its image processing, highlighted the growing importance of AI and hinted at how the future of Samsung cameras may involve making your photos look totally different from mine.
Perfect trendiness
If you've read any recent review of a premium Samsung smartphone — seriously, pick one at random — you'll probably find a bit that calls out the company's approach to image processing. Fine details in photos are noticeably sharpened, sometimes overly so. Depending on the year, you might find mention of Samsung auto-smoothing people's faces with a computational airbrush, though that's less of a problem this year. Maybe most notably, colors and brightness are amped up to the point where they look punchier and more appealing than reality. Like it or not, this is The Samsung Look, and we've been dealing with variations of it for years. But why is it what it is?
Well, you have your fellow users to blame — or thank, as the case may be.
Despite Cho's role as de facto head of camera R&D, he doesn't act as an impassioned visionary articulating what Samsung's pictures should look like. No single person does. Instead, Cho said through an interpreter that the company relies on in-house imaging experts and color scientists, along with a panel of professional photographers, to hone in on the factors that make photos eye-catching. But expert advice only takes Samsung so far; that's where the company's global survey comes in.
Every year, Samsung reaches out to thousands of customers in all of its major markets, from the US to Canada to Spain and China and beyond, to find out why they like the photos they like. At a high level, representatives ask these people — be it through focus group or distributed questionnaires — about their photographic preferences and the pain points they keep running into. The conversations can get pretty granular, though. In trying to suss out what people like about their favorite images, Cho says discussions can veer toward subjects like color tone and saturation, noise levels, sharpness of details, overall brightness and beyond, all so Samsung can tune its HDR models and its smart scene optimizer to deliver what he calls "perfectly trendy" photos.
In other words, if you don't like the way the S21 sharpens details and amps up saturation, you're not just disagreeing with Samsung — you're also disagreeing with everyone they polled. (Not that that's a bad thing.)
Samsung places a lot of weight on the results of this survey because it desperately wants to please everyone, even if it knows it can’t. How could it? The company accounts for roughly one-fifth of all the smartphones used around the world and individual tastes, nurtured through personal experience and shaped by shifts in culture, don't sit still for long. By gathering and chewing on all of this purely subjective information, though, Cho and his team are trying to capture a sense of a year's visual zeitgeist and reflect it back at you.
How much is too much?
Apart from image quality this year, the best thing about the Galaxy S21's cameras is the flexibility they offer — the 12-megapixel main shooter is a solid performer, while the ultra-wide and telephoto cameras do a great job adding and subtracting space between you and your subject. Naturally, the Ultra amps up the excess a little further by packing two distinct telephoto cameras, one with a 3x optical zoom range and another with a 10x optical range. And then there's Space Zoom, which — like last year — allows you to zoom in as far as 30x or 100x depending on which phone you're using.
The feature mostly felt like a technical flex last year, but improvements to the way these phones stabilize these super-long-range photos are starting to make Space Zoom feel legitimately usable. But at what cost? Yes, being able to shoot reasonably detailed images of the moon at 100x is undeniably cool — but what about the ability to watch people from extreme distances? How does a company that accounts for roughly one-fifth of all the smartphones in the world balance the usefulness of this technology with some obvious ethical implications?
Honestly, I was hoping for some kind of thorough, reasoned argument, but Samsung didn't have one for me.
"I understand your concerns, but whenever we launch a product we receive thorough legal advice at a global level, and we only launch products we can be proud of," Cho said. "I think you can be assured that we always follow through all of the legal processes and procedures thoroughly."
Pride in a technical achievement is perfectly valid, and Cho is also right about legality. In the United States, at least, it isn't illegal to walk around with a camera and a super-long zoom lens and take photos of people in public spaces. Creepy, sure, but not illegal. But let's not forget about privacy.
Frankly, it would be hard to forget about it. Samsung takes every opportunity during its new smartphone launches to tout its Knox on-device security, which features multiple layers of protection designed to shield your personal data from external threats. This year, the Galaxy S21s will also ship with a feature called Private Share that lets you control access to the documents and images you send to other Galaxy users, and even strip location data from photos you may want to pass around.
Clearly, your privacy is a major concern for Samsung, but our conversation didn't leave me with the impression that the company was overly concerned with other people's privacy when building out this feature. For what it's worth, the company provided a statement on Joshua’s behalf a few days after our conversation:
"Our mission is to bring the best mobile experience to our consumers and make their lives more convenient and enriched. In order to do so, we’ve relentlessly pushed innovation to provide Space Zoom capabilities in our smartphones. With Space Zoom, people can capture more than ever before, even from a distance. So far, our innovation has been beneficial to many and we remain committed to democratizing the latest technologies. As with all our innovations, we hope and ask for consumers to use their technology responsibly and respectively."
Samsung’s next steps
As we've already established, the Galaxy S21s come with a lot of cameras — arguably enough for most situations you'd feasibly find yourself in. But now that all those bases seem adequately covered, where does Samsung go from here?
One potential approach is obvious: just add more cameras. Cho believes there's no upper limit to how many cameras a smartphone should have, so long as they all jointly meet Samsung's core criteria: producing eye-catching images, no matter the conditions. But the flip side is also true. There's nothing stopping Samsung, or any other company for that matter, from moving toward one single camera that — thanks to some help from AI — can perform multiple roles. With the right kind of training, for instance, machine learning could help a single sensor pull double duty as a macro or telephoto shooter. (To be clear, that's my example, not Cho’s.)
"When there are ten people taking a picture of the same object, I want the camera to provide ten different pictures for each individual based on their preference."
While he wouldn't confirm whether Samsung actively planned to embrace a one-size-fits-all camera empowered by AI, it's an area of obvious interest to the company's imaging software chief — he mentioned the possibility himself in a piece published to the Samsung’s media site.
"Our AI technology is continuously improving thanks to the image training technology we have developed at Samsung," he said in November. "At the same time, we are also working to deliver the best camera hardware for our devices. If we are able to achieve the same results from multiple cameras with just a single AI-powered camera, then we will merge the technologies."
For now, he says, the limiting factor is silicon — or more specifically, the neural processing units currently available in chipsets like Qualcomm's Snapdragon 888 or Samsung’s own Exynos 2100.
"At the moment, we are in year three of the NPU," Cho said. "I would call that still an initial stage. You have to reach at least year four or five from the initial launch of the NPU in order to see it replace servers. And when that time comes, I think that one of the players in the field — it doesn't have to be Samsung — could come up with a single AI camera."
Given the weight the company places on its global survey, and its keen desire to satisfy as broad a swath of its audience as possible, it's perhaps no surprise that Cho seems more enamored by the idea of tuning Galaxy cameras to individual tastes. In other words, in the near future, the photos your Galaxy phone takes may look nothing like the ones mine take.
"My goal is to provide a camera that can satisfy everybody 100 percent through personalization," Cho said. "When there are ten people taking a picture of the same object, I want the camera to provide ten different pictures for each individual based on their preference of the brightness, the color tone, the detail enhancing, etcetera."
The underlying mechanisms to make this on-device learning happen aren't quite ready, again because neural processing units are still essentially in their infancy. When asked how long it would take before a smartphone company could make a truly "personalized" camera, Cho admitted that "no one can answer that at the moment." Even so, he has a firm sense of where he and his team could start once the silicon starts catching up to his ambitions.
"You could look at the user's album to find out what pictures they have saved over the past years as opposed to deleting," he noted. "Also, you could look at what kind of editing they do, what filters they used the most. Those are some of the things that we could look at in order to ensure the system learns about the user."
While the future may bring ultra-personalized photos that shine for you and you alone, expect Samsung to stick to its tried-and-true formula in the meantime. That means developing more new camera sensors, cooking up kooky features polling its users to capture the visual spirit of the moment, dutifully tweaking its image processing year after year.. And if you don't like the vibe Samsung landed on? Just tweak the photos yourself — maybe the company will pick your brain this year.