HappyShutter brings smile recognition to iOS

Smile-recognition technology is nothing new. Dozens of camera brands support this feature but it isn't yet part of iOS's built-in camera system.

Now for US$0.99, you can purchase HappyShutter. It's built using new iOS Core Image technology to detect both that faces are within-frame, i.e. not cut off, and when subjects are smiling. What makes this app interesting is that the detection is done in software, not hardware.

Dr. Roberto Valenti, ThirdSight CTO, explained that his team built detection tests using OpenCV, an open-source library that supports real time computer vision and ThirdSight's face analysis algorithms. The road to the app store, however, was a rocky one.

Realtime detection in software rather than hardware is limited by processor capabilities. Valenti writes, "ThirdSight owns technology which recognizes any facial expression in a video, and although we succeeded porting them on the iPad 2, the iPhone 4 was not able to run our technologies close enough to real time. Therefore we chose to implement a new frame-by-frame detection based system, which would decide whether the subject was smiling as soon as a face was detected."

So what they did was to train the system in advance. They downloaded thousands of face images from the Internet, manually labeling them as either smiles or not. "The prototype of the software was already working before last summer," he wrote. "However, we still encountered some technical difficulties while porting the software to the iPhone."

One big issue was a bottleneck of the standard OpenCV face detector. It was too slow to be usable in their app. "We started optimizing the OpenCV code with ARM NEON SIMD instructions, achieving quite a good performance improvement."

The introduction of iOS 5 brought the project into the realm of reality. "As soon as the new functions were documented, we replaced our OpenCV face detector with the one provided with the SDK. The speed improvement was not massive, but it was definitely a cleaner solution."

With face detection moved into Core Image, smile detection itself remained in OpenCV. "The improvements on the OpenCV library were still used in the smile detection part of the code, which allowed us to achieve fast smile detection on the device. However, we still had to go through a choice of image features representation and classification frameworks, and we had to choose to slightly cut down in accuracy in order to reduce the loading time and to keep the application size under 20 MB."

In the end, the merging of Core Image and OpenCV allowed HappyShutter to hit the app store in a form that was usable, with a software-only approach. If you'd like to give it a try, it's only a buck -- and it may help capture happier-looking memories.