Google knows people don't like having cameras in their homes, but it still wants to offer useful features that depend on devices knowing if you're there. After all, the whole idea behind ambient computing -- a concept Google is pushing across its various hardware departments -- is predicated on devices all around sensing and adapting to you. Since the Nest Hub Max is the only Google smart home device to pack a camera, the company had to find a way for its other products to know when you're around. With ultrasound sensing, a new feature it announced today, Google may have figured out an effective workaround.
Gallery: Nest ultrasound sensing hands-on | 8 Photos
Gallery: Nest ultrasound sensing hands-on | 8 Photos
If you'll recall, the Hub Max's camera enabled it to recognize who was in front of it and surface what Google called Proactive Notifications -- alerts that are relevant to specific members of your household. Say your partner walks in front of the display, the Hub Max can surface their upcoming appointments and commute time (as long as they've been added to your Home network, of course).
The smaller Nest Hub, on the other hand, can't do this, since it doesn't have a camera and thus can't detect if anyone's around, not to mention who in particular is walking by. Ultrasound sensing, though, could help. It's rolling out to newer Nest devices, specifically the Nest Hub, Nest Hub Max, Nest Mini and Nest WiFi Point in the coming months.
The feature works by using the device's speaker to send out a high-frequency chirp and having the mics listen for feedback. (If you're worried about your beloved pets, by the way, fear not: Google told Engadget that even though the frequency used here is something dogs can hear, it's similar to typical noise in the home like that from a fridge.) With lots of math, the machines can tell whether there is a person within four or five feet and do a variety of things. For now, ultrasound sensing has the most obvious uses on the Hub.
When the devices detect that you're standing close to the display, it'll show touch controls and detailed information for things like timers, weather forecasts and your commute. Walk beyond the four- or five-foot range, and the interface changes to feature large font that you can read from further away, and removes the onscreen controls since you can't reach them anyway.
I was impressed by how well this worked during a recent demo. As I paced further away from the Nest Hub, a timer countdown expanded to fill up the entire screen when I was distant. When I got within range, the timer shrunk to its regular size and buttons to stop and pause appeared.
When Google Nest product manager for smart displays, Ashton Udall, asked the Nest Hub to pull up his commute information, a map with step-by-step directions and alternate routes appeared. When we walked away from the device, the screen changed to show just the amount of time his commute would take, in the same larger font as the countdown. The digits were green, indicating traffic was smooth, and Udall said the font color would change to yellow or red depending on how congested the route was.
This stretching and shrinking also apply to weather reports -- walk closer to the Hub and you'll get detailed forecasts for the next few days. Go further away and you'll just see the current temperature, high and low for the day as well as the chance of precipitation.
The smart display also gains the ability to surface Proactive Notifications like the Hub Max can. Sort of. When it can tell someone is near, the Nest Hub will show a carousel of cards with your latest alerts and upcoming calendar appointments. But since the device can't discern who exactly is standing in front of it, it can't cater its content to each individual, and you'll just get cards that are available to everyone in your home.
For now, that's about all the Nest Hub can do with ultrasound sensing: adapt the size and layout of its timer, commute and weather cards, and display proactive notifications if someone's around. But it's easy to see how Google can quickly implement new applications to make better use of the system on existing and older hardware. It's trickier though, when for devices that don't have a display.
This presence detection still comes in handy for the Nest Mini and WiFi Point speakers, although what they can do based on knowing where you are is pretty limited for now. When you hover your hand about an inch away from the devices, their indicator lights glow to show you where to tap to change the volume. Since they don't have physical buttons, you wouldn't otherwise know at first how to manually adjust those levels. And that's pretty much it for these two devices.
Another feature that's coming to recent Nest products with speakers can help you hear the Assistant or your news and podcasts over ambient noise. It's called Ambient IQ and works differently from the ultrasound technology. Say you're playing a podcast on the Nest Hub in your kitchen and turn on the blender. The display will realize there's noise competing for your hearing and tune its audio so it's easier to hear.
I played an NPR podcast on a demo Hub Max and then turned on a motorized toothbrush in front of it. When I brought the whirring gadget close to the display, the host's voice became tinnier and more nasal, but I was able to hear it over the sound of the toothbrush. I tried again with a hairdryer, and once again, the Hub Max tweaked its audio output to match and go over the frequency of the ambient noise.
It's not the most elegant solution, but it's at least more sophisticated than merely bumping up the volume to fight aural fire with fire. And from my very controlled demo, it appeared to be effective, and I'm eager to see if this works at home. Most of all, I'm heartened to see Google coming up with a potential solution to a problem that I thought I had to live with.
I still haven't tested these new features in the real world, but I'm already impressed by what I've experienced. It might sound creepy that Google is finding a way to detect your presence without a camera, but think of this as more of a motion sensor. Importantly for some of us, it can't actually recognize who you are. These audio-based tricks and features are making Google's smart speakers and displays even more intuitive than before.
Update (at 10/17, 1:20pm ET): This article was edited to say that the frequency being used for ultrasound is something dogs can hear, but that it's similar to typical noise in the home.