Google's latest developer tools could make Assistant more capable

Google wants to make the Assistant more powerful, and is announcing today a slew of developer-centric updates that will make it easier to build tools for the digital helper. Three new features in particular are most intriguing: Home Storage, Continuous Match mode and AMP support on smart displays. Google is also releasing a new web-based console and SDK that will make it easier for developers to create for Assistant. These announcements are the largest since the company acquired conversational user experience platform Dialogflow in 2016, according to director of product management Payam Shodjai.

With Home Storage, programmers can tap into a pool of information shared by devices in the same household (or, in developer speak, connected on the home graph). They can use that to create experiences with saved contextual info. For example, a persistent jigsaw puzzle that can show up on all synced smart displays, and each authorized user’s moves will reflect on all the screens. More useful scenarios could include a shared whiteboard or recipe pages with notes on ingredient substitutes — it’s up to developers to come up with these applications.

Another update with potential to improve the Assistant is Continuous Match mode, which is rolling out in the next few months. It enables a temporary session during which the device’s mic will remain open instead of only listening after the Assistant has finished speaking. Google said the implementation will be transparent, and that “before the mic opens, Assistant will announce the mic will stay open temporarily so users know they can speak freely without waiting for additional prompts.” This will allow for more realistic back-and-forth between you and the Assistant, making for some fun interactions. Web game maker Coolgames is making use of this feature by launching a new “Guess The Drawing” activity in a few weeks that will let you keep shouting out answers until you get it right.

To make smart displays more useful, Google is also announcing AMP support for those devices, so you can see compatible content on your screens. Since you can only really use existing Google smart displays as sort of a digital photo album with alerts from your calendar, this will greatly expand what you can do on them. You can read articles or look at recipes, and who knows what else once developers get more creative with the feature. AMP is coming to all Google smart displays (and that’s including third-party devices from companies like Lenovo, JBL and KitchenAid) later this summer, starting with News articles.

Google is also updating its Media API to support longer-form content so users can resume playback at specific points or where they last left off across a variety of devices.

Giving developers access to these functions is pretty futile if there was no easy way to build for the Assistant. Shodjai told Engadget that his team heard feedback that “building actions for Assistant is pretty hard,” involving jumping back and forth between Dialogflow and the rest of Google’s console. Today, the company is releasing a web-based Actions builder that consolidates all of that in one place. The integrated development environment (IDE) will let developers code, test, debug and release from the same window. Those who prefer using their own local IDEs can get the Actions SDK to do so.

While most of the changes that will make a difference to consumers still aren’t available yet, there’s plenty to look forward to. Shodjai stepped into his current role within the last year, and he said that we can expect more news later this year. In the coming months, for example, we’ll be hearing more about web content for smart displays. “There’ll be some moments this summer,” he added, “where we focus on a lot of the family-oriented use cases.” He also teased some news coming in the education and gaming areas — “a lot of things that people are doing while they’re at home.”

The Assistant continues to get more powerful, but not everyone knows the breadth of skills it offers. Even Shodjai didn’t know how much it could do. “As someone who just joined the team, there’s so much that the Assistant does that I didn’t know before,” he said, acknowledging that this is one of the greatest challenges he wishes to tackle. “This question of how does someone discover features and functionality, how do they continue to remember that the Assistant can do certain things for them,” he explained. It’s even trickier when you consider that the main way people interact with Assistant is by speaking. “When the primary interaction model is voice, and you're not actually tapping on apps or things on the screen, you have to remember a lot more.”

It’s a challenge we might see addressed soon. “It's something that my team is thinking about deeply and I actually quite like the challenge because it's a completely new way of thinking about how to make things more discoverable,” Shodjai said, before adding that we’ll also see more news on this front later this year. We’ve already seen Google run an experiment on some Nest devices in Washington State where Shodjai is currently, that tests a card-based interface to surface “What can you do?” tips, so it’s not hard to imagine something similar might be on its way.

Further down the line, though, Shodjai sees the Assistant being able to fulfill more complex tasks end to end in the next couple of years. “Wouldn’t it be great if the Assistant could help you put together your weekly shopping list or figure out which stores actually have your items from, or actually order them for you as well,” he described, saying that’s a direction he sees Assistant moving towards as well.

Like most other Google executives across products like Nest and Pixel, Shodjai also firmly believes that the idea of ambient computing is where the future is headed. He sees the concept as a “constellation of devices” — some communal like a smart display, some personal like a phone — all working with the cloud to deliver help wherever you are. It won’t all be voice-based interactions, either. Gestures and touch will also play a role in a “multimodal” system. “The promise of the Assistant is that it ushers in a new era of ambient computing where we are less tethered to our devices and more present doing the things we love, while still having the confidence that the technology is actually there at any moment to help us,” he said.

“That for me is like the Holy Grail.”