He made Tom Cruise 'forget the mouse.' Now it's our turn.

In a way, John Underkoffler's like Hollywood's own Wizard of Oz. He's the man behind the curtain responsible for infusing blockbuster fantasy with real-world tech. He created the futuristic UI in Minority Report, worked on the timeline for Hulk's transformation and found a Soviet fusion reactor to blow up Stark Industries in Iron Man. He also recently received the Cooper Hewitt National Design Award for his work as an interface designer both in films and in the real world.

While his work as a technologist in films has been iconic, he's drifted away from that world. He now runs Oblong Industries in LA, where he stays laser-focused on creating UI that's more human than machine-like. I caught up with the designer for insight into the world of UIs and the delicate balance between fact and fiction in sci-fi films.

What is the significance of a technologist in sci-fi filmmaking? What does that job entail?

At the end of the day, I'm responsible for proposing ways in which real science and technology might be threaded through the narrative, so it's not merely decorative, which is what usually happens. But it actually becomes an element of the story or is a critical supporting factor in it. That was the case with UI in Minority Report. We were able to push the narrative forward through the depiction of the UI and what it enabled the "PreCrime" cogs to do. In Hulk, Ang Lee realized that all of the characters were scientists and that was the storyline, so it was about proposing plausible mechanisms by which the Hulk might actually work in a real world.

At what point do you come in to the process?

If science and technology are going to be a part of the story, someone like me comes in right at the beginning as the script is taking shape. Working on Hulk, for instance, there was a lot at the script level that needed context and had to be figured out. The main thing that Ang [Lee] was interested in was the history -- what series of experiments or mishaps actually gave rise to the Hulk? That process took six months until we figured out how all the pieces would fit together. What we ended up with is a kind of Frankenstein story, which is completely laid out in the opening credits.

An employee demos Oblong Industries' Minority Report-like user interface.

In the fantastical world of sci-fi films, how important is it to be grounded in reality? What is the equation between fact and fiction?

Audiences do have a fairly sophisticated understanding of science these days. If you show people something that looks like an extrapolation of what they already know, something that incorporates elements that are by now familiar, then there's a real plausibility that ties it all together and you've got this huge booster behind the story. Then you've hooked into the warp and weft of the actual world. Those results are always much more exciting.

In Iron Man, the very first one for instance, there was a need for a giant, dangerous thing for the final fight scene. And presumably, whatever it would be, needed to explode and cause massive havoc and be the punctuation mark at the end of the story. When I was looking around, I thought you know, Tony [Stark] is interested in energy, obviously; he's got this thing on his chest that's not an infinite energy source, but it's really important to power his suit. So I said, let's take a look at the tokamak, a ring-shaped fusion-containment device that originated in the former Soviet Union, but it's now a focal area as an energy source for nuclear fusion. It's resonant; it's Toroidal [a doughnut-shaped object] just like the thing on Tony's chest. So, of course, he would have a giant tokamak fusion reactor at Stark Industries. In a way, that illustrates how you're looking at all the properties of some real-world technology -- the color, the shape, what it sounds like, what's the scale, how do you connect to it -- when enough of those elements belong in a story, you know you have a very important piece.

How much do you think technology in film influences the evolution of technology in the real world?

Hugely. There's a long legacy. Famously, Arthur C. Clark invented the communications satellite in a fictional setting. What's interesting is, these days the feedback cycles are shorter. There were 30 years between Star Trek communicators and the Motorola flip phones. Now it might be a few years. And it goes in both directions. Now more than ever, film productions are actively looking to technologists to paint the big canvas of the future where they want to set their story.

That's definitely evident in sci-fi films, especially over the last decade or so. Filmgoers also seem to have the palate for the complexities of science and technology that have become integral to that genre. What do you think brought on this evolution?

I think the catalytic moment had to have been Minority Report, with all the clarity that we could muster in the medium of film -- what a new UI might look like? Why is it important? And why is it a critical element in enabling someone to do a really important job? We've seen dozens of recapitulations of the gestural UI since then -- in movies, music videos, TV shows, advertisements and even company vision videos. Then there's also a really exciting spectrum of non-gestural UIs. All of that taken together has raised the basic literacy levels and a demand among audiences. There's a certain new threshold of plausibility.

What about ideas that aren't plausible? Have you ever worked with a concept that's a little too far out?

One of the things that was really interesting about Aeon Flux and director Karyn Kusama, who is absolutely fantastic, is that there was an idea that was deeply appealing, but it was far enough out there on the edge of science. In that world, there was a kind of technological communication that was purely chemical. The idea was that spies would send messages to each other in the form of vials or capsules that they would ingest. It would go through their bloodstream and reach the brain. It would reconfigure or be stimulated in a way that they'd be injecting ideas or visuals into the brain. I thought that was really beautiful.

Even though the world of films sounds incredibly exciting, you've stepped away from it to focus on your work at Oblong Industries. How would you summarize your work at the company?

We're still using the same UI to talk to our computers that we did decades ago. It's largely irrelevant that the screens are bigger and there's more pixels and color, but the language that we have available to speak to those machines hasn't changed. It's the same pointer-based system in a 2D plane, overlapping windows, little icons, pull-down menus and that sort of thing. It feels endangered. It's really a fantastic opportunity to build something new, but also something that's necessary and has a logical extension of what we've got. So we're building UIs that let humans talk to the machine in a much more literate way. What's important is that to make it work and make it more sophisticated, the UI has to be much more human than machine-like and that's the beauty of this thing.

But what are the implications of that?

I don't understand this recent panic over the last nine months about AI as if suddenly something has changed and the machines are coming. I don't actually believe strong AI like that will ever come around. I think the UI have to be designed much more on human terms than the machine's terms. Arguably, the UI we have now is closer to the machine than it is to the human. Instead, at Oblong we're building interfaces and HMIs [human-machine interfaces] that take place in the real world. Human pointing is at the center of that. If your walls are covered in screens, then you can pick up one of our pointing devices and point at any of the screens. Your ability to address pixels, communication, data information and applications does not end with the physical boundaries of one screen as it does today. If you think about how a mouse works, you can't get it off the edge of the screen. We're saying forget the mouse; the human is important. The human can use fingers, a device or a phone to point and we can make the machines obey that gesture.

This interview has been condensed and edited.

[Image credits: John Underkoffler (lede image); Nicholas Kamm/AFP/GettyImages (gesture-based UI); Marvel (Iron Man)]