Video: 3D Eye Tracking from TAT, the guys behind the T-Mobile G1 UI

Ever notice the word "Malmo" emblazoned across the face of Android's analog clock widget? That's a subtle nod to the city where the Swedish design team called TAT (The Astonishing Tribe) is headquartered -- a group of nerds responsible for finessing the look and feel of the T-Mobile G1's user interface including such innovations as the window shade menu and 9-point visual key-lock. In fact, TAT's software prowess can be seen on the Samsung Instinct, S60 handsets from Nokia, and a variety of devices from Motorola and Sony Ericsson. So when TAT releases a new "3Deyetracking UI" concept that lets you see behind on-screen objects, well, you can bet that the entire cellphone industry is paying attention. Check the video after the break.

Read -- TAT's industry influence

[JamesR @ Feb 24th 2009 8:58AM]

This just blew my mind ... And my depth perception.

[G Man @ Feb 24th 2009 10:18AM]

Holy crap, is the camera wearing 3D glasses?

[Tiptup300 @ Feb 24th 2009 1:42PM]

I see how it could be done semi-easily. Have a motion senser and a camera on the front. NAILED IT.

[chefgon_ign @ Feb 24th 2009 5:27PM]

You wouldn't even need a motion sensor, the front-mounted camera is all you need to establish a line-of-sight angle between the user's eyes and the screen.

[Tiptup300 @ Feb 24th 2009 9:14PM]

You're expecting that camera to be extremely high quality to be able to see how the face is angled. Why care? It would be dumb to worry about people moving their heads, You'd just tilt the phone itself instead of paying for more expensive parts for simple tech.

[Nate @ Feb 24th 2009 8:59AM]

Rather than "3D Eyetracking" looks more like "Use accelerameter to modify display". Looks cool I guess, but seems to have absolutely no practical use.

[Gabo @ Feb 24th 2009 9:22AM]

Practical? It's a smartphone! I thought they were using an accelerometer, too, but if they say eye tracking, they must be using a camera. Otherwise they just have to guess the angle at which you naturally hold your phone. It's a fancy solution to a problem that really doesn't exist. That's what's known in the industry as 'innovation.'

[Skrying @ Feb 24th 2009 10:11AM]

The eye tracking gives it an absolute position to go by. Accelerometers this accurate would be far to expensive, and even then likely still not able to do it smoothly. Additionally your last point basically explains why we have researches AND engineers.

[chris @ Feb 24th 2009 10:35AM]

Its an accelerometer that's controlling the effect. There are ALREADY programs in existance that do this "special effect" for the HTC Touch Diamond, Touch Pro, and Touch HD.

[Skrying @ Feb 24th 2009 12:04PM]

Sigh. You can mimic it, but the accuracy and where they're actually reading data from is totally different. Eye tracking is not something new at all, the applications of it have been limited so far in the real world. The demonstration in the video doesn't show some of the advantages of tracking the eye, but with an accelerometer you can easily fool it, with eye tracking you can't. In fact, eye tracking all around could produce vastly more accurate and impressive results than an accelerometer with regard to effects based on user perspective.It is however harder to get right and costly with regard to computation. Which is why we have a simple demo and not seeing it on a released phone.

Some people...

[VampireHunter Z @ Feb 24th 2009 12:16PM]

I can think of one practical use. If I'm multitasking I could peek over at an app running in the background to see it's status or progress. Number of clicks....zero.

Simple things like installations ,downloads or video buffering can all happen in the background while I do something else provided it doesn't take up all the screen. Your screen real estate literally expanded. I can't wait to see where they go with this.

[Chris @ Feb 24th 2009 1:31PM]

@VampireHunter Z or peek under a girls skirt. O_o

[fh @ Feb 24th 2009 1:56PM]

Or say if you're watching a video full-screen, you can simply tilt the screen to look at status messages (new email, messages, call notifications) without having to click through the interface, minimize the movie, or dismiss popup messages.

[sam @ Feb 24th 2009 5:04PM]

This is not using an accelerometer. Using an accelerometer wouldn't work at all. To see why, imagine what happens if you move your head to look at it from a different angle, leaving the phone where it is. Accelerometer: no reading. Eye tracking: fine.

However, I notice it said 'concept video'. I.e. they may well not have hardware working at all...?

Note that typical eye tracking is used to detect the direction your eyes are looking in. For example, you can use eye-tracking devices to analyse how people view websites - which area of the screen do they look at first, etc. That requires it to detect the direction your eyes are looking in.

This task is much simpler: it only needs to direct the direction your eyes are looking *from* (relative to the phone) - so it only needs to know where your eyes are and not the exact direction of the pupil etc. If you can believe the trailers, even Nintendo's DSi is able to detect facial position well enough to give people cat-ears in live video using a VGA camera and a 133 MHz ARM processor (less powerful than high-end phones). That's more or less what we're talking about here, so it seems perfectly feasible. However, if it's going to work over a wide range of angles, it would need a correspondingly wide-angle camera so it can actually see your face.

You might have already seen a video demonstrating the exact same effect by Johnny Lee who did experiments with a Wii remote. Using two infra-red LEDs mounted on the sides of somebody's spectacles, and the camera from a Wii remote located on top of the TV facing toward the player, he was able to make a normal computer screen show this 3D look-behind-things effect. This is the same technique but instead of the infrared camera and LEDs, you use a normal camera and face-recognition technology to spot your actual eyes (treating those as the 'LEDs'.)

Oh, and not that this isn't cool - it's *really* cool - but I don't see much actual point to it in phones? Johnny's demo showed some kick-ass game potential if it were applied in that way - you could duck down behind in-game objects to take cover - but I'm not sure that applies very well to a two-inch screen.

[zellbz @ Feb 24th 2009 8:59AM]

This look real nice....TAT also designed some of WM6.5 as well(lock screen)

[Jeff @ Feb 24th 2009 9:01AM]

Please bring this to android!!!!

[Jay @ Feb 24th 2009 9:10AM]

Awesome UI... better late than never... a little eye candy cant be negative on smart phones... besides its intelligent 3d illusion without the glasses?

Id laugh my ass off if apple struck an exclusive deal to lock this down with them... it would screw up the rest of the market thats for sure....
but what I want to know is what handset are they using... processor, graphics, memory.... or is it a normal smart phone...

looks similar to the toshiba snap dragon handset...

[Major4Play @ Feb 24th 2009 10:59AM]

Yes I'm sure TAT would settle for exclusively working for a small player like Apple. They get BIG money from Nokia already.

[Brad @ Feb 24th 2009 5:51PM]

Plus, then Apple would have to do things like have a front-facing camera, and a system that supported multi-tasking.

And they'd have to admit to something coming from outside their golden palace design labs. None of those is going to happen.

[Jay @ Feb 24th 2009 6:33PM]

Haha... your right... but arent they supposed to be launching a refresh to the iphone 3G in june which should have a front facing camera?
They deff wont have this by then thats for sure...

Maybe I am too skeptical but I am not convinced about the eye tracking thing on the above video.... as i am sure you can recreate the same affect using a decent accelerometer?... just look at one of the iphone apps dual level (spirit level app) which is very much aware of its orientation which is turn changes the reading on the GUI.... marry this is a GUI that changes slightly on the desktop and you have a killer interface... I think they will need to marry it with a faster CPU / graphics.... threesome... foursome... take your pick - its total bigamy!!!

Besides Apple never invented capacitive touch screen, accelerometer etc... they just married it to a product...

Funny how no ones mentioned battery drainage... or that the interface implies apple with its icon layout...

but ur right.... im not expecting anything... but I just want to be entertained.... cus if they do get it it will screw up 99.5 % of the market again... making SE and nokia cry....

thats enough of my rant!

[Bokal @ Feb 24th 2009 9:10AM]

Really impressive! I especially like how they gain some screen real estate by hiding the clock and the operator...

[Neeko @ Feb 24th 2009 9:10AM]

Hopefully this will revolutionize the cellphone UI.. Imagine a Cellphone with such a interface were you could see multiple levels of screens at the same time. Kind of like a 3D Ferris wheel. Or 3D unlock codes on your phone.

[Sagaris @ Feb 24th 2009 9:12AM]

Really would like this feature, and can see a lot of uses for it.

But how would this work in a moving vehicle, as this obviously uses the accelerometer of the device?

[Michael Fan @ Feb 24th 2009 1:29PM]

omg

[chefgon_ign @ Feb 24th 2009 5:31PM]

You got to the comments section without even reading the headline? It uses eye-tracking, not the accelerometer. Front-mounted camera, like that eye-toy thing on the PS2 that could track where you moved your head.

[Berka @ Feb 24th 2009 9:28AM]

absolutely amazing

[zmanza @ Feb 24th 2009 7:45PM]

This looks AWESOME!!!!!!!!!!

[Alan Strangis @ Feb 24th 2009 9:31AM]

I love the 3D interface. TAT also showed off another UI design on the TI chip at MWC:

http://www.rgbfilter.com/?p=304

Second link down (the first is the Stantum touchscreen)

[Phillip @ Feb 24th 2009 9:41AM]

Wow, those crazy Swedes. This is going to be very cool when Apple invents it.

[Chris @ Feb 24th 2009 1:32PM]

Ha!

[felix @ Feb 24th 2009 9:45AM]

so cool, apple should talk to this guys.

[RoroCo @ Feb 24th 2009 10:33AM]

Except that Apple does not have a camera in the front of the device... I would imagine it is tracking the eye movement with a camera.

[Menos @ Feb 24th 2009 9:47AM]

insane.

[thebomberman @ Feb 24th 2009 9:59AM]

Wow - looks awesome :O

[RadicalxEdward @ Feb 24th 2009 10:12AM]

I don't get why everyones saying they're using the accelerometers. The guy is just moving the phone for the sake of the camera. It has to track eyes, so either he can move the phone back and forth to show the effect, or move his entire head back and forth to show the effect. Moving the phone just makes more sense.

[Benson @ Feb 24th 2009 8:33PM]

Moreover, he'd have to move his head _and_ the camera for it to look natural in video.

[Fredrik Berglund @ Feb 24th 2009 10:16AM]

We're not nerds!

[TT @ Feb 24th 2009 10:26AM]

I want that so badly!

[Alex @ Feb 24th 2009 10:29AM]

OH-EM-GEE!

[scape @ Feb 24th 2009 10:40AM]

i don't think it's tracking his eyes...i'll go on record to say that

[poke @ Feb 24th 2009 11:01AM]

I find it very unlikely that this uses 3d eye tracking. In this scenario it is too expensive in terms of processing power and battery life to use this method. Accelerometer or Gravity sensor makes more sense. The resolution on the front camera of phones suck anyway so computer vison techniques are not likely to work very well and be less accurate.

The guy constantly moves the phone and not the camera and in one clip not even his own head.

Good concept though to apply it to the user interface. I created the Touch Diamond VR Hologram which is basically the same concept but this takes it slightly further.

[TareX @ Feb 24th 2009 11:16AM]

1) Yeah even MIT eye-tracking isn't that smooth.
2) What if more than one eye is tracking (which could happen if u want to show off ur uber-cool 3D ui phone)?
3) It looks fucking brilliant.
4) Now I know who to blame for Android's cartoony icons.

[Lee @ Feb 24th 2009 1:17PM]

Can someone highlight a potential use for this, other than that of some sweet eye-candy?

I look at it and think its like those lenticular posters - cool, but you can't actuall *do* anything with them...

I can't actually see how this would bring any new features, or UI usability...

[Frankfurter @ Feb 24th 2009 12:19PM]

Wow, Android has a clock widget?

[Jason @ Feb 24th 2009 12:40PM]

this is so brilliant i almost wet my pants. you can make it work with a front facing camera, BUT it will also work with a rear facing camera too. the camera probably doesn't track your eyes as much as it tracks your head and body. with the rear camera, it tracks the ground and can tell if you tilted the camera by the change in the scenery.

i used to have a palm app where it used the camera to overlay images on the screen. think of space invaders, but then the aliens would be coming out of chairs and desks in the office. You could move around 360 degrees by spinning yourself around to shoot the aliens. it was an awesome concept.

and i just found the game....
http://www.clickgamer.com/moreinfo.htm?pid=7916§ion=PALM

[medios @ Feb 24th 2009 1:01PM]

This really does not feel like eye tracking, for eye tracking to work this well, you not only need a front-facing camera, but also a light(generally IR) to illuminate the retina of your eyes so that the camera has a high contrast object to track. If it were true eye tracking, they wouldn't have to move the phone in such a deliberate fashion. Yes accelerometers, especially used in this phone, aren't all that accurate, but it can at least send the general direction of movement to the UI and thus can simulate "EYE tracking". With the angles that they are using to demo it, I am not sure that an on board front facing camera would have the quality and large enough field of view to be able to track the person's eye's to that large of an angle. But, that said, I have been working on making a larger hacked together web cam and near IR light based eye-tracker to control Android menus, so I totally believe it can be done.

[Jon @ Feb 24th 2009 1:02PM]

What phone is he using to demonstrate this? Is it the iphone or what? Anybody know?

[DirtyVegas @ Feb 24th 2009 1:42PM]

I want to know what the The Wonderful Astonishing Tribes at Apple will do.

[Shank @ Feb 24th 2009 2:11PM]

I think iphone already has this,.,.,

[Zac @ Feb 24th 2009 2:15PM]

Are people reading the words "eye tracking"? Why are so many people assuming it's using an accelerometer? It's simply eye tracking in the way that a digital camera has face recognition, and it adjusts the screen image according to angle.