Sony's PS3 headtracking system does it all with a camera

We were already blown away by Johnny Lee's headtracking demo using a Wiimote, but it looks like Sony's trying to take it mainstream in a much slicker way. The company was demoing a PS3-based headtracking system at GDC that doesn't require any special IR glasses to function -- it just uses a bone-stock Playstation Eye camera to track your ugly mug around the room. Sony says it's not announcing the system for any games yet, but we can't imagine devs aren't clamoring for this to get official -- this would be amazing for all sorts of genres. Video after the break.

[phanbouy @ Feb 27th 2008 5:08PM]

Jason Bourne'd better go save this guy's ass before he's taken out.

[Jordan @ Feb 27th 2008 5:09PM]

**DOES require camera...doesn't require goofy IR goggles**

[Jordan @ Feb 27th 2008 5:15PM]

:) good fix - still hiring editors?

[AlexL @ Feb 27th 2008 5:12PM]

Congratulations, your article wins the award for Most Misleading Title.

[Killer @ Feb 27th 2008 6:02PM]

You do have a point, I will give you that.

"Sony's PS3 headtracking system does it all with a camera"

Isn't that a given? I mean what is the PS3 going to use the headtracking system with besides a camera? The Blu-ray player?

[AlexL @ Feb 27th 2008 6:08PM]

When the article was first published, the headline read "Sony's PS3 headtracking system doesn't need a camera". (See the comment Jordan made above mine). It was changed post publication.

[catapults.music @ Feb 27th 2008 10:09PM]

@Killer

Not too long ago, there was a dude that did the same thing using a Wii remote and some custom goggles with infrared emitters attached to the sides to emulate the Wii IR sensor. The remote is put under the TV and tracks the movement of the IR signals through space. I think they were taking that into consideration. Probably find it pretty easy on youtube.

[numlok @ Feb 28th 2008 1:54AM]

@catapults.music

His name is Johnny Chung Lee.
Youtube: http://www.youtube.com/watch?v=Jd3-eiid-Uw
Project Page: http://www.cs.cmu.edu/~johnny/projects/wii/

Good stuff!

[linumax @ Feb 27th 2008 5:18PM]

Hmm... Headshots, interesting FPS ideas come to mind!

[Jordan @ Feb 27th 2008 5:20PM]

BOOM! head-tracked!

[Sean @ Feb 27th 2008 5:26PM]

How does it know which person to track?

[Jordan @ Feb 27th 2008 5:33PM]

Thats what the gnomes they keep locked up inside are for.

[Killer @ Feb 27th 2008 5:37PM]

"How does it know which person to track?"

The Cell in the PS3 decides who's important and who isn't. So if your PS3 hates you, this might not be the best thing to get.

[Jordan @ Feb 27th 2008 5:54PM]

It's dependent on how much money you have in your wallet.

[Reader @ Feb 27th 2008 6:24PM]

Just a fair warning to a fellow Engadgeter, don't let it see the 360.

[cduran01 @ Feb 27th 2008 9:50PM]

It will track whichever face is closer to the camera. So unless someone with a HUGE head walks behind you, it shouldn't have a problem tacking.

[BladeSling @ Feb 28th 2008 4:35AM]

During setup you will be asked to blink twice, turn around once, and lick your nose and then it will know it is you.

More likely the Camera picks up certain aspects of the face for detection and uses track the face after setting up who's face you want to track in the first place.

[Neonic @ Feb 29th 2008 2:00AM]

It didn't really work that well. I was surprised at how shoddy it seemed at GDC, although I must say there were a lot of people coming and going as the display was set up with the camera pointing to the entrance of the fricking expo hall. The Sony rep kept saying that it tracks your head by recognizing your facial features (not just the face by itself, but multiple faces) and if there is someone who looks like you (someone there did :) ) and they're in the frame it starts messing. I thought it started messing up whenever it caught a glimpse of anyone's head coming into the doors of the expo hall.

Overall, I was very excited about this project and even more so when I saw it was being demo'd at GDC. I was sorely disappointed with the performance of it. I do bet it will do a lot better if it is in a home environment without a lot of background distraction though.

[Nate the Prophet @ Feb 27th 2008 5:34PM]

Wawa wee woo!

This has the potential to be much more important than a Wiimote/sixaxis type of motion sensing. That's been around since the PowerGlove anyway.

You could set this up so that when you head moved at a certain speed, the character would roll or dive...crap, this is the closest thing to Virtual Reality that the mainstream has seen yet!

Think of an FPS or a boxing game: Imagine a dual input of a hand-held controller (even just a DS3) and the PSEye with this software. Amazing!

[engine @ Feb 27th 2008 5:42PM]

Neck injuries in real time! Realistic whiplash tracking!

[wickedpheonix @ Feb 27th 2008 5:49PM]

Or you could just, you know, play laser tag or paintball.

Want to know why this will be a dud? Point of this is really so that, if you tilt your head right, then your character in Rainbow Six will poke his head to the right around the corner sort of thing. When was the last time Sony tried to do something like that? Oh yeah - the Sixaxis motion controls, where before, you would tilt the controller because you wanted the car to turn right just a little faster. Did those work well? Uhmmm... in the vast majority of cases, no, and in some cases it made the game much worse (see Lair).

[ingus @ Feb 27th 2008 5:46PM]

come on guys, u are missing the implication here...

with proper software and firmware upgrades, any webcam can do this. thats a huge leap right there~ if sony can do this, no reason why microsoft, logitech, or even creative can't.

[Killer @ Feb 27th 2008 5:57PM]

isn't this similar to what they have on some digital cameras now? It was implemented well as a webcam ability, but it's really not that revolutionary, just very well placed and fitting.

[Marcos @ Feb 27th 2008 6:33PM]

The image processing algorithms used for this are quite computer intensive. This is not true for the Wii-based ones we saw, in which the computer doesn't have to chew on color video, just on the dots seen by the IR camera. The nice thing here is that the Cell has presumably enough processing power to run these algorithms AND run the cool game with cool graphics at the same time.

[Sean D. @ Feb 27th 2008 7:00PM]

My Logitech webcam does this, except it tracks my head to insert a Pirate Eyepatch over my eye. My Sony PS2 Eyetoy facilitates headtracking in the games designed for the Eyetoy. My XBox Live Cam does something similar in Pinball FX, and that cam game that I downloaded and can't remember the name of...

The benefit of the Wii Remote solution is that you're not reliant upon good lighting or good camera optics... You just have to duct-tape the sensor bar on your forehead.

[cduran01 @ Feb 27th 2008 9:59PM]

@Sean D:

It has been demonstrated before using a PS EYE and some glasses with IR LEDs doing this. It can be done either way, thats the beauty of it.

Also one thing that makes this VERY special is that the PS EYE can operate at 120Hz, which means it can track motion faster then most webcams can.

[Mojo @ Feb 28th 2008 11:22AM]

There is pretty serious distinction between face tracking done here and motion tracking done in ... uh... you know... that xbla motion-tracking game.

[Killer @ Feb 27th 2008 5:51PM]

"You could set this up so that when you head moved at a certain speed, the character would roll or dive...crap, this is the closest thing to Virtual Reality that the mainstream has seen yet!"

Great, in the future I'm not only going to have to worry about Carpal Tunnel. Now I'll have to worry about Neck Strains and Whiplash

[Rob @ Feb 27th 2008 5:58PM]

I recently saw a Sony demo of one of their digicams that has some sort of face detection and I was blown away by its tracking speed, accuracy and consistency. It seemed light years ahead of what other manufacturers have. This is surely based on the same technology. Very cool.

It appears that the Sony of old is coming back.

[CharlieX @ Feb 27th 2008 6:01PM]

Yeah, I saw this feature on their new digital cameras. Don't think it's some new fancy tech that's groundbreaking.

Cameras track pretty decent to several faces at once - I imagine with Ps3 horsepower it would track really well.

[will clifton @ Feb 27th 2008 6:04PM]

umm... from the comments i'm reading, it looks like nobody understands that its not for controlling the player... its for 3D perspective to change when you move your head.

real 3D gaming without the eyewear.

[Bernhard @ Feb 27th 2008 6:13PM]

Judgement Day is inevitable

[phanbouy @ Feb 27th 2008 6:47PM]

Is that what you goons are calling WGA these days?

[Just Antother Film Guy @ Feb 27th 2008 6:19PM]

Wow. That is quite possibly the worst camera work ever. Maybe I'm being picky... it's not like I watched the video to see something in 3D or anything, silly me.

[sdh @ Feb 27th 2008 7:00PM]

That's perfectly allright, you're not picky. Don't worry about it. But you are a douchebag though.
Well.. Maybe a wee bit picky too. That is pure genius technology right there man, take it in and
think about the possibilities. I can't wait to see FPS and horror games that uses this.

[Brett @ Feb 27th 2008 7:29PM]

People on here are totally missing the point. Yes, head/face tracking isn't new technology, as they have been using it in cameras for years. What is unique about this technology is that is uses the facetracking to create a 3d gaming experience.

[ck @ Feb 27th 2008 7:25PM]

Douchebag is a bit harsh, unless you're talking about the camera guy.

"IF you shoot over my shoulder you might be able to see better"

** Camera guy moves and immediately zooms into the screen, cutting the guy out of frame.

/facepalm

[SOOPERGOOMAN @ Feb 27th 2008 6:42PM]

video not accessible in Canada? FTW? Irregardless Nintendo has got them beat with a game ready to go.
head tracking built into BOOM BOX http://gonintendo.com/?p=36198

get us a working vid please?

[Bass Akwards @ Feb 27th 2008 7:06PM]

Apparently it only works if its looking at you in a mirror.

[skulldriveshaft @ Feb 27th 2008 7:32PM]

okay so they got head tracking, whoopdeedoo.

the headgear is for orienting the display shown on screen, are they going to incorporate "SMILE DETECTION" next so your PS3 knows how depressed you are?

[Alex Padilla @ Feb 27th 2008 7:56PM]

Back before the 360 was released, there was a news article on MSNBC which stated that Sony had patented technology which could induce sensational experiences (like making you smell pie when it really wasn't there, etc). This head-tracking technology seems like a logical step forward towards this technology. I can't wait for this to revolutionize First-Person Shooters.

[holycow @ Feb 27th 2008 7:57PM]

All the Sony Digital Cameras come with the FACE DETECTION system... a PS3 firmware update soon to add that feature to PS3 ? maybe.

[LoTekJunky @ Feb 27th 2008 8:45PM]

I got to try this out at GDC last week... I was thoroughly unimpressed with the demo. Despite intending to be a direct ripoff of Johnny Lee, the thing left no impression on me of 3d movement like Johnny Lee's demo did. The targets did not change size, and I'm not sure if the thing could even track movement too and from the screen. It did fairly well at side to side and up and down.

[Josh Rose @ Feb 27th 2008 9:53PM]

I'd gladly wear IR goggles at half the price using a Wii over a PS3. I did get to see Sony's headtracking setup in action at GDC though, pretty cool stuff. Although it did seem like it was a little less responsive than Johnny Lee's system. That and the guys at the Sony booth totally shrugged me off for some chick.. so I didn't actually get to try it out myself.

[Jona @ Feb 27th 2008 9:59PM]

I can't wait for this to be common in games. Ignoring the imersiveness that it brings at a basic level (as seen in the youTube videos), how fun would it be to be able to peek around corners in First Person Shooters (ie.COD and Haze).

Perhaps the LED lights on the controllers being used to match up faces to player numbers. They could just flash them in a particular sequence. maybe.

[Alex M @ Feb 27th 2008 10:09PM]

my GE camera can already do this...and its been out since the beginning of last year. why is everyone so impressed with this?

[Motoken @ Feb 27th 2008 10:18PM]

wow, that is really awesome, i cant wait to look around corners like in real life. would make for a very interesting and realistic fps.

[Jon Rick @ Feb 27th 2008 10:19PM]

I was at GDC this year, and I tried this very demo and I have to say it was awful. The Wii version truly made you feel like you were looking into a window. Sony's version was simply a way to move the scene left and right and had no depth what-so-ever. Not only that, it was a direct rip-off from Johnny Lee demo with the targets. I was very disappointed.

[skulldriveshaft @ Feb 27th 2008 11:42PM]

algorithms need to be optimized to go BEYOND face tracking - how about T zone eye tracking?

the math required is horrendous for a video feed, and grabbing some headgear for a FPS game would slightly lend itself to the immersion experience.

remove headgear to save and exit game.

[Pixi @ Feb 28th 2008 3:21AM]

I agree with skulldriveshaft... and what a waste of CPU cycles... an inefficient tracking for a decent FPS except perhaps for "Dora the Explorer" and stuff like that.