Sony's PS3 headtracking system does it all with a camera
By Nilay Patel 
posted Feb 27th 2008 5:04PM
posted Feb 27th 2008 5:04PM
Please note: If you're a current Engadget commenter, you do not need to re-register. Simply login and you will be prompted to select a username.
Jason Bourne'd better go save this guy's ass before he's taken out.
It doesnt pop out in the video as much as Johnny;s
**DOES require camera...doesn't require goofy IR goggles**
:) good fix - still hiring editors?
Congratulations, your article wins the award for Most Misleading Title.
You do have a point, I will give you that.
"Sony's PS3 headtracking system does it all with a camera"
Isn't that a given? I mean what is the PS3 going to use the headtracking system with besides a camera? The Blu-ray player?
When the article was first published, the headline read "Sony's PS3 headtracking system doesn't need a camera". (See the comment Jordan made above mine). It was changed post publication.
@Killer
Not too long ago, there was a dude that did the same thing using a Wii remote and some custom goggles with infrared emitters attached to the sides to emulate the Wii IR sensor. The remote is put under the TV and tracks the movement of the IR signals through space. I think they were taking that into consideration. Probably find it pretty easy on youtube.
@catapults.music
His name is Johnny Chung Lee.
Youtube: http://www.youtube.com/watch?v=Jd3-eiid-Uw
Project Page: http://www.cs.cmu.edu/~johnny/projects/wii/
Good stuff!
Hmm... Headshots, interesting FPS ideas come to mind!
BOOM! head-tracked!
How does it know which person to track?
Thats what the gnomes they keep locked up inside are for.
"How does it know which person to track?"
The Cell in the PS3 decides who's important and who isn't. So if your PS3 hates you, this might not be the best thing to get.
It's dependent on how much money you have in your wallet.
Just a fair warning to a fellow Engadgeter, don't let it see the 360.
It will track whichever face is closer to the camera. So unless someone with a HUGE head walks behind you, it shouldn't have a problem tacking.
During setup you will be asked to blink twice, turn around once, and lick your nose and then it will know it is you.
More likely the Camera picks up certain aspects of the face for detection and uses track the face after setting up who's face you want to track in the first place.
It didn't really work that well. I was surprised at how shoddy it seemed at GDC, although I must say there were a lot of people coming and going as the display was set up with the camera pointing to the entrance of the fricking expo hall. The Sony rep kept saying that it tracks your head by recognizing your facial features (not just the face by itself, but multiple faces) and if there is someone who looks like you (someone there did :) ) and they're in the frame it starts messing. I thought it started messing up whenever it caught a glimpse of anyone's head coming into the doors of the expo hall.
Overall, I was very excited about this project and even more so when I saw it was being demo'd at GDC. I was sorely disappointed with the performance of it. I do bet it will do a lot better if it is in a home environment without a lot of background distraction though.
Wawa wee woo!
This has the potential to be much more important than a Wiimote/sixaxis type of motion sensing. That's been around since the PowerGlove anyway.
You could set this up so that when you head moved at a certain speed, the character would roll or dive...crap, this is the closest thing to Virtual Reality that the mainstream has seen yet!
Think of an FPS or a boxing game: Imagine a dual input of a hand-held controller (even just a DS3) and the PSEye with this software. Amazing!
Neck injuries in real time! Realistic whiplash tracking!
Or you could just, you know, play laser tag or paintball.
Want to know why this will be a dud? Point of this is really so that, if you tilt your head right, then your character in Rainbow Six will poke his head to the right around the corner sort of thing. When was the last time Sony tried to do something like that? Oh yeah - the Sixaxis motion controls, where before, you would tilt the controller because you wanted the car to turn right just a little faster. Did those work well? Uhmmm... in the vast majority of cases, no, and in some cases it made the game much worse (see Lair).
come on guys, u are missing the implication here...
with proper software and firmware upgrades, any webcam can do this. thats a huge leap right there~ if sony can do this, no reason why microsoft, logitech, or even creative can't.
isn't this similar to what they have on some digital cameras now? It was implemented well as a webcam ability, but it's really not that revolutionary, just very well placed and fitting.
The image processing algorithms used for this are quite computer intensive. This is not true for the Wii-based ones we saw, in which the computer doesn't have to chew on color video, just on the dots seen by the IR camera. The nice thing here is that the Cell has presumably enough processing power to run these algorithms AND run the cool game with cool graphics at the same time.
My Logitech webcam does this, except it tracks my head to insert a Pirate Eyepatch over my eye. My Sony PS2 Eyetoy facilitates headtracking in the games designed for the Eyetoy. My XBox Live Cam does something similar in Pinball FX, and that cam game that I downloaded and can't remember the name of...
The benefit of the Wii Remote solution is that you're not reliant upon good lighting or good camera optics... You just have to duct-tape the sensor bar on your forehead.
@Sean D:
It has been demonstrated before using a PS EYE and some glasses with IR LEDs doing this. It can be done either way, thats the beauty of it.
Also one thing that makes this VERY special is that the PS EYE can operate at 120Hz, which means it can track motion faster then most webcams can.
There is pretty serious distinction between face tracking done here and motion tracking done in ... uh... you know... that xbla motion-tracking game.
"You could set this up so that when you head moved at a certain speed, the character would roll or dive...crap, this is the closest thing to Virtual Reality that the mainstream has seen yet!"
Great, in the future I'm not only going to have to worry about Carpal Tunnel. Now I'll have to worry about Neck Strains and Whiplash
I recently saw a Sony demo of one of their digicams that has some sort of face detection and I was blown away by its tracking speed, accuracy and consistency. It seemed light years ahead of what other manufacturers have. This is surely based on the same technology. Very cool.
It appears that the Sony of old is coming back.
Yeah, I saw this feature on their new digital cameras. Don't think it's some new fancy tech that's groundbreaking.
Cameras track pretty decent to several faces at once - I imagine with Ps3 horsepower it would track really well.
umm... from the comments i'm reading, it looks like nobody understands that its not for controlling the player... its for 3D perspective to change when you move your head.
real 3D gaming without the eyewear.
Judgement Day is inevitable
Is that what you goons are calling WGA these days?
Wow. That is quite possibly the worst camera work ever. Maybe I'm being picky... it's not like I watched the video to see something in 3D or anything, silly me.
That's perfectly allright, you're not picky. Don't worry about it. But you are a douchebag though.
Well.. Maybe a wee bit picky too. That is pure genius technology right there man, take it in and
think about the possibilities. I can't wait to see FPS and horror games that uses this.
People on here are totally missing the point. Yes, head/face tracking isn't new technology, as they have been using it in cameras for years. What is unique about this technology is that is uses the facetracking to create a 3d gaming experience.
Douchebag is a bit harsh, unless you're talking about the camera guy.
"IF you shoot over my shoulder you might be able to see better"
** Camera guy moves and immediately zooms into the screen, cutting the guy out of frame.
/facepalm
I got to try this out at GDC last week... I was thoroughly unimpressed with the demo. Despite intending to be a direct ripoff of Johnny Lee, the thing left no impression on me of 3d movement like Johnny Lee's demo did. The targets did not change size, and I'm not sure if the thing could even track movement too and from the screen. It did fairly well at side to side and up and down.
video not accessible in Canada? FTW? Irregardless Nintendo has got them beat with a game ready to go.
head tracking built into BOOM BOX http://gonintendo.com/?p=36198
get us a working vid please?
Apparently it only works if its looking at you in a mirror.
my GE camera can already do this...and its been out since the beginning of last year. why is everyone so impressed with this?
okay so they got head tracking, whoopdeedoo.
the headgear is for orienting the display shown on screen, are they going to incorporate "SMILE DETECTION" next so your PS3 knows how depressed you are?
Back before the 360 was released, there was a news article on MSNBC which stated that Sony had patented technology which could induce sensational experiences (like making you smell pie when it really wasn't there, etc). This head-tracking technology seems like a logical step forward towards this technology. I can't wait for this to revolutionize First-Person Shooters.
All the Sony Digital Cameras come with the FACE DETECTION system... a PS3 firmware update soon to add that feature to PS3 ? maybe.
I'd gladly wear IR goggles at half the price using a Wii over a PS3. I did get to see Sony's headtracking setup in action at GDC though, pretty cool stuff. Although it did seem like it was a little less responsive than Johnny Lee's system. That and the guys at the Sony booth totally shrugged me off for some chick.. so I didn't actually get to try it out myself.
I can't wait for this to be common in games. Ignoring the imersiveness that it brings at a basic level (as seen in the youTube videos), how fun would it be to be able to peek around corners in First Person Shooters (ie.COD and Haze).
Perhaps the LED lights on the controllers being used to match up faces to player numbers. They could just flash them in a particular sequence. maybe.
wow, that is really awesome, i cant wait to look around corners like in real life. would make for a very interesting and realistic fps.
I was at GDC this year, and I tried this very demo and I have to say it was awful. The Wii version truly made you feel like you were looking into a window. Sony's version was simply a way to move the scene left and right and had no depth what-so-ever. Not only that, it was a direct rip-off from Johnny Lee demo with the targets. I was very disappointed.
algorithms need to be optimized to go BEYOND face tracking - how about T zone eye tracking?
the math required is horrendous for a video feed, and grabbing some headgear for a FPS game would slightly lend itself to the immersion experience.
remove headgear to save and exit game.
I agree with skulldriveshaft... and what a waste of CPU cycles... an inefficient tracking for a decent FPS except perhaps for "Dora the Explorer" and stuff like that.
hmm, so people leave their ps3 cameras on, connected to the internet, on a system that's the exact same everywhere so only one hack would be needed, and it's also connected to sony 'maker of rootkits'.
Draw your own conclusions.
This can potentially add a lot to the gameplay!
This is not much more than the tech in the current gen of Sony Cybershot cams, $299 at a store near you. Face recognition / auto focus. They added their eyetoy motion tracking code to the face auto-focus code. Voila, face tracking.
Camera based tracking system is very dependent on the environment - lighting, noise, other people in the room, etc. IR glasses don't have to be goofy looking at all, in fact, they don't have to be glasses at all - why not attach IR emitters to the same headphones you use when you're playing with mic on - not exactly the slickest looking things on the street but nobody has trouble with them. And if you don't play with headphones then the IR emitters can have a bluetooth headset-like form factor. Cheaper, more effective, more reliable - if games start taking advantage of head tracking technology I think IR emitters is the way to go.
These guys have a downloadable demo for the PC using a Quickcam:
http://www.kuubee.com
This system is pure crap, seriously.
I mean the Eyetoy worked brilliant .... in carefully controlled conditions. In my house however its next to useless as it picks up reflections off all the furniture and I do not have a big enough house to create an "Eyetoy safe room".
So yeah, in theory all these things can be done. But the REASON why it has to be done with silly glasses/headware is to make it work in ANY house regardless of what junk might otherwise confuse a camera.