IBM's SiSi virtually translates speech to sign language

We've seen a wide array of devices designed to help the deaf communicate and experience life more fully, and IBM is hoping to make yet another advancement in the field with its SiSi (Say It Sign It) system. Developed at an IBM research center in Hursley, England, the technology works "by using speech recognition to convert a conversation into text," after which SiSi "translates the text into the gestures used in sign language and animates a customizable avatar that carries them out." Currently, the system is still labeled a prototype and only works with British sign language, but there's already plans to commercialize the invention in due time. For a better look at exactly what SiSi can do, take a peek at the video demonstration waiting after the jump.

[HineyWipe @ Sep 13th 2007 2:14PM]

SiSi ??? Come on. Ya Big Sissy! What? What you say?

[jaden82 @ Sep 13th 2007 2:15PM]

Oh man...this would make a great technology for the deaf people. I'm deaf and I depend on technology such as computers and smartphones to help me communicate with the rest of the world. However, I couldn't do anything about having actual conversations with people who can hear. If I could fit this on my smartphone or something similar, it'd make my life a lot easier if it actually works and relays what other people are saying to me on the fly.

[Paul @ Sep 13th 2007 4:20PM]

I think it would, but I also think it would be time better spent further improving Cochlear Implant technology. My mom has been deaf all her life and after dual cochlear implants can hear very well. But these sorts of things take time, to get them down in size, down in cost, and not require as much time to get the sound mapping work smoothly.

[Dan @ Sep 13th 2007 4:40PM]

Surely the Wii technology, refined, would allow sign language users to 'speak' to hearers. Of course the gadget would be split up into two gloves.

[John @ Sep 13th 2007 5:54PM]

@Paul

Cochlear implants involve a completely invasive and dangerous procedure that, no matter how advanced the technology becomes, is not the magic panacea for all forms of Deafness. Instead of helping only one group of the Deaf community that some researchers deem "curable," this technology helps the whole Deaf community; definitely time well-spent.

[blade417 @ Sep 13th 2007 2:18PM]

What you talkin bout' Willis!?

[oshean @ Sep 13th 2007 2:30PM]

Yes, but can it translate cuss words?

[Todd @ Sep 13th 2007 2:31PM]

Why take the second step? Sign language was developed because there was no means to convert speech into viewable text in real time. But if the machine can show the text of what is being said, as its being said, what is the point of reverting it to gestures. People with hearing impairments can read text.

[Gav @ Sep 13th 2007 2:43PM]

My thoughts exactly. Also, I'm sure the vocabulary avaliable through text is far better than that in sign language.

[Mike @ Sep 13th 2007 2:50PM]

That depends on which sign language you're talking about.

English Sign Language? No. They have the same vocabulary.

American Sign Language (which they wouldn't dare use for this purpose)? Also no. While ASL's finite vocabulary is, in fact, somewhat limited, other features of the language allow for the creation of words far more easily than any other language. ASL can be infinite in it's use and vocabulary.

Still, I agree with the parent poster. Converting speech-to-text-to-sign seems like a convoluted step.

So odd.

[oshean @ Sep 13th 2007 2:58PM]

I personally think that virtual reality would be more immersive without text balloons floating around.

[Froggy @ Sep 13th 2007 4:04PM]

maybe because if you display it on a huge screen, you're more likely to see the gestures then to be able to read the text. still silly. and the name?... oh God.

[otto @ Sep 13th 2007 3:43PM]

Anyone else think that guy in the pic is a SiSi?

[beyond vision @ Sep 13th 2007 4:01PM]

there goes a few jobs

[rrandrews @ Sep 13th 2007 4:15PM]

I'm confused as well. How is the signing part of this some kind of new technology? Certainly speech recognition could use a lot of refining, but once you've got that down it doesn't seem like there's anything tricky to getting a 3D model to follow a script.

[Typhoid Mary @ Sep 13th 2007 4:53PM]

Now they just have to work it the other way around and I'll be impressed. Lets see them do sign to speech!

[Philippe @ Sep 14th 2007 3:13AM]

Hi, Sign-To-Text (i.e. Video-to-Text) is what we try to do for example at the RWTH Aachen University in Germany, and Text-to-Speech synthesis software is already available.

Have a look at the video here:
http://www-i6.informatik.rwth-aachen.de/~dreuw/research.html
http://www-i6.informatik.rwth-aachen.de/~dreuw/download/021.avi

[Gunnar @ Sep 13th 2007 5:39PM]

it´s OK

[John @ Sep 13th 2007 5:43PM]

I could see this being a big hit in the Deaf community. Technology has been a double-edged sword for the Deaf, since concepts like cochlear implants have long tried to "convert" a Deaf person into a hearing person. This technology embraces sign language as its own language (which it is, sign language has its own vocabulary and grammar that are substantially different from spoken languages), and allows a Deaf person to interact in his/her native language.

There are a variety of applications for this technology:

-Hospitals without on-site interpreters (especially at night)
-Interpretation for children who have not learned to read
-Online classes given by a hearing instructor where there can be no interpreter present

Overall, while I can't see this being a replacement for an interpreter (much like Altavista translations won't ever power a UN conference), I can see it as a positive step forward in recognizing that sign languages are not simply bastardized imitations of spoken languages, which will hopefully help further the dissolution of the Deaf and hearing worlds.

[Jenny @ Sep 14th 2007 1:25PM]

I would have to agree with you in that this won't (just yet) be a replacement for live interpreters or VRS (video relay service). So much of sign language is dependent on emotional gestures such as frowning or smiling and also large gestures for emphasis, etc. I'm a telecom (voice-text) relay operator and even I can see how the slightest inflections in my voice can help or hurt a conversation.

Until the technology can "understand" the speaker's intended emotion, live interpreters do not have to fear losing their jobs. Plus, there's something to be said about the human connection within the deaf community.

[Samsam @ Sep 13th 2007 6:44PM]

Hopefully they did a better job than Windows Vista speech recognition.

[JoeSmack @ Sep 13th 2007 9:35PM]

notice the lack of affect, expression, eyebrow movements, lip movements, etc from the electronic model? did you know all these things are extremely important to sign language? imagine a hockey-1992-robot-voice reading to you what someone over the phone was saying. it would be worse than that in this case for signers. ick.

besides this, sign languages involve a ton of role shifting, classifiers and flexibility. this technology would be severely limited in what it could say, and definitely how it could say it.

[indigo @ Sep 14th 2007 6:01PM]

I've seen this avatar working before and I know that it's capable of a great deal of facial and other bodily movement, so I guess it's really a case of improving the quality of the signs - the avatar's capable of a lot.

The other thing is that British Sign Language (I believe unlike ASL) has its own syntax and grammar, making it sometimes very difficult for a deaf person to comprehend written/spoken English, and vice versa - I've read a story written by someone in BSL that, when read, made no sense to me - but when signed, was much more readable, so that's why not just speech-to-text is the easiest thing.

[Kim @ Sep 14th 2007 11:46PM]

ASL, like BSL and other signed languages, also has its own syntax and grammar. Much more like Japanese than English, except in 3D.

[Mike @ Sep 16th 2007 10:32AM]

What the article doesn´t mention is that this technology was developed by 4 students during the 12 week Extreme Blue internship program. I've seen and played with the system at the expo in Hursley this past week, and it is very impressive indeed!

[Deaf258 @ Sep 18th 2007 1:34PM]

This is just another version of a "hearing aid" for hearies to speak to the Deaf, but how can the Deaf communicate back with the hearies??? This is not a great technology until it allows and opens up a 2-way street of communication!

[Ricardo Carvalho @ Nov 21st 2007 3:54PM]

It's great to see the SiSi technology. We work in a solution called "Rybena" (www.rybena.org.br). It's really a recursive question about the need to convert signs to text : if the deaf can read how should it be usefull convert voice-to-signs-and-text ? Normally the deaf native laguage does not have all the vocabulary and the conversion to text will give more understanding to the deaf reader in a special translated text format. We already convert text to signal language called LIBRAS (LIBRAS - is a signal language specific for Brasil), if we connect with SiSi we can convert the text to LIBRAS. I would like to contact someone that is working in this IBM product.