Voice to text translation for the deaf

[StevePoulsen]

I am amazed at the accuracy with which Google Voice turns my voice into text in emails, notes, and other documents. It is not perfect, and the original voice needs to be clear and accurate.

Perhaps someone can figure out how to wire a hearing impaired receiver to Google Voice - that way the sound is "direct" and as clear as you can get it.

I have heard of members doing this, with good (not perfect) results.

Also, depnding ont he hearing aids you have (they need a T-coil) you can add a neck loop to the receiver so that your aids will pickup the signal from the ASL system.

[JonSevy]

...one dear sister sits beside me in all the meetings and writes quickly brief phrases about what the speaker is saying. She is adept at summarizing. It is a life-saver! I feel so grateful to her! Feel a part of the lessons and talks.

Tweet for deaf AND other translations.

Great discussion. We recently had a couple move in, both totally deaf. Active in Arizona, fell away in their last ward d/t no translation. I pulled out my Android cell phone and started dictating into "Memo". Very accurate, and large enough to read, but they had severely limited comprehension. That puzzled me.

My sister in law is very skilled in ASL. She said signing is Pidgeon: simple, Tweet-like phrases, basic words. Normal grammar is to hard them to follow, and too hard to type or read. So a key seems to be . . . talk like to foreigner who no speak English. Type like Tweet. No points grammar spelling.

Chilean sister, too. New our ward. Need translate Spanish. Me think full-screen lap-top or cell phone app, show summary AND 2nd language, serve both deaf and no - [English] speakers. Slow type OK. Quiet mic OK. Big screen front of room OK.

[daveyjones]

I'm a software developer currently working on an automated LDS captioning system. I have approval from our stake to test the software in our ward and I'm hoping to have the system up and running in the next few weeks. If it goes well, I'll certainly share with those interested.

There's actually a very good chance that automated captioning could work well in a Sacrament Meeting setting—the sound could be piped directly into the captioning device (e.g. a tablet), so there would be very little noise, and a custom language model could be developed around LDS language.

Speech recognition relies on statistical language models, and those models are developed using a corpus—a body of texts intended to represent normal speech patterns. The problem is, LDS language is quite unique; not only do we have a massive set of unique pronouns in the Book of Mormon, but we also have unique speech patterns (e.g. the words "bear" and "testimony" aren't used together very often in everyday speech, so a generic language model would find it statistically unlikely for those words to be near each other, but an LDS-specific language model would find it very likely). If only there was a massive corpus of LDS language...

Using the thousands of General Conference talks that are freely available, along with the standard works, I believe a very good LDS language model can be developed. The plan is to develop on iOS, so you would plug the house sound directly into the tablet and then hook a tablet up to a TV, which in most chapels could be placed opposite the sacrament table (down low so that only the first couple rows can see the captions, so they don't become a distraction for everyone else).

I'll keep you updated!

[drepouille]

It is easy to use the ComTek receivers to inject audio from the house sound system into a laptop/tablet. I tested it, and it does work. However, the free speech recognition web site I was using was too slow, and our DSL connection was too slow, and I was sharing the Wi-Fi with everyone else in the meetinghouse. When you add all those problems to the unique vocabulary you noted, and the fast and rambling speech we often hear, it all makes the task quite daunting.

In reality, people can't read as fast as they can listen. So if your speech-to-text conversion was absolutely perfect, the deaf reader could not possibly keep up. That is why most captions are simplified, and omit a lot of "verbal noise" that is not necessary for the conversion. That is why we just have a real person type a simplified version of what he hears on a laptop, so the deaf person can read and keep up with the gist of the talk or lesson.

[russellhltn]

In reality, people can't read as fast as they can listen.

Not quite true. The average person reads about twice as fast as the average speaker. In some cases, three times as fast. However, for some deaf people, their first language is ASL (which like many languages has a different word order), so they are more like English as a second language. It all depends on when they lost their hearing.

[adamlcalvert]

I'm a software developer currently working on an automated LDS captioning system. I have approval from our stake to test the software in our ward and I'm hoping to have the system up and running in the next few weeks. If it goes well, I'll certainly share with those interested.

We have several wards in our Stake that are facing the same challenge and looking into dictation solutions. It would be great to see a LDS captioning system. Would it be possible for us to be involved with testing your system?

Have you considered how you would support classroom dictation? Would it be possible to set up multiple microphones around the room and pick up a classroom discussion?

[smc84]

My stake would also be interested in this software. If not for helping the deaf, but also for helping transcribe Patriarchal blessings. Our Patriarch's scribe hand is going numb from so much typing.

[daveyjones]

I just wanted to provide an update on this. Our system has been set up for a couple months now and is working even better than expected. We have two deaf members and several hard-of-hearing members who use the system every week. Not once have I heard any concerns about not being able to read fast enough to keep up with the speakers.

Members in our ward who want to use the system simply connect their tablet up to a special WiFi network called LDSCaptions (as opposed to LDSAccess). Once connected to this network, all they have to do is visit ldscaptions.org in their browser. The captions will then begin streaming to their tablet.

For those interested in the technical side, we installed a wireless router in the chapel to create the LDSCaptions network. We also have a Node app running on a MacBook Pro, which is connected up to the house sound. The Node app interfaces with the IBM Watson Speech to Text engine and delivers the captions via WebSockets to any browser connected to the correct port on the same network. (When users visit ldscaptions.org, it simply redirects them to the local server running on the MacBook Pro). Currently, I'm working on getting all of this tech into a single 5x7-inch box—so all you'd have to do is plug the sound and network cable into the box and then you're good to go (you wouldn't even need WiFi in the room since the box would act as the wireless router).

In terms of accuracy, I've only formally measured the word error rate (WER) on one short talk—the WER came out to be around 5% (near-human-level accuracy). This means 95% of the words in the transcription were transcribed correctly, while roughly 5% were either added, omitted, or misrecognized. Note that the exceptional WER is due in part to the custom language model, which includes all conference talks since 1980, the entire standard works, and our ward directory (this could be customized for each ward).

If anyone would like to discuss further just shoot me an email or text—you can find my contact info at dillpixel.com.

[daveyjones]

Have you considered how you would support classroom dictation? Would it be possible to set up multiple microphones around the room and pick up a classroom discussion?

Yes, most STT engines (including IBM's) will identify speakers, so you can code the application to use different colors for each speaker and even reuse the same color when the same person makes multiple comments. I'm planning to work on this software feature after I consolidate the hardware.

[sbradshaw]

My stake would also be interested in this software. If not for helping the deaf, but also for helping transcribe Patriarchal blessings. Our Patriarch's scribe hand is going numb from so much typing.

Are you able to call more than one scribe?