Voice to text translation for the deaf

[GGTTLJB]

hi my name is Terry and I just read your article about you needing some kind of program for transcribing typist I didn't quite understand it but maybe this'll help you out Google has a free program that's used for translating two different languages from English to to any language you want but what it does is as you speak it puts the words down in print. maybe somehow this could help you or put you in the right direction or I don't know but I thought I'd give it a shot good luck to you and I hope it helps have a great day Terri Holmes Sacramento California User I'd: GGTTLJB. END

[lfanelli]

Creating a Speech to Text ‘Server’ on MacOS 10.12

All credit goes to @daveyjones for his code and help. I just made the tutorial. I use the Focusrite Scarlett 2i2 USB Audio Interface for the connection from the Aux line out of the chapel to the MacBook Pro.

SERVER-
Get an IBM Dev Account - Standard – https://console.bluemix.net/catalog/ser ... ch-to-text
In Terminal install Homebrew -

Code: Select all

 usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" 

Install node and npm2 via homebrew -

Code: Select all

$ brew install node

Install the ‘mic module’ on MacOS using

Code: Select all

brew install sox 

or Linux

Code: Select all

 brew install arecord 

Using the following directory scheme, create the following:

/usr/local/bin/
|----public
|----|+-- index.html
|----|+-- server.js
|----|+-- package.json
|----|---- scripts
|----|----+-- main.js
|----|---- styles
|----|----+-- main.css


Create the text file named ‘Server.js’
[Note: you will need to use YOUR username and password from IBM’s dev center below]

Code: Select all

var app = require("express")();
var events = require("events");
var server = require("http").Server(app);
var mic = require("mic");
var os = require("os");
var io = require("socket.io")(server);
var watson = require("watson-developer-cloud/speech-to-text/v1");
var wav = require("wav");
 
events.EventEmitter.prototype._maxListeners = 100;
 
//====================================================================================================
// Watson
//====================================================================================================
 
var speechToText = new watson({
        	url: "https://stream.watsonplatform.net/speech-to-text/api",
        	username: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx",
        	password: "xxxxxxxx"
});
 
var micInstance = mic({rate: "44100", channels: "1", debug: false});
var micInputStream = micInstance.getAudioStream();
var wavStream = new wav.Writer({sampleRate: 44100, channels: 1});
 
var recognizeStream = speechToText.recognizeUsingWebSocket({
        	content_type: "audio/wav",
        	//customization_id: "##########",
        	interim_results: true,
        	objectMode: true,
        	inactivity_timeout: 3600
});
 
 
var cleanTranscript = (data) => {
        	transcript = data.results[0].alternatives[0].transcript;
        	// Remove hesitation indicators
        	transcript = transcript.replace(/\s*\%HESITATION\s*/g, " ");
        	// Remove supposed profanity
        	transcript = transcript.replace(/\*/g, "");
        	// Remove extra spaces
        	transcript = transcript.replace(/ +/g, " ").trim();
        	// Return the cleaned transcript
        	return transcript;
}
 
micInputStream.pipe(wavStream);
wavStream.pipe(recognizeStream);
micInstance.start();
 
//====================================================================================================
// Server
//====================================================================================================
 
app.get("/", function (req, res) {
        	res.sendFile(__dirname + "/public/index.html");
});
 
app.get("/scripts/main.js", function (req, res) {
        	res.sendFile(__dirname + "/public/scripts/main.js");
});
 
app.get("/styles/main.css", function (req, res) {
        	res.sendFile(__dirname + "/public/styles/main.css");
});
 
io.on("connection", function (socket) {
        	recognizeStream.on("data", function(data) {
                    	var transcript = cleanTranscript(data);
                    	if (transcript.length > 0) {
                                	socket.emit("data", {
                                            	final: data.results[0].final,
                                            	transcript: cleanTranscript(data)
                                	});
                    	}
        	});
});
 
server.listen(8000);

Create the text file called package.json

Code: Select all

{  "name": "local-server",
  "version": "1.0.0",
  "description": "",
  "main": "server.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "express": "^4.16.3",
    "mic": "^2.1.2",
    "socket.io": "^2.1.1",
    "watson-developer-cloud": "^3.5.0",
    "wav": "^1.0.2"
  }
}

Create file called index.html

Code: Select all

<!DOCTYPE html>
<html>
<head>
        	<meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no">
        	<meta name="apple-mobile-web-app-capable" content="yes">
        	<title>LDS Captions</title>
        	<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Libre+Baskerville">
        	<link rel="stylesheet" href="/styles/main.css">
        	<script src="https://cdnjs.cloudflare.com/ajax/libs/socket.io/2.1.1/socket.io.js"></script>
        	<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
        	<script src="/scripts/main.js?v=2"></script>
</head>
<body>
        	<div id="content">
                    	<span id="t0"></span>
        	</div>
</body>
</html>

Create a folder ‘scripts’ and text file called main.js
[Note: change ‘localhost’ with the IP of the server]

Code: Select all

var socket = io("http://localhost:8000");
var i = 0;
socket.on("data", function (data) {
        	$("#t" + i).text(data.transcript);
        	if (data.final) {
                    	var words = $("#t" + i).text().split(" ");
                    	if (words.length < 2) {
                                	$("#t" + i).prev(".dot").remove();
                                	$("#t" + i).remove();
                    	}
                    	i++;
                    	$("#content").append(` <span class="dot">&bull;</span> <span id="t${i}"></span>`);
        	}
});
 
setInterval(function() {
        	$("#content").scrollTop($("#content").scrollTop() + 1);
}, 10);

Create a folder 'styles’ and text file called main.css

Code: Select all

body {
        	background: black;
}
 
#content {
        	position: fixed;
        	top: 48px;
        	right: 48px;
        	bottom: 48px;
        	left: 48px;
        	line-height: 1.75em;
        	font-family: "Libre Baskerville";
        	font-size: 48px;
        	color: white;
        	overflow: hidden;
}
 
.dot {
        	font-size: 48px;
        	opacity: 0.4;
}
 
.dot:first-child {
        	display: none;
}

Now to change the default location of apache’s Document Root Directory location from /Library/WebServer/Documents to our new location of /usr/local/bin/public

Open Terminal and type

Code: Select all

sudo nano /etc/apache2/httpd.conf

and press enter.
Press Ctrl+W which will bring up a search.
Search for /Library/WebServer/Documents and press enter.
Change the two (2) instances of /Library/WebServer/Documents to /usr/local/bin/public
Press Ctrl+O followed by Enter to save the change you just made.
Press Ctrl+X to exit nano.

Type

Code: Select all

sudo apachectl restart

and press enter.

Type

Code: Select all

cd /usr/local/bin/public/

and press enter.

Type

Code: Select all

npm install

and press enter. [This installs all dependencies for the server.js]

Let’s test by running node server.js
Type

Code: Select all

node server.js

and press enter.

On the server go to http://localhost (Note: you may need to enter the full IP of the machine.)
You should see a black screen and if you talk, you should get speech to text!

If all works out, lets run the server.js in pm2 so if it crashes the pm2 app will restart it automatically and keep it up and running.
Type

Code: Select all

pm2 start server.js

and press enter.


CLIENT-
Open a web browser and go to the http://IPofServer:8000

[rmrichesjr]

Creating a Speech to Text ‘Server’ on MacOS 10.12

All credit goes to @daveyjones for his code and help. I just made the tutorial. I use the Focusrite Scarlett 2i2 USB Audio Interface for the connection from the Aux line out of the chapel to the MacBook Pro.

SERVER-
Get an IBM Dev Account - Standard – https://console.bluemix.net/catalog/ser ... ch-to-text
In Terminal install Homebrew -
...

A brief discussion among moderators resulted in approval of the above post. During that discussion a few concerns were expressed:

• Caution is advisable regarding sending Church meeting audio to third-party computing infrastructure.
  
  The moderators cannot guarantee the safety of the posted code.
  
  Moderators do not guarantee whether the use of an IBM developer account for this purpose or the use of the Watson resource are within the permissions for such an account.

[laengland]

Don't count on our church to provide any help for the hearing impaired.

[russellhltn]

Don't count on our church to provide any help for the hearing impaired.

They might not help as much as you want, but considering that a transmitter is part of every stock installation for a chapel, they commitment is fairly significant.

[rmrichesjr]

Also, along with the cost of the transmitter for each meetinghouse, last time I checked each of the receivers is pretty spendy.

[russellhltn]

Google has announced a new app "Live Transcribe" which is designed specifically for helping the deaf.

Making audio more accessible with two new apps

I found it in Google Play on my Pixel 1. Dirt simple. It works, but I do see some errors when I play back some YouTube videos with speech. But then I doubt of any technology is perfect at this point. It is interesting to see it change it's word choice as more words come in. You don't want to be reading at the very edge but about a line or so back.

[mnshowlett]

I just moved into a ward where they have a cheap and simple solution using Google Translate to capture audio from the pulpit mic and in classes from a roaming mic and Google Translate transcribes it to text. The text is then exported to a Google Drive document and the deaf person accesses the Google Doc on her smartphone and reads the transcription in real time. It works and the deaf sister is happy with the solution.

Our ward has many Spanish speaking members who struggle with understanding the proceedings of the meetings, so we have a real-time translator - legacy style to translate what people say to wireless headphones. One day one of the youth was watching the Google Translate setup we have and it translated the a testimony from a Spanish speaker to English on the fly for the deaf sister. It was a serendipitous find and we are now looking at using Google Translate and displaying the results of language translation on big screen TVs enabling the Spanish people to read what is being said - and. - translation of Spanish speakers to English for the rest of the congregation.
I am setting up a small proof-of-concept of this solution in Elders quorum this Sunday. I hope that we can get it to work properly. Everyone will need to speak into wireless mics that are not for audio augmentation, but rather to feed the clear audio into the speech capture system.

We will have the following setup:
* Chromebook with access to internet through Ethernet cables to ensure high bandwidth and transfer of audio to Google Translate.
* Icicle Speech digitizer device
* Two setups - since Google Translate can only translate one language at a time - unless I can figure out how to do both ways.
* Output to HDMI

Has anyone had experience with this? Do you have any recommendations or ideas?

Thanks!