#speech
-
I’m pretty sure it’s pronounced “Wesley-Ann” (thebloggess.com)
and she keeps saying “Recalculating” in this really judgey, condescending way
-
Random Hacks of Kindness - December 2010
On the weekend of December 4th and 5th, people gathered in coworking spaces around the world for the Random Hacks of Kindness hackathon. In Seattle, two teams worked on Population Centers in Disaster, and Mobile Assessment of Damage for the Public.
Results
After a weekend of much hacking, good food, and good company, we came away with solid problem definitions, UI mockups, and prototypes of an iPhone app, Android app, a website backend, and a voice-powered phone menu.
Mobile Assessment of Damage for the Public
The goal of MadPub.org is to build an application that streamlines the process of reporting and assessing the amount of damage after the occurrence of a natural disaster in the United States. Beta testing will be for the State of Washington.
In the event of a disaster, each county has a 72 hour window to report a dollar estimate of damage to the state and federal government in order meet the threshold for a federal declaration. The current system for assessing and reporting damage has many inefficiencies, requiring government personnel from each county to manually fill out a form after talking on the phone with the person reporting the damage.
To solve this, we are building a system that allows businesses and individuals to report damage on a variety of platforms including: a phone tree with voice recognition managed with Tropo, a web form, and Android and iPhone apps.
Information Flow
Mobile Apps
Laurel (@laurelfan) and Chi-Bong Ho built working prototypes of iPhone and Android apps! The apps allows you to enter the required information to gather information about the damage, and you can take photos and send them in the report as well. Mark (@malept) prototyped an HTML5 form using jQuery Mobile.
Server
Mark (@malept) and Pat Tressel built a Sahana-backed web form for submitting data, which the mobile applications can also send data to. While that was getting ready, Kav (@kavla) threw together a quick Rails app so uploading from the apps could be tested.
Phone
Aaron (@aaronpk) worked on a phone interface for entering the data. This involved a Tropo app to handle the speech recognition.
Speech recognition is a difficult problem in general, but Tropo provides a great platform to make it a lot easier.
When someone calls the hotline, the Tropo application answers the call and offers to provide an overview of the information the caller will need to answer the questions. It then asks a series of questions needed to fill out the form.
Most questions can be answered with a simple "yes" or "no." More complicated questions require more computing power to understand. Even the top of the line speech recognition technologies have a long way to go to understand what people are saying on the phone. How many times have you, a person, asked someone on the phone whether they said "b" or "d"? Now imagine a computer trying to hear that without any context!
Luckily, Tropo provides a way to recognize speech patterns from a smaller set of vocabulary. Tropo has built-in speech recognition to understand numbers, currencies and dates, and you can even specify your own set of options. What this means is that it's far easier to get a computer to tell the difference between "sales" and "suppport" than it is to tell you what word a person said in no context. If you give the computer two options, it can more accurately match what the caller has said.
With this in mind, I set up a system for building survey applications with sets of options. For any questions that have completely free-form answers, such as "please describe the damages to the property," the system records the audio from the caller so that the operator can play back that part of the message later. In fact, every response to every question has the audio recorded automatically so that the individual responses can be played back at a later date.
Here is some sample code showing what it takes to create a survey in this framework.
You can see the different types of responses the Tropo can understand. I added the "[BOOLEAN]" and "[RECORD]" types to the framework which causes Tropo to listen for a "yes" or "no" response, and that will just accept a recording of whatever the caller speaks.
My goal in designing this Tropo application was to make it as easy as possible to set up survey questions and change them in the future. The application takes care of recording responses and tracking answers, and provides the callers' responses in the admin's web interface.
The data gathered from the call is presented to admins in a web view. Here is an example of a transcript of one of the calls. There are "listen" links which will play the audio for that response. Some questions don't have recordings because the answers were determined based on other answers. For example, entering a zip code determines the county and state.
If you'd like to check out the source code to this application, you can find it on Github. For more information on Tropo, check out the Tropo documentation.
