55°F

Aaron Parecki

  • Articles
  • Notes
  • Photos
  • Random Hacks of Kindness - December 2010

    December 8, 2010

    On the weekend of December 4th and 5th, people gathered in coworking spaces around the world for the Random Hacks of Kindness hackathon. In Seattle, two teams worked on Population Centers in Disaster, and Mobile Assessment of Damage for the Public.

    Results

    After a weekend of much hacking, good food, and good company, we came away with solid problem definitions, UI mockups, and prototypes of an iPhone app, Android app, a website backend, and a voice-powered phone menu.

    Mobile Assessment of Damage for the Public

    The goal of MadPub.org is to build an application that streamlines the process of reporting and assessing the amount of damage after the occurrence of a natural disaster in the United States. Beta testing will be for the State of Washington.

    In the event of a disaster, each county has a 72 hour window to report a dollar estimate of damage to the state and federal government in order meet the threshold for a federal declaration. The current system for assessing and reporting damage has many inefficiencies, requiring government personnel from each county to manually fill out a form after talking on the phone with the person reporting the damage.

    To solve this, we are building a system that allows businesses and individuals to report damage on a variety of platforms including: a phone tree with voice recognition managed with Tropo, a web form, and Android and iPhone apps.

    Information Flow

    Mobile Apps

    Laurel (@laurelfan) and Chi-Bong Ho built working prototypes of iPhone and Android apps! The apps allows you to enter the required information to gather information about the damage, and you can take photos and send them in the report as well. Mark (@malept) prototyped an HTML5 form using jQuery Mobile.

    MadPub.org iPhone Version MadPub.org Mobile Web Version

    Server

    Mark (@malept) and Pat Tressel built a Sahana-backed web form for submitting data, which the mobile applications can also send data to. While that was getting ready, Kav (@kavla) threw together a quick Rails app so uploading from the apps could be tested.

    Phone

    Aaron (@aaronpk) worked on a phone interface for entering the data. This involved a Tropo app to handle the speech recognition.

    Speech recognition is a difficult problem in general, but Tropo provides a great platform to make it a lot easier.

    When someone calls the hotline, the Tropo application answers the call and offers to provide an overview of the information the caller will need to answer the questions. It then asks a series of questions needed to fill out the form.

    Most questions can be answered with a simple "yes" or "no." More complicated questions require more computing power to understand. Even the top of the line speech recognition technologies have a long way to go to understand what people are saying on the phone. How many times have you, a person, asked someone on the phone whether they said "b" or "d"? Now imagine a computer trying to hear that without any context!

    Luckily, Tropo provides a way to recognize speech patterns from a smaller set of vocabulary. Tropo has built-in speech recognition to understand numbers, currencies and dates, and you can even specify your own set of options. What this means is that it's far easier to get a computer to tell the difference between "sales" and "suppport" than it is to tell you what word a person said in no context. If you give the computer two options, it can more accurately match what the caller has said.

    With this in mind, I set up a system for building survey applications with sets of options. For any questions that have completely free-form answers, such as "please describe the damages to the property," the system records the audio from the caller so that the operator can play back that part of the message later. In fact, every response to every question has the audio recorded automatically so that the individual responses can be played back at a later date.

    Here is some sample code showing what it takes to create a survey in this framework.

    You can see the different types of responses the Tropo can understand. I added the "[BOOLEAN]" and "[RECORD]" types to the framework which causes Tropo to listen for a "yes" or "no" response, and that will just accept a recording of whatever the caller speaks.

    My goal in designing this Tropo application was to make it as easy as possible to set up survey questions and change them in the future. The application takes care of recording responses and tracking answers, and provides the callers' responses in the admin's web interface.

    The data gathered from the call is presented to admins in a web view. Here is an example of a transcript of one of the calls. There are "listen" links which will play the audio for that response. Some questions don't have recordings because the answers were determined based on other answers. For example, entering a zip code determines the county and state.

    If you'd like to check out the source code to this application, you can find it on Github. For more information on Tropo, check out the Tropo documentation.

    Wed, Dec 8, 2010 12:08am -08:00 #hackathon #rhok #speech #speech recognition #tropo #voice
Posted in /articles

Hi, I'm Aaron Parecki, Director of Identity Standards at Okta, and co-founder of IndieWebCamp. I maintain oauth.net, write and consult about OAuth, and participate in the OAuth Working Group at the IETF. I also help people learn about video production and livestreaming. (detailed bio)

I've been tracking my location since 2008 and I wrote 100 songs in 100 days. I've spoken at conferences around the world about owning your data, OAuth, quantified self, and explained why R is a vowel. Read more.

  • Director of Identity Standards at Okta
  • IndieWebCamp Founder
  • OAuth WG Editor
  • OpenID Board Member

  • 🎥 YouTube Tutorials and Reviews
  • 🏠 We're building a triplex!
  • ⭐️ Life Stack
  • ⚙️ Home Automation
  • All
  • Articles
  • Bookmarks
  • Notes
  • Photos
  • Replies
  • Reviews
  • Trips
  • Videos
  • Contact
© 1999-2025 by Aaron Parecki. Powered by p3k. This site supports Webmention.
Except where otherwise noted, text content on this site is licensed under a Creative Commons Attribution 3.0 License.
IndieWebCamp Microformats Webmention W3C HTML5 Creative Commons
WeChat ID
aaronpk_tv