Enabling deaf or hard of hearing accessibility in live theaters through virtual reality

Abstract

Recent advancements in Virtual Reality (VR) have made them a potential technology to improve understanding between Deaf or Hard of Hearing (DHH) and hearing people. Based on that, this study aims to extend these advances to enable live entertainment like theater plays to DHH people. At first, this work presents a survey, which covers some findings of the accessibility research using virtual and augmented reality systems, ranging years from 1996 to nowadays, and fields such as children, autism, motor rehabilitation, Parkinson disease, and inclusion for the impaired. After this initial review, it’s presented one solution in details, which is a live theater accessibility service for the deaf and hard of hearing people, aiming to be a concrete solution designed to bringing accessibility using virtual reality technology combined with Automatic Speech Recognition (ASR), Sentence Prediction and Speech Correction to generate text and sign language subtitling. In order to evaluate this method efficiency, a quantitative and qualitative study were performed and results showed that DHH spectators had good understanding of all evaluated theater plays and also good satisfaction using the proposed method. The best results are related to text subtitling. Regarding sign language subtitling, it is a promising technology, but a huge effort is necessary to start a standard for displaying definition in virtual or augmented reality device. Other main contribution of this work is to present the procedures and its results executed to evaluate the integration of ASR and Speech Correction based on Semantic Similarity of a solution designed to bring accessibility to DHH People in live theaters. Five datasets were submitted to two different ASRs and its outputs to the module of Speech Correction considering three groups of pairs to observe the semantic and syntactic errors presented by the modules as well as its performance under different configurations representative of possible scenarios of actual theatrical plays. The ASRs presented an error of 50% in average when applied to the audio of an actual play. The module for semantic similarity presented an average error of 18% on sentences not modified by the ASR. However, the output of the Semantic Similarity module is affected by the error introduced by the ASRs. Speech Correction based on Semantic Similarity would present less error than Syntactically based.