This study examined effects of three text release features used with Automatic Speech Recognition (ASR)-based automatic captioning and of hearing status on quality of performance on problem-solving tasks, number of words two team members exchanged with each other, and on member's judgments regarding the extent the ASR generated text contained errors. Each of 36 teams, consisting of one deaf/hard of hearing (DHH) and one hearing member, worked online to complete three mapping tasks. These mapping tasks required naming of map locations that were marked by numbers along a pathway indicated by a line with numerous turns. For each of three mapping tasks, the hearing member used a different ASR-based text release feature to produce captions for the DHH member to read, either: (a) controlled release; (b) automatic release with no editing; or (c) automatic release with editing. Results suggested that for DHH and hearing members to solve a problem using automatic captioning with ASR synchronously, it is important for the captioning technology to facilitate rapid exchange of words between members and, at the same time, facilitate dealing with the errors produced by the ASR. Consistent with this proposition, problem-solving performance was highest under the automatic release with editing feature.

The study investigated effects of computer-based messaging and training in communication strategies on interactions of deaf and typically hearing (TH) teammates in completing decision-making tasks without interpreter support. Fifteen teams, two deaf and two TH college students each, completed three decision-making tasks, one without messaging, one with messaging, and one with messaging and training in communication strategies. Each interaction was coded for (a) communication method, (b) to whom the interaction was directed, and (c) the function of the interaction. Without messaging, teams used speech, sign, or paper and pencil; with messaging, they predominantly used this technology to communicate with each other. Without messaging, teammates directed communications to members of the same hearing status; with messaging, they directed communications to the whole team. Teammates made fewer communication repairs with messaging than without. In focus groups conducted after the decision-making tasks, participants noted messaging's limitations and benefits.

This study examined the effects of (a) schema-enriched communication and (b) computer-based messaging on the sharing of knowledge and problem solving in teams with deaf or hard of hearing (DHH) and typical hearing (TH) postsecondary students. Four-member teams comprising either all DHH, all TH, or two DHH and two TH postsecondary students solved a complex problem. Measures consisted of (a) shared written information, (b) creation of a matrix with information for solving the problem, (c) recognition of information shared by team members, and (d) quality of the team’s problem solution. A total of 126 DHH and 126 TH postsecondary students participated in the study in teams with one of the three combinations of hearing status. Enriched communication increased teams’ sharing of written information, creation of a matrix, recognition of information shared by teammates and quality of the problem solution in teams.

This study examined knowledge sharing and problem solving in teams that included teammates who were deaf or hard of hearing (DHH). Eighteen teams of four students were comprised of either all deaf or hard of hearing (DHH), all hearing, or two DHH and two hearing postsecondary students who participated in group problem-solving. Successful problem solution, recall, and recognition of knowledge shared by team members were assessed. Hearing teams shared the most team knowledge and achieved the most complete problem solutions, followed by the mixed DHH/hearing teams. DHH teams did not perform as well as the other two types of teams.

Deaf and hard of hearing (DHH) individuals face barriers to communication in small-group meetings with hearing peers; we examine generation of captions on mobile devices by automatic speech recognition (ASR). While ASR output displays errors, we study whether such tools benefit users and influence conversational behaviors. An experiment was conducted where DHH and hearing individuals collaborated in discussions in three conditions (without an ASR-based application, with the application, and with a version indicating words for which the ASR has low confidence). An analysis of audio recordings, from each participant across conditions, revealed significant differences in speech features. When using the ASR-based automatic captioning application, hearing individuals spoke more loudly, with improved voice quality (harmonics-to-noise ratio), with a non-standard articulation (changes in F1 and F2 formants), and at a faster rate. Identifying non-standard speech in this setting has implications on the composition of data used for ASR training/testing, which should be representative of its usage context. Understanding these behavioral influences may also enable designers of ASR captioning systems to leverage these effects, to promote communication success.

Abstract Technology is playing an increasing role in the education of deaf and hard-of-hearing (DHH) students. This chapter discusses two uses of technology to provide communication access for DHH students (real-time captioning in classrooms and messaging and related technologies to facilitate communication in small groups) and two ways that technology is used for instruction of these students (multimedia materials for development of literacy and online tutoring). Several studies on real-time captioning and multimedia materials indicate that these approaches are beneficial in the education of DHH students. The limited work to date on messaging to support communication in groups with DHH and hearing students and online tutoring indicates that they are potentially important because they address unmet needs of DHH students. It is important to conduct additional studies to determine the effectiveness of these latter two approaches because current findings are preliminary.

Automatic Speech Recognition (ASR) and the wide use of smart phones and their apps have allowed huge inroads when preparing deaf and hard-of-hearing (D/HH) students to be effective and productive in the hearing workplace. This paper presents both a hearing instructor's experiences and a deaf researcher's observations when preparing deaf and hard of hearing students as computer technicians for the hearing work place.

To compare methods of displaying speech-recognition confidence of automatic captions, we analyzed eye-tracking and response data from deaf or hard of hearing participants viewing videos.

This study investigated use of automatic speech recognition (ASR) in 12 pairs where one individual was deaf or hard-of-hearing (DHH), and the other one was hearing, with the hearing individual using speech and ASR and the DHH one using typing. Each of the pairs used prototype software for messaging to communicate while completing a standardized decision making task. Results suggested that ASR produced text at a faster rate than a keyboard. When both participants used keyboards, they exchanged more messages than when one or both of them used a smartphone with a miniature keyboard.

32nd Annual International Technology and Persons with Disabilities Conference Scientific/Research Proceedings, San Diego, 2017

In this investigation, one study examined the perceptions and motivation of 55 deaf/hard-of-hearing (DHH) high school students who used the C-Print speech-to-text service as an accommodation in one mainstream course and interpreting and note taking in a second mainstream course. A second study examined the perceptions and motivation of 88 DHH college students who used each service in a different course. Students in the two studies completed a survey that asked separate sets of questions for the speech-to-text and for the interpreting/note taking supported courses. Results indicated that students rated the printed or electronic file text, saved for the purpose of after class study as part of the speech-to-text service, as more helpful than notes from a note taker.

This study investigated user experiences of participants testing a prototype messaging app with automatic speech recognition (ASR). Twelve pairs of participants, where one individual was deaf or hard-of-hearing (DHH), and the other one was hearing used the app, with the hearing individual using speech and ASR and the DHH one using typing. Participants completed a standardized decision making task to test the app. Regardless of hearing status of the participants or the type of device used, participants were generally satisfied with the app. These findings indicate that ASR has potential to facilitate communication between DHH and hearing individuals in small groups and that the technology merits further investigation.

This work has been supported by National Science Foundation grants: Analyzing the use of C-Print Mobile technology in STEM lab settings across multiple postsecondary sites, HRD-0726591; and Supporting deaf and hard of hearing undergraduate hearing students in STEM field settings with remote speech-to-text services, HRD-0726591. For more information, visit our website at http://www.rit.edu/ntid/cprint/ Benefits Potential Uses The C-Print system requires a trained transcriptionist who uses computerized abbreviations and condensing strategies to produce the text display of spoken information. At the same time, the spoken information is displayed on a computer or mobile device. The content can be saved and distributed as a transcript afterward. C-Print is a captioning technology for providing ccess to spoken communication. The C-Print technology is used to produce a text display of spoken information for individuals who are deaf and hard of hearing (or other individuals who may have difficulty understanding speech).

This survey-based study investigated deaf and hard of hearing (DHH) individuals' perceived need for technologies that may facilitate communication when meeting in small groups with hearing colleagues. Participants were 108 DHH postsecondary students who participated in co-op (internship) and capstone experiences at workplaces with hearing employees within the past two years. Participants' responses to a survey indicated that they were generally not satisfied with their current strategies and technologies for communicating with hearing persons in small groups.

Four cases from different countries are used to identify critical issues in the inclusion of deaf and hard-of-hearing (DHH) students in higher education programs. A case from Australia identifies students’ learning and vocational experiences at a university offering an extensive deaf student support program for 25 years. The case from Greece identifies key factors associated with academic access in higher education institutions with limited support services, provided mainly by volunteers. The case from New Zealand reports on student academic readiness and transition experiences in higher education institutions with small populations of DHH students. The case from the United States reports on student outcomes of the C-Print speech-to-text service for communication access by deaf students enrolled in mainstream college science and technical laboratory courses. These various components of inclusion and ongoing challenges are identified, and future are directions described.

This poster/demonstration session showcases C-Print, a typing-based transcription system. This form of real-time captioning will be provided for approximately one half day during the ASSETS 2014 Conference and will be part of a real-time caption challenge. The C-Print system requires a trained transcriptionist who uses computerized abbreviations and condensing strategies to produce the text display of spoken information. This spoken information appears as text on a computer or mobile device for viewing by the consumer approximately two seconds later.