Virtual scene rendering for deaf people using AI based audio analysis

Today, the estimated population of Pakistan is approximately 228 million, out of which approximately 10 million people are hearing impaired. The hearing-impaired people find hard to fit in with the local community due to their disability. A deaf and dumb person is unable to experience his/her surrou

Project Title

Virtual scene rendering for deaf people using AI based audio analysis

Project Area of Specialization

Electrical/Electronic Engineering

Project Summary

Today, the estimated population of Pakistan is approximately 228 million, out of which approximately 10 million people are hearing impaired. The hearing-impaired people find hard to fit in with the local community due to their disability. A deaf and dumb person is unable to experience his/her surroundings, creating a boundary with the world of sounds.

This project tends to bridge this gap by using Artificial Intelligence (AI) based analysis of the sounds and virtual rendering scenes accordingly on the screen of a wearable device that represent the surrounding environment for the deaf people. This helps the deaf and dumb in recognizing the sounds to which they are alienated.

The deliverable of the project is a wearable device, capable of processing the surrounding audios signals with AI and displaying the virtual scenes to the hearing-impaired people. The audio data of voices surrounding our environment is collected. Noise is removed and speech features are extracted using software techniques. Machine learning techniques are used for the recognition of audio signals. After recognition the virtual scenes are displayed in the screen of the wearable device.

Project Objectives

The objectives are:

• To convert surrounding audio into meaningful images for scene rendering for hearing-impaired people
• To design, lightweight, portable and easy to use wearable device for the hearing-impaired people

Project Implementation Method

The project is implemented in four steps. First three steps are software-based steps and the last step is development of wearable device for the project.

In the first step, audio data of surrounding voices is gathered. The data includes daily life voices such as dog barking, jack hammer, car horn, engine idling, drilling, street music, construction sounds, gunshot noise, air conditioner sound, Ambulance siren and children playing.

In the second step, speech analysis of the collected data is performed. It consists of processes like noise removal and speech feature extraction. The speech feature extraction is based Mel Frequency Cepstral Coefficients (MFCC) technique. The technique splits the given audio into frames and compute MFCC for each frame. 

In the third step various Machine Learning (ML) technique are trained for audio signals recognition. For this purpose, first the dataset is divided into disjoint train-test sets. Training is performed on 70% data (train set) and Testing on remaining 30 % data (test set). The ML techniques used are KNN, Support Vector Machine, Random Forest, Logistic Regression, Decision Tree and Naïve Bayes.  All these techniques are evaluated on the test data to identify the best one for the recognition of surrounding audio signals 

In the fourth step the wearable device is developed. It consists of Raspberry pi 4, a display screen and a microphone. The Raspberry pi 4 is programmed with the best Machine Learning technique and the Mic connected to it fetches the real-time audio. Based on ML judgments, the virtual screen is displayed on the screen.

Benefits of the Project

The project is developed to help the hearing-impaired people:

• Providing visual images of the sounds around them.
• Light weight and portable wearable device
• Low power consumption

Technical Details of Final Deliverable

The deliverable of the project is a wearable device, capable of processing the surrounding audios signals with AI and displaying the virtual scenes to the hearing-impaired people.  

The procedure entails gathering audio data from nearby speakers and noise, extracting speech features, and training with Machine Learning (ML) algorithms for audio signal recognition.

Finally, a wearable device is used to gather audio signals from the surrounding area and to display a virtual scene on the wearable device's screen in response to an ML-based judgement on the surrounding voice.

Final Deliverable of the Project

HW/SW integrated system

Core Industry

Medical

Other Industries

Others

Core Technology

Artificial Intelligence(AI)

Other Technologies

Others

Sustainable Development Goals

Good Health and Well-Being for People

Required Resources

If you need this project, please contact me on contact@adikhanofficial.com