Sense Aural

This project is for the ease of communication between a hearing-impaired and a normal person, as it is difficult to understand sign languages used by a hearing-impaired person. So, to overcome this problem we have come up with a smart device which provide a dual way communication allowing to convert

Project Title

Sense Aural

Project Area of Specialization

Software Engineering

Project Summary

This project is for the ease of communication between a hearing-impaired and a normal person, as it is difficult to understand sign languages used by a hearing-impaired person. So, to overcome this problem we have come up with a smart device which provide a dual way communication allowing to convert hand gestures (sign language) to text/ audio & text/audio to hand gestures.

The hand gestures will be recorded by a sensor called usens fingo hand tracking sensor, which will allow us to calculated the fingers coordinates.

Total 26 hand gestures are stored in the database, further will be updated. An effective algorithm is used to calculate the mean of different coordinates for a single hand gesture, which will help in differentiating out each gesture.

In first phase, when a hearing-impaired person will make hand gestures, it will be recorded by the sensor, and then hand gesture will be recognized through the database and the corresponding sentence or a word will be displayed on the smart phone screen allowing you to listen it through audio too.

In second phase, when a normal hearing person will speak, the corresponding hand gesture to each word will be fetched out from the database, making a complete hand gestured sentence.

This smart device is a fast and an efficient solution to this stated problem.

Procedure Of Work

First user will open main menu screen there will be three options to choose, Audio, Text, and Gestures. If a user chooses Audio, a new screen will be open where user can record sentences which will later on converted into text, checked with respect to spellings and grammar and if a user chooses Text S/he will enter text which will also be checked with respect to spellings and grammar after that those sentences would be divided into words and checked for corresponding hand gestures in database. If a word is not found it will further be broken into tokens (Alphabets) and then their corresponding gestures will be fetched out and displayed to the screen for other user. If a user selects gestures, then s/he will make gestures whose mean coordinates will be calculated further through geometric patterns and mean value its corresponding words, letters, and audio will be fetched out and displayed on screen.

Project Objectives

Project Objective:
The project will meet the following objectives:
Objective #1: Conversion of hand gestures to text/audio.
Objective #2: Conversion of text/audio to hand gestures.
Objective #3: Dual way Communication.

Objective #1: Conversion of hand gestures to text/audio.
To solve this problem, we have bought uSens Fingo that detects sign language/ hand gestures, the sdk is supported with unity so that we have created the algorithm according to it, which then converts gestures (sign language) to text & audio.
Objective #2: Conversion of text/audio to hand gestures.
For the next way around, we will be working on android studio to convert audio to text, and then text to gestures (sign language). The gestures are stored in cloud in the form of database, we are using Firestore database. When audio comes as input, it will be converted into text, the text will then be sent to the database to fetch the corresponding gesture image/video.
Objective #3: Dual way Communication.
This whole project will be executed as an android application, with a complete hardware, including uSens Fingo Sensor & a smart phone, that will provide dual way communication for both hearing-impaired and a normal hearing person.

Project Implementation Method

The methodology of the work is divided into two phases. In first phase we will discuss the way to translate text/voice to gesture conversion in application and in second phase gestures to text/voice conversion will be discussed.

Step 1: Text/voice to gesture

Following key steps are required to analyze and working with interpreter application.

Step 1.1: Enter voice

voice will be entered by normal hearing person, which will be converted into text, stored in string and then will be broken into tokens and sub-token that will be converted into gestures.

Step 1.2: Spell Check

Spell check module will ensure that the correct text is given as input, otherwise it will not be processed, and the user need to write the correct spelling.

Step 1.3: Grammar check

Grammar check module also incorporates to ensure that the correct grammar is used else its will not be processed and asked to re-enter.

Step 1.4: Creating Token & sub-tokens

The user input will split into words and characters both stored in their arrays respectively. These arrays will help in generating gestures.

Step 1.5: Fetching

All the characters and words along with their corresponding gestures will be stored in the    database. These characters and words will be fetch from the database to convert into images and video.

Step 1.6: Concatenation

All the character-based and word-based gestures tokens are concatenated and save in sentence. This file sends gestures data to the GUI.

Step 1.7: Playing videos on GUI

Images and Videos can be played on GUI for understanding.

Step 2: Gesture to text/voice

Following key steps are required to analyze and working with interpreter application.

Step 2.1: Enter gesture

Gesture will be entered by a deaf person, which then be converted into words by certain patterns and these words will be joined together to form a sentence.

Step 2.2: Creating words:

The user gesture input will be fetched from database and then will be converted into corresponding words. These simultaneous words will then be combined to form a sentence.

Step 2.3: Creating sentences:

All the corresponding words will then be combined to form a sentence.

Step 2.4: Playing text/voice on GUI

Text/voice will be generated of the corresponding gestures.

Benefits of the Project

As hearing-impaired person uses hand gestures which are called sign language and it is not understandable by everyone around. This gap leads them to isolation from society. They feel backward and special. Psychological and mental problems surround them. They get caught by complexes as they see other people doing normal day to day things easily but for them, its hard. At this stage, they feel more secure in keeping themselves away from the normal world. The solution they usually come up with is they try to mix with the people of their own type. It’s not a bad thing, but these people are also suffering the same like they do. So, it doesn’t help. This project will help hearing-impaired people to get out of isolation and feel normal among hearing person and can communicate easily without hesitation.

Emotions play a vital role in one’s life. Happiness, sadness, angriness, feeling of love, hate, enthusiasm, anxiety, all of these feelings are normally expressed through words, actions or expressions. This project will help them be more expressive and able to communicate their problems, happiness with others.

When they can’t communicate properly, they can’t alert others in emergency too. Like when one gets stuck at any place where he isn’t visible, one needs his voice to call for help. Similarly, when countering with thieves, robbers, or terrorists, they need voice to call for help. It will be easier for hearing-impaired people to ask for help and communicate easily.

Technical Details of Final Deliverable

Technical Details of final Deliverables:
Problem:
Communication for hearing-impaired person is difficult, it creates a communication gap between a hearing-impaired person and a normal hearing person. As hearing-impaired person uses hand gestures which are called sign language and it is not understandable by everyone around, this leads to communication problem for them at shopping, at schools, at hospital, with their family and other social gatherings and may face security issues. This social isolation for the hearing-impaired person lead us to the idea of this project. This project domain is in health technology.
The project will work as an interpreter, a dual mode from gesture to text/voice and text/voice to gestures. It is a hardware and software-based application.

Goals:
The Goal is to achieve dual way communication between hearing-impaired person and a normal hearing person, that can be achieved by the conversion of hand gestures into text/audio and text/audio to hand gestures.

Development Architecture:
We have used agile development architecture to complete this project. As agile emphasizes on pair programming, test-driven development, stand-up planning sessions and sprints. Agile is a different approach in software development as it focuses on the team work and the working environment.

Coding Architecture:
PIPE FILTER is the coding/software architecture that we are using in our project. A pipe Filter is used to convey the processing in sequential way. There exists a data resource which provides the system with the data and the pipe carries out the data or any processing throughout the system. The Filter is a box in which processing and any other functionalities takes place.
In our project the uSens Fingo & Firebase acts as data resources for the modules, respectively.
Unity acts as the pipe for the first module, Conversion of gesture to text/audio, and Android studio acts as the pipe for the second module, Conversion of text/audio to gesture.
Visual studio acts as filter for the first module and Android studio for the second module.

Final Deliverable of the Project

HW/SW integrated system

Core Industry

Health

Other Industries

Education

Core Technology

Wearables and Implantables

Other Technologies

Artificial Intelligence(AI)

Sustainable Development Goals

Good Health and Well-Being for People

Required Resources

If you need this project, please contact me on contact@adikhanofficial.com