SmarterChat

Project Title

SmarterChat

Project Area of Specialization

Computer Science

Project Summary

There are many people in the world that are auditory impaired and mute by birth and by some infelicitous medical conditions which leads to vanish their facility to verbalize and aurally perceive. Communication for such people with other people is a huge challenge for them in this way. When it is a matter of online meeting, it becomes a challenge to convey our message towards them so we are planning to build a system which will avail these people as well as other people to communicate with them because we will translate their sign language into text as well as narration to avail the other person communicate with them. We will engender a meeting app just as Google Meet or Zoom which will have the functionality of translating the sign language into text and narration.

Project Objectives

As mentioned above we are building an app SmarterChat just like Zoom and Meet but it will have a functionality of communicating with the auditory impaired and mute people. During covid-19 when we have been taking lectures online, we felt that if there is an auditory impaired and mute person in the meeting, how will he/she be able to take lectures or participate equally in the class? So, we commenced to develop this kind of application which will be the most refined solution for this difficulty and will be a helping hand for the deaf and mute personalities as well as other people to communicate with them easily

Project Implementation Method

Our goal is to translate American Sign Language (ASL) to English for all the members who have joined the meeting. The system has three main components: camera, a mobile app, and machine translation tools that run on a remote backend. The special user will stay in front of the camera and communicate with the people in the meeting using American Sign Language. Camera reads the gestures and sends them to the app. The app will process the sign language gestures to translate them into English in real time. Then it sends the English translation back to all the other devices which are connected to that call, which plays it out loud using a text to speech tool.

On the other hand, we’ll also be working on speech to text conversion from the other members to the person who is deaf and can’t hear them. He/She will be able to see the spoken English in the form of captions on their side.

Benefits of the Project

Project will revolutionize the meeting by adding a very new feature of translating the sign language to actual narration of english language. In this way people who are deaf and mute will be able to communicate with other people and can participate equally into the meetings by just performing the signs that they perform everywhere for communication. This app will also increase the hiring rate of the deaf and especially mute people.

Technical Details of Final Deliverable

Final deliverable will be a hybrid mobile application for android and ios created using flutter and will use WLASL video data set for training the model. Model will work at 25 fps and will trim the performed gesture from the live stream and will compare and classify the gesture. Now it will pass the classified words to the NLP library to create a complete sentence from the words passed and will pass those sentences to the peers and on the peer level a text to speech library will narrate that sentence to all the meeting participents.