Assistive Device for Converting Urdu Sign Language into Speech

Any society?s progress in social welfare can be measured by looking at how well their disabled population is integrated into becoming a useful member of the society. A society where the disabled are left dependent on others will burden its economy and halt its overall growth. For a developing countr

Project Title

Assistive Device for Converting Urdu Sign Language into Speech

Project Area of Specialization

Biomedical Engineering

Project Summary

Any society’s progress in social welfare can be measured by looking at how well their disabled population is integrated into becoming a useful member of the society. A society where the disabled are left dependent on others will burden its economy and halt its overall growth. For a developing country like Pakistan, with its limited resources, and having the world’s fifth-largest population of disabled, their integration becomes a matter of utmost importance. Mute, and Deaf-Mute people can easily be integrated into society as useful members. This will not only reduce their dependency on others but also improve their quality of life. The only thing in the way of their progress is their ability to communicate. Even if they learn sign language, a vast majority of the Pakistani population do not understand sign language. Integration of disabled people into society becomes the responsibility of every member of society.

This project proposes a wearable system that can recognize gestures of the Pakistan Sign Language (PSL), and a speaker unit attached, which can speak out loud the signed message. This way a mute/ deaf-mute person would be enabled to communicate his thought to people who do not understand sign language. One of the challenges in designing such a system is that each sign language is unique. Just like spoken language, there are differences in each sign language. Though similar systems have been developed for the interpretation of some sign languages such as ASL, these systems may not perform as well in other languages such as Pakistan Sign language (PSL) because within each language there is a difference in emphasis on the part of the hand that is moved, the orientation of the hand, and the manner in which the fingers are bent.

For these reasons, a gesture recognition system, with sensors specifically selected to capture the movements of the Pakistani Sign Language, is required. Initially, in our project, a pair of wearable gloves were designed that contained several sensors of different types, at various locations all over the hand. Then a dataset of sensor values was generated for the most widely used signs from the PSL dictionary. An AI algorithm was trained to perform recognition of PSL gestures from the collected dataset. Finally, the glove design was refined and optimized by reducing the number of sensors and finding the most optimum location for their placement on the glove.

The completed project comprises a pair of fully functional gloves, with the ability to recognize signs from PSL and output them in the form of speech in real-time. We believe that such a system will directly play its part in the overall progress of Pakistan, as each member of the society will become a useful contributor to the country’s economy.

Project Objectives

• Develop a glove-based assistive device for communication of the mute/ deaf/mute Pakistani individuals.
• Interface 5 flex sensors, two MPU 6050s, and an HC12 wireless communication module with a microcontroller on each of the two gloves.
• Develop a program for the Arduino Nano microcontroller, to obtain values from five flex sensors, and 2 MPUs, and transmit this information wirelessly to a laptop where it gets stored.
• Data can be wirelessly transmitted to a computer from the gloves during the data collection phase.
•  Develop a dataset containing sensor values that include flexion of fingers, acceleration of hand movements, and tilt of the hand along three axes, which can be used by researchers to perform hand-gesture recognition.
• Apply an AI algorithm to the data which can then recognize sensor values in real-time and identify the corresponding PSL gesture with an acceptable level of accuracy.
• Interface a speaker system with the microcontroller through which the PSL gestures can be outputted in the form of speech.
• Power the glove pair with a rechargeable battery that works for 3-4 hours continuously, once charged.
• Develop a glove pair that is capable of recognizing at least 10 frequently used gestures of the Pakistan Sign Language.
• The glove pair is capable of functioning wirelessly.

Project Implementation Method

Implementation of this project was classified into the following stages:

1. Designing and fabrication of a system for the recognition of hand gestures during signed communication
2. Design of mathematical model for sensor optimization

• Reduction in the number of sensors used
• Burn simple classification rules onto the microcontroller

The first stage began with a literature review conducted across different scientific databases with keywords such as ‘gesture recognition’, ‘sign language recognition, ‘sign language translator’, ‘human-machine interface’ etc. to explore methods used so far for sign language recognition, their strengths and shortcomings, and the need for a new device.

To select the gestures to be translated, surveys were conducted in special schools to understand the day-to-day needs of the deaf/mute end users and choose the gestures accordingly.  The survey also helped acquire end-user expectations and functional requirements. The responses showed that two gloves would be required in order to be able to recognize the necessary gestures. 

Guided by the literature review and the surveys, the design of the glove was started. The approach for the initial design of the glove was to use a maximum number and type of sensors for better data collection. This design was to be revised later on to better cater to end-user requirements by removing the unnecessary sensors. A flex sensor was placed along the length of each finger of the glove to detect its degree of bend. Two MPU-6050 motion tracking devices were used: one placed on the knuckle region and another on the wrist. The MPU-6050 contains an accelerometer to measure acceleration along the x, y, and z-axis as the gesture is performed, and a gyroscope to measure rotation along each axis. All the sensors were connected to a microcontroller (Arduino- Nano) to record sensor values and generate a database for the classification algorithms. Bluetooth modules were used to transmit the data wirelessly to a laptop for recording. To make the device completely wireless, a power bank was used to power the hardware on each hand.

The dataset will be generated by making deaf/mute individuals in special schools perform the selected gestures while wearing the gloves. An attribute reduction and classification algorithm will be applied to the collected dataset. Firstly, the attributes in the collected dataset will be pre-processed and the data will be cleaned up. The attribute reduction algorithm will then be applied to remove redundant features from the data. The dataset will then be divided into training data and testing data. By applying the classification algorithm to the training data, rules for the classification of objects will be generated. The rules will be used to classify the testing dataset. Once classified, a speaker will be used to output the message as speech through speakers connected to the microcomputer.

Benefits of the Project

Sign language is the most widely used form of communication for speech/ hearing-impaired individuals. However, the communication barrier between these individuals and the rest of society, which is mostly unfamiliar with sign language, creates various hurdles in the lives of the specially-abled. Additionally, like any other language, sign languages too, vary across regions, so commercially available assistive devices can only serve a certain portion of the deaf community.

The assistive device we have designed would cater specifically to the needs of the Pakistani speech-impaired community which predominantly uses the Pakistan sign language. The glove would allow these individuals to communicate with non-speakers of sign language, and in doing so aid their better integration into society.  It would enable these individuals to perform day-to-day tasks that involve conversing with non-signers without assistance, such as going to the shop, etc. As a result, it would allow them to be useful members of society and reduce their dependency on others.

Having a speech or hearing disability severely reduces job opportunities for individuals who otherwise possess great intellectual and physical capabilities. Employers are hesitant in making the extra effort involved in hiring individuals with special needs. By enabling communication with non-hearing/speech impaired individuals, our gloves would increase employment opportunities for these people and thus reduce the national unemployment rate. This would in turn make the individuals financially self-sufficient, and reduce the burden imposed on the country’s economy.

Additionally, the work we have done within this project will contribute significantly to the body of knowledge of identifying Pakistan Sign Language Gestures and converting them into speech. The deliverables of this project will provide valuable information to future researchers regarding PSL gesture recognition. This information includes insight into the day-to-day requirements of a deaf/mute Pakistani individual regarding an assistive device for communication, the gestures most useful/ frequently used, the nature of these gestures (i.e. static or continuous), the number of hands involved in identifying each gesture, and the number and type of sensors needed to identify each of these gestures. The knowledge base generated can be employed by future researchers for a number of applications such as automation control using hand gestures.

Technical Details of Final Deliverable

• A pair of gloves equipped with flex sensors, MPU-6050, and a power delivery system, that operates wirelessly, for the recognition of Pakistan Sign Language Gestures, and their conversion into speech. The device would be rechargeable with the ability to run for 3-4 hours once fully charged.
• An AI-based classification algorithm that classifies the sensor data into different gestures of PSL with an accuracy of more than 75%.
• A data set of sensor values (10 flex sensors, 4 MPUs) comprising 20 subjects, performing 10 different PSL gestures.
• A code for the microcontroller to acquire values from 5 flex sensors, 2 MPU-6050s, and transmit the values wirelessly onto a laptop through HC-12.
• A block diagram and circuit schematic of the device, illustrating the components, their physical arrangement, and connections.
• A survey questionnaire to understand the everyday needs of the speech/ hearing impaired Pakistani individuals, and their expectations and requirements for an assistive device for communication. 
• A document containing end-user requirements regarding the functional performance of the glove-based assistive device and the PSL gestures frequently used on a day-to-day basis based on the responses of the conducted user requirements survey.
• A product design specification document containing necessary information based on literature review and surveys, to guide the design team in architecture and system design in order to meet technical, functional, and end-user requirements.
• A table that classifies selected, common PSL gestures into the categories of bilateral hand use (with the symmetrical motion of both hands), bilateral hand use (with the asymmetrical motion of both hands), and unilateral hand use (right hand).
• A table that identifies the different sensors on both gloves that would be involved in recognizing each of the selected gestures. 
• A protocol designed to obtain sensor values for PSL gesture recognition.

Final Deliverable of the Project

HW/SW integrated system

Core Industry

Health

Other Industries

Medical

Core Technology

Wearables and Implantables

Other Technologies

Artificial Intelligence(AI)

Sustainable Development Goals

Good Health and Well-Being for People, Decent Work and Economic Growth, Reduced Inequality

Required Resources

If you need this project, please contact me on contact@adikhanofficial.com