Real time implementation of facial expression recognition for blind people using fpga and labview platform

Project Title

Real time implementation of facial expression recognition for blind people using fpga and labview platform

Project Area of Specialization Electrical/Electronic EngineeringProject Summary

Facial expression is an important indicator of a person’s emotion.  Computers and other electronic devices in our daily lives will become more user-friendly if they can adequately interpret a person’s facial expression, thereby improving human-machine interfaces. Specially,this system would be designed for blind persons who can’t predict the facial expressions of human. The main idea of this project is to focus upon the system delays and time efficiency of different platforms that have been widely used to implement facial expression recognition systems, commonly facial expression Systems have been developed with different techniques of image processing algorithms embedded on variety of  embedded processors and personal computers which are bulky, expensive and nonportable and are faraway to implement in real time environment as standalone application. Analysis of facial expression needs fastest computing platform that must have good transient responses without having any processing delays which can be employed in real time, as the response for blind person should be quick enough like live stream if suddenly the person changes his face expression from within limit of 1 second it also predicts the change in emotion for this reason we intend to design the fastest algorithm to recognize facial expressions by using parallel computing platform MyRIO( FPGA) control through  LabVIEW . The software and hardware both utilize their parallel processing techniques to bring the designed system in real time environment. Proposed platform is far better than traditional computing techniques can be used in various applications where delays of Nano second cause a significant damage. MyRIO board contains the 7 series FPGA along with integration of ARM A9 processor which has the processing speed of 667MHz which is highly responsive in practical environment.  The board itself is a module ideal to perform real time operations. Here we don’t go for complexity of very hard description language like VHDL and Verilog however MyRIO can be  programmed through LabVIEW and it  is good in many aspects as it  which is flexible and easy to use, it has very simplest graphical  programming environment connecting blocks instead of tons of lines of code and one of the highlighting factor for this platform is that; it is easily integrated with open source libraries like in this project OpenCV python library will be used, that  further improves the efficiency of  facial expressions algorithms. We will modify the existing systems which are suffering from nail of delay factor by implementing them into smart real time application after comparative analysis of different platforms.

Project Objectives

Over 285 million people are visually impaired around the world, 39 million of them are blind and the rest have low vision, they could not interpret human emotions and their expressions. We intend to develop an efficient hardware, that can recognize the facial expressions and convert those into the audio within milliseconds with high accuracy i.e. least delay factor. 

The common system of facial expression recognition almost takes personal computer as development platform. In our project we are converting PC based system into an embedded system with fast processing speed, small volume and price concessions so that the product begins to step  into the range of people's horizons The objective are:

                                                         

•   Increasing processing speed
•   Converting PC based system into   embedded system
•   Feasibility of the system
•  Attain maximum accuracy

 Increasing processing speed

  First objective of this project is to modify the existing face recognition system which are lagging from various latencies and throughputs in terms of transient response and delay on large amount of data base, by upgrading existing system and implementing in emotion expression application.

Converting PC based system into embedded system

  Integrate myRIO FPGA with camera and Audio device as standalone application.

Feasibility for the system

 Another objective is to control the FPGA Controller with simple and efficient graphical environment of LabVIEW which does not use tons of lines of Verilog and VHDL language that are too complex to control the system design accessed through FPGA.

 Attain maximum accuracy

  Last but not the least, it is our prime objective to obtain maximum accuracy keeping all parameters     under control by overcoming following issues:

• Environmental (resolution, illumination and dust)
• Synchronization errors (interface between camera and processor)
• Algorithm issues (complexity)

Project Implementation Method

The project implementation method is divided into four main domains.

Data Acquisition

The first step deals with acquiring data i.e. images, videos without any noise or

environmental issues. The algorithm and the database aare stored in the embedded system (MyRIO) and the embedded system is interface with the camera, the camera is working as an acquisition device which takes the input. The fps, resolution, illumination effects, dust effects and other environmental effects can be attenuating so that the embedded system can give efficient output. There are features given to user where the accuracy is more important so the user can increase the resolution, where time efficiency (no delay) is required so user can increase fps, or it can be any situation.

Interfacing

The second step involves the process of interfacing, after the setting of modes the input images in the form of data is transferred to MyRIO (embedded system). The interface between MyRIO and camera in our project is a USB cable. The cable which we are using is USB 2.0 and USB 3.0. We synchronize the camera with the device (embedded system) so the interfacing delay is very low or negligible.

Image Preprocessing

Finally, the embedded system takes the input with enhanced quality of image and removal of noise effects. We used some strategies to filter out the unwanted the impurities and to improve the resolution of image i.e. the use of histogram improves the brightness of Image by providing the equal parts to whole image rather than focusing one part of image. Applying Algorithms.

Algorithms that we are using detects the existence of human’s face, the algorithm is Camshift algorithm with Haar classifier. Camshaft detects the face area through maximum pixel

density. It can detects the face when the object(person’s face) is near or far from camera and also with a rotation then finally apply emotion analysis techniques to recognize the expressions of  human and the algorithm uses Haar cascade classifier which is a machine learning based approach and match the feature of expressions.

Removal of delay factor

The face classifier finishes face recognition by contrasting face with the feature extracted from image. Even a 24x24 window results over 160000 features and among all these features extracted by the Haar classifier, some of them are irrelevant which also makes the system to take time. MyRIO is a parallel processor, additionally we are using parallel buffer which doubles the processing speed and reduce the effect of delay.

Benefits of the Project

Key Benefits 
 
It operates without user cooperation. It is the fastest facial expression technology with no delays and have fastest transient response. It can be used in patient healh monitoring purpose in intensive Care units                                                                                         
Adoptable with concerned optimal response Flexible according to the user’s demand 
 
 Due to parallel processing nature of both hardware and software its output response would be far better than any other platform. It is most suitable combination to design the vision-based application that can be implemented and deployed as standalone system in real time environment so that the purpose to chop off the processing delay factor can be served. Blind person will be able to interpret the facial expressions of human without any delay.

Technical Details of Final Deliverable

The project consist on two main components that are MyRIO (embedded system) and a camera. The camera is working as an acquisition device which takes input in the form of images and video and apply preprocessing techniques to enhance the quality of image for better output results. The other component is the embedded system which handles the database of different expression styles and the programs that contains algorithms and other libraries With the aid of image acquisition device(the camera) the acquire images, after a series of preprocessing (as shown in the figure) which aims to enhance the precision of image and prepare for the better basis of feature extraction and recognition, are passed to the embedded system 
 
DESIGN DETAILS 

The complete design is divided into two parts, face static recognition and dynamic tracking. We extract the features of facial expressions from image and use algorithm in a template matching to calculate the degree of similarity between image and templates of  different facial styles. 

TIME EFFICIENCY OF THE SYSTEM

To reduce this delay and get transient response with a database, we use an embedded system which is designed for real time application, MyRIO. MyRIO has built-in FPGA board which enables it to do parallel processing and have a sufficient storage which reduces the time a processor requires during the interfacing with an SD card. MyRIO have latest technology of ZINC IC and its another version is using in industry with the name CompactRIO. 

Final Deliverable of the Project Hardware SystemCore Industry MedicalOther Industries Health Core Technology Wearables and ImplantablesOther Technologies Robotics, NeuroTech, OthersSustainable Development Goals Good Health and Well-Being for People, Industry, Innovation and Infrastructure, Reduced InequalityRequired Resources

Item Name	Type	No. of Units	Per Unit Cost (in Rs)	Total (in Rs)
			Total in (Rs)	67997
MyRio	Equipment	1	63000	63000
Camera	Equipment	1	3000	3000
Headset	Equipment	1	1997	1997