Implementation of Semi Assistant Robot with Image Processing and Speech recognition capabilities

As technology is going sky high day by day. We want everything automated in our daily life. In the near future it will be a need of every person to get their simple task done by a system. We are going to assemble a robot that can easily perform the tasks for humans by being able to see and speak. On

Project Title

Implementation of Semi Assistant Robot with Image Processing and Speech recognition capabilities

Project Area of Specialization

Robotics

Project Summary

As technology is going sky high day by day. We want everything automated in our daily life. In the near future it will be a need of every person to get their simple task done by a system. We are going to assemble a robot that can easily perform the tasks for humans by being able to see and speak. Only the mind work of a human will be a requiement.

This project is about a multitasking robot whose core functions are Image processing, Voice recognition and biomedical purposes. On successful implementation the robot will be able to detect and recognize humans and describes visual perceptions, understands their voice commands and act according to it and make Electrocardiogram of any person and display the data and do other biomedical functions.

Project Objectives

The main objective of this project is to create a robot that can perform the daily tasks of humans with ease and can dissolve with humans by understanding their language, seeing what they see, speaking what they speak. The project should be the root to a tree which means that it can be upgraded and updated with time until it become fully functional as someone wants.

Key objectives are a robot that can do:

Face recognition

Object tracking

Speech recognition

Weather report

VOC pollution report

bio-medical function like ECG, spO2 level, heartbeat, fever.

Project Implementation Method

To implement our assistant robot we have divided our work into four blocks: 

1-    Speech Recognition block 

2-    Image processing block

3-    Multiple functions block

4-    Integration of above

(1)   In speech recognition block we used a Raspberry Pi        Zero W in which we installed Raspbian. After that we  installed Google speech API and made that interact with the other blocks to be formed.

(2) In image processing block we used a rapberry pi 3b+ in which we installed Raspbian Stretch and then further we installed python 2.7 in it. We have two goals in this block

    Face recognition

   Object tracking (based on color)

To implement face recognition we installed OpenCV library in our Pi 3b+ and then installed some dependencies which are dlib and  imutils. Then we made our face recognizer using Deep Metric learning and OpenCV. Our rebort detect and recognizes face in videostream (using pi camera) and display the result on LCD and also saves the name of the person it recognized.

To implement color tracking we used some functions of OpenCV. We deteced the specific color and made a mask then we calculated the real time position of the object using openCV to move our robot left and right and used OpenCV to calculate the area of the object in order to move our robot forward and backward, like if area is small the object is away from the robot which means that robot should move towards the object in  other words the robot follow the object.

(4) In multi-functions blocks we have three functions

Weather report

VOC pollution report

Bio-medical tests

For weather report we have DHT11 for humidity measurements, BMP280 for pressure and height measurements and DS18b20 for temperature measurements. All sensors are connected to teensy 3.6. Robot speaks the values, using Emic2 speech to text module, it also have the ability to display on LCD and hard print it using thermal printer present within the body of the robot.

For VOC pollution report we are using CCS811 which will provide the values of volatile organic compounds in  the atmosphere in parts per million. This sensor is also connected to Teensy 3.6 and our robot has the ability to speak the values, display on the lcd and hard print it using a thermal printer.

For bio-medical tests we are using AD8232 for real time ECG, max30105 for heart beat and spO2 levels and DS18b20 for fever measurements. All of these sensors are connected to the Teensy 3.6 and again the robot have the ability to display the results on LCD, speak using Emic2 and hardprint using thermal printer.

(4) It is the most important block in our project. There we join all these functions to make an assistant robot. Firstly, pi zero recieves the voice command and speaks accordingly using Google Alexa. If some specific commands are given by the user then the robot will act accordingly, which means that our voice command will decide whether our robot should recognize us, follow an object, tell us weather report, VOC pollution report or do our ECG, measure our heartbate, spO2 level and fever.

Benefits of the Project

This project is the first step towards a fully automated robot. Considre this type of machine present in our homes that can recognize us and talk to us and do our little tasks, a robot that can give us weather report, pollution report. It can also take care of our health (which is very important in these days because every second person is suffering from a disease and has to spend many on tests in the clinics) by measuring ECG, spO2 levels, heart beat and fever.

We can also make functions like it can take care of our when we are not available and inform us when an unknown enters our house, it can sing songs and poems for amusement children. In short by making a little application specific changes to it we can use it in such a lot of ways that no one can not even imagine.

As we describe above that by making a little changes in our prototype our assistant robot can be used in hundereds of ways. We can also upgrade the bio-medical mode of our assistant robot by introducing artificial intelligence after which the redings can easily be understood by lay man and from the ECG the robot tells where the problem lies like a doctor does.

It made us realize that the robots are not something always made for the factories they can also be used in our homes performing our several tasks for us.

Technical Details of Final Deliverable

To acquire Alexa we used Google API. An API is a application programming interface which is a set of clearly defined methods of communication among various components. We used a Raspberry pi zero w for Alexa, we used another pi because we want that part of project to be active all the time, the reason behind that every time active is that it recieves speech signals in the form of voice and act according to the user wants.

We implemented our face recognition part using OpenCV and deep metric learning (deep metric learning actually learns metric function from data used in training to calculate similarities or difference between samples and also in DML we output a real valued feature vector instead of one label) Raspberry recieves images from video frames using pi camera, face is detected in it using Haars cascade method and the 128-d vector is calculated using a deep metric network, this vector is compared to the vectors we obtained from the images of known persons and compare it using support vector machine (SVM). This network has an accuracy of 98.3%.

OpenCV functions are used for color tracking. First of all a color is detected in HSV color space and then we make a mask so that no other color is detected by pi. After that we calculate real time position using OpenCV and also we calculate the area of the object. Position information helps in moving left to right and area information helps it to move forward, means that if area decreases to a certain threshold the robot moves toward it.

All the sensors used for weather report, VOC pollution and bio-medical purposes have a very good accuracy and all of them are connected to Teensy 3.6 which is better than any arduino availaible in performance and specifications also.

Final Deliverable of the Project

HW/SW integrated system

Type of Industry

IT , Manufacturing

Technologies

Artificial Intelligence(AI), Robotics

Sustainable Development Goals

Industry, Innovation and Infrastructure

Required Resources

If you need this project, please contact me on contact@adikhanofficial.com