Automated human action recognition using Kinect Sensor
Due to advances in information technology (e.g., digital video cameras, ubiquitous sensors), the automatic detection of human behaviors from video is a very recent research topic. Human detection and their corresponding behaviors have been studied under distinct perspectives in a wide variety of dis
2025-06-28 16:30:22 - Adil Khan
Automated human action recognition using Kinect Sensor
Project Area of Specialization Artificial IntelligenceProject SummaryDue to advances in information technology (e.g., digital video cameras, ubiquitous sensors), the automatic detection of human behaviors from video is a very recent research topic. Human detection and their corresponding behaviors have been studied under distinct perspectives in a wide variety of disciplines, such as psychology, biomechanics and computer graphics. An action consists of action primitives and describes a, possibly cyclic, whole-body movement. Finally, activities contain a number of subsequent actions, and give an interpretation of the movement that is being performed.
The main research question underneath this work can be characterized as follows: given a succession of pictures with one or more persons performing an action, can a framework be outlined to distinguish: who is performing the action and what action was performed? The particular interest in this area has dramatically increased with the advances of information technology. In particular, depth-imaging (3D) has substantially improved in the last few years, finally reaching an affordable consumer price (e.g., Kinect for Xbox 360). As indicated by BBC news, in 2011 Microsoft Kinect was the most selling gadget on record, with its sales achieving 23 million units in 2013. One of the highlight of this sensor is the low price for consumer market and the research intensive computing. The data capturing is cheap and the data is stored in a form ready for post-processing. This is a high resolution sensor and allows data to be captured in color, grayscale, infrared and depth. Using this device, it is possible to capture multiple users at the same time. The device allows to capture gait and gesture easily.
In this project, we propose an automated human action recognition system based on Kinect sensor. The proposed system will have the ability to recognize normal and abnormal behaviours in real time. By using this system, it will be possible to identify significant joint patterns (i.e, postures) with the help of Kinect sensor. The overall system architecture can be divided into 4 phases i.e., data acquisition, pre-processing, feature extraction and classification. In this project, we also aim to develop our own dataset at Abdul Wali khan University Mardan campus and compare the results with the state-of-the-art methods. As a novel solution to automatic action recognition, challenges and application of the developed system will be explored in health care, university campuses, surveillance and activity recognition.
Project ObjectivesRecently, the area of human action recognition has been analysed by various researchers. Many solutions have been proposed regarding the processing of color images captures from simple traditional cameras. In majority of works, the authors are using binary images to represent the silhouette of an individual. These silhouettes are obtained by processing RGB images, which requires a vast number of computational steps e.g., background removal. Other research works use sensors worn by an individual on their body to capture body posture.
With the recent advances in 3D depth sensors such as Microsoft Kinect have created many opportunities for surveillance, security and entertainment. To maintain the security of both people and infrastructure, new technologies are contributing to the realization of more powerful systems that detect varying human behaviours . In order to detect varying human behaviours irrespective of lighting conditions in buildings or production facilities, researchers have extensively attempted to use depth information from 3D sensors.
The proposed work will use Kinect sensor for obtaining the skeleton of an individual. These cameras are widely available and provide high definition result. The vision system of the Kinect sensor is composed of two cameras (i.e., an RGB camera and an IR camera) with 640x480 resolution, and an IR projector that is responsible of shooting infrared rays toward the environment. The distortion degree of each ray projected against the scene is used to estimate a depth map in which each pixel value represents the distance of a specific 3D point from the Kinect.
Project Implementation MethodThe automatic human action recognition system proposed in this project will be divided into four phases i.e., data acquisition, pre-processing, feature extraction and classification. We will use the skeleton joint position obtained from software development kit (SDK) of the Kinect sensor. The project is focused on how the Kinect sensor captures the 3D information of a scene and recognizes the action being performed by the human body by retrieving the depth image information and real-time skeletal tracking.
The recorded actions will be divided into two categories i.e., normal and abnormal behaviours. To perform skeleton tracking, the Kinect sensor detects 24 joints in the human body which represent different body parts. Utilizing this 3D joint information, the Kinect identifies the gestures and actions being performed by the human body. Pre-processing will be done in order to remove noise from the data. Finally, classification will be done using neural algorithm. The implementation for the proposed system will be done using Python with OPEN CV.
Benefits of the ProjectAutomated action recognition using Kinect has a great number of applications in various fields like robotics, health care, computer science, and many commercial areas. Users can play games by utilizing their own body movements. Indeed, even in the field of medicine, the doctor can operate on a patient from a remote location by using Kinect. Kinect sensors has a wide range of applications in software development for human interaction. Action recognition using Kinect has been a great advancement in computer vision based HCI (Human Computer Interaction).
Technical Details of Final DeliverableThe possible main contributions of this work includes:
-
a systematic literature review on automated human action recognition. This study will help in identifying the underlying challenges like pose, illumination, clutter and the proposed solution in the literature. We will try to use these results for comparing the outcome of our project to existing methods, to bring novelty to the existing system and to cover the research gape;
-
developing a prototype of the proposed system that will be able to work in real time for performing automatic action recognition. The system can be deployed in universities, schools, banks, health care and other surveillance areas for abnormal action recognition;
-
will also try to publish articles from the proposed study i.e., a systematic literature review, and another covering the technical detail carried out in this project.
| Item Name | Type | No. of Units | Per Unit Cost (in Rs) | Total (in Rs) |
|---|---|---|---|---|
| Total in (Rs) | 69000 | |||
| Kinect Sensor | Equipment | 2 | 11000 | 22000 |
| Kinect adapter for PC | Equipment | 2 | 14000 | 28000 |
| Surveillance Cam | Equipment | 2 | 6000 | 12000 |
| raspberry pi B+ | Equipment | 1 | 7000 | 7000 |