Visual stabilization for snake like robot

Snake robots are designed as hyper redundant structures to achieve the agility and adaptability of real snakes. Due to high agility and many degrees of freedom (DOF), the visual information captured from the camera mounted on the head of a snake robot is very blurry. Therefore, it becomes challengin

Project Title

Visual stabilization for snake like robot

Project Area of Specialization

Robotics

Project Summary

Snake robots are designed as hyper redundant structures to achieve the agility and adaptability of real snakes. Due to high agility and many degrees of freedom (DOF), the visual information captured from the camera mounted on the head of a snake robot is very blurry. Therefore, it becomes challenging to extract distinct features from visual data, which results in degradation of the performance and efficiency for various tasks such as object detection and recognition. There is a requirement to improve the performance of visual data by video stabilization techniques.  

Furthermore, to operate the snake robot in unknown scenes; environment mapping, path planning, and motion planning are required.  We use Simultaneous Localization and Mapping (SLAM)-based algorithms for environment mapping and path planning. To accomplish this task, we use LiDAR to capture the point cloud data of an environment for 3D-mapping. Furthermore, we use both monocular cameras and LiDAR for SLAM and path planning.

Project Objectives

1. Snake robot prototyping 
2. Video stabilization 
3. Environment mapping and localization 
4. Path planning 

Project Implementation Method

Video stabilization comprises three steps; motion estimation, motion compensation, and video completion. The video stabilization techniques are divided into two sub-categories; point feature-based and optical flow-based algorithms. The point feature-based scheme extracts key-points from each frame, finds corresponding feature pairs, and then derives the transformation between two consecutive frames[1]. In an optical flow-based technique, we estimate the motion of objects between consecutive frames of the sequence, caused by the relative movement between the object and camera. 

In this video stabilization scheme, first, we detect the feature using any feature detection algorithm and calculate the optical flow. From optical flow and feature matching, we compute an affine transformation matrix. We apply average filtering to compensate for the variations in the transformation matrix. From that, we compute the inverse transformation matrix. Finally, the inverse transformed matrix is wrapped with the original frames to obtain the stabilized video.

Furthermore, for 3D environment mapping, we use a LiDAR sensor by implementing the SLAM[2]. For path planning, we use both SLAM and stabilized video. 

[1] Y. H. Chen, H. Y. S. Lin and C. W. Su, "Full-Frame Video Stabilization via SIFT Feature Matching," 2014 Tenth International Conference on Intelligent Information Hiding and Multimedia Signal Processing, Kitakyushu, 2014, pp. 361-364, doi: 10.1109/IIH-MSP.2014.96.

[2] Taketomi, T., Uchiyama, H. & Ikeda, S. Visual SLAM algorithms: a survey from 2010 to 2016. IPSJ T Comput Vis Appl 9, 16 (2017).

Benefits of the Project

In the past century, hundreds of thousands of people died in earthquakes in Pakistan [3]. Except this, thousands of people lost their lives in many other disasters of residential building collapses and plane crashes. During most of the rescue operations, the workers could not enter into small openings of the collapsed structures, or even if they enter in such areas, they may put their lives at risk. In most of these scenarios, the wheeled, legged, and wagged robots are inoperable because of limited movements. The snake robots can navigate congested areas because of their thin structure and high agility. These facilities offer snake robots for automated search, rescue, and exploration tasks.

The environment mapping and SLAM-based path planning algorithms execute the snake robots more efficiently and independently; in an unknown environment. Using a snake robot for rescue will speed up the process of locating the survivors. With the help of a map generated by robots, rescuers can easily plan and optimize their rescue operations. 

[3 ] “Ten worst disasters in Pakistan,” Dawn, p. 1, 24-Sep-2011.

Technical Details of Final Deliverable

In this project, we deliver a snake robot with a camera and a lidar sensor mounted on its head. The camera is used to acquire visual information of the environment. This information is sent to Jetson nano (processing board) for video stabilization.

Lidar sensors are used to acquire the 3D map generation of the environment. The stabilized video and 3D map are utilized for path planning algorithms. 

As a result, a snake robot is able to plan and map its motion in an unknown environment without human intervention. 

Final Deliverable of the Project

HW/SW integrated system

Core Industry

Others

Other Industries

IT , Health , Security

Core Technology

Robotics

Other Technologies

Artificial Intelligence(AI), Robotics, Others

Sustainable Development Goals

Climate Action

Required Resources

If you need this project, please contact me on contact@adikhanofficial.com