Multi-modal Sensor Fusion (Camera, LiDAR) for Environmental Depth Perception for Autonomous Vehicles

Project Title

Multi-modal Sensor Fusion (Camera, LiDAR) for Environmental Depth Perception for Autonomous Vehicles

Project Area of Specialization Artificial IntelligenceProject Summary

Our final year project is based on data acquisition, and visual perception for autonomous vehicles. The main objective is to develop sensor fusion model by gathering results from LIDAR and stereo-cameras and fusing it to estimate a depth map of short range. At the end of the project, we will be able to perform LIDAR and stereo-camera calibration and create an algorithm that will apply sensor acquisition techniques utilizing LIDAR, computer vision and deep learning models to estimate depth maps obtained in real time. We will also use coding libraries such as OpenCV in our perception model. Completion of the project also includes mounting and acquiring data from sensors and testing the optimum algorithm on Jetson development kit.

Project Objectives

• Perception: Our objective is to build a perception system which can create a real time depth map of its surroundings. 
• Enhancing Skillset: Building this project will require us to gain knowledge about autonomous systems and computer vision which will be a great learning experience for us.
• Industrial Implementation: By using the learning outcomes from this project and applying it in industry will help speed up the production of autonomous cars in Pakistan.
• Research: Researching the development and use of autonomous vehicles in Pakistan. 

Project Implementation Method

  

1: Hardware & Sourcing :  

• Acquiring ATV, LIDAR And stereo-cameras.
• Designing the sensor brackets in solidworks.
• Mounting those brackets on an ATV.

2: Software & Design: 

•  Acquire data from sensors.
•  Calibrating the sensors.
•  Sensor synchronization: Synchronizing the   frame rate of both LIDAR and stereo-camera.
• Transforming sensor inputs.

3: Implementation of Sensor Fusion:

• Generating a depth map and using kalmal filtering.
• Improving depth map.

4: Depth Map Estimation 

Benefits of the Project

Technical Details of Final Deliverable

• Sensing system utilizing LIDAR and stereo cameras.
• Short range depth perception. 
• Real time depth perception in a controlled environment.

Final Deliverable of the Project Software SystemCore Industry TransportationOther IndustriesCore Technology Artificial Intelligence(AI)Other Technologies OthersSustainable Development Goals Good Health and Well-Being for People, Industry, Innovation and Infrastructure, Reduced Inequality, Climate ActionRequired Resources

Item Name	Type	No. of Units	Per Unit Cost (in Rs)	Total (in Rs)
			Total in (Rs)	79900
Benewake: Long-Distance LiDAR Module TF03	Equipment	1	40000	40000
Nvidia: Jetson Nano Developer Kit B01 - 4GB	Equipment	1	30000	30000
Sensor Brackets	Miscellaneous	3	3300	9900