AI Based Driver Assistance Algorithm and Real-Time Testing for Datsun Car
The aim of the proposed project is to develop and deploy a low cost driver assistance system that would be capable to autonomously steer the steering wheel of an actual Datsun 120Y car. The far goal is to convert the Datsun car shown in Figure 1 into a driver assistance vehicle being able to complet
2025-06-28 16:30:10 - Adil Khan
AI Based Driver Assistance Algorithm and Real-Time Testing for Datsun Car
Project Area of Specialization Artificial IntelligenceProject SummaryThe aim of the proposed project is to develop and deploy a low cost driver assistance system that would be capable to autonomously steer the steering wheel of an actual Datsun 120Y car. The far goal is to convert the Datsun car shown in Figure 1 into a driver assistance vehicle being able to completely take over the steering function of the car using state-of-the-art computer vision and deep learning techniques. We are inspired by the methodology adopted by Tesla autonomous cars, which mainly rely on multiple cameras to achieve self driving capabilities. On the same lines, the driver assistance system that we are pitching in this proposal rely on the video feed from a dash cam, which will then be fed to a convolutional neural network in order to autonomously steer the car.
The team working on this project at HITEC University consists of five members and project supervisor. Due to limitation of Ignite NGIRI funding on the number of students, the overall project is divided into two projects:
Project 1: AI Based Driver Assistance Algorithm and Real-Time Testing for Datsun Car. (this project)
Project 2: Instrumentation and Control System for AI Based Driver Assistance System Onboard Datsun Car. (the other project)
About this Project (Project 1):
This proposal is for Project 1 which deals with the algorithm and software related to AI Based Driver Assistance System. In this proposal, the technical, financial and other details related to this project are discussed. Figure 2 depicts a high level diagram showing workflow of steering wheel angle prediction. The input from the camera is first preprocessed using image processing algorithms to improve the quality of input image frames. The processed frames are then fed to a trained deep neural network which will predict the steering angle. The predicted steering angle will then be given to the hardware (Project 2) that would maneuver the car.
- To train a convolutional neural network to perform semantic segmentation. The first objective is to train a convolutional neural network (CNN) in order to perform semantic segmentation of the road. The frames after semantic segmentation will segment the road from rest of the scene in the frames captured by the dash cam. The segmented image will be given to another convolutional neural network to predict the steering angle.
- To train a second convolutional neural network to perform steering angle prediction. The second objective is to train a second CNN to correctly predict the steering angle of the car’s steering wheel. This is one of the most crucial aspects of this project as the maneuverability of the car will depend on the predicted steering angle.
- Testing the trained CNN in a simulated environment. The designed steering angle prediction system will first be tested in a simulated environment on Karla Simulator and Udacity Simulators. This part of the project has already been done with a reasonable accuracy. This step has already been achieved and the Network was tested on Udacity Simulator with an impressive accuracy.
- Porting the trained deep network to an embedded system. Once the desired accuracy of the trained steering angle prediction CNN is achieved, the trained segmentation and steering angle prediction CNN models will the ported to a Jetson TX2 board for real-time inference.
- Testing the system in real environment. After hardware implementation of the trained CNNs on the Jetson TX2, the complete system will be tested in a real but controlled environment.
The team working on Project 1 (this project) will be responsible to develop and deploy a Driver Assistance System that can take control of the car’s steering wheel using the input from a dash cam. It is imperative to mention that the CNN which will be predicting the steering angle will get video feed from a single dash cam installed on the rear windshield of the car. One of the main challenges in this project is to train the CNN such that it predicts the steering angle with high accuracy. The team working on Project 2 will deal with the hardware part of the overall project, ensuring successful interfacing of the AI Based Driver Assistance System (software, this project) with the car.
Following are main steps illustrating implementation of this project:
Step 1:
CNN for Semantic Segmentation
The trained CNN for Semantic Segmentation would segment road, lanes and all other moving objects (other cars, people and bikes as one object). This segmentation network will be trained using online datasets like KITTI and CityScapes. In addition, the CNN will also be trained on the images collected from the controlled area where the actual car will be tested. The CNN will be trained and tested to achieve good segmentation accuracy (mIOU) while keeping computational requirements to the minimum. Some initial training and testing of semantic segmentation has already been done. A sample image showing segmentation of road located at HITEC University campus is shown in Figure 3.
Step 2: Train a Convolutional Neural Network for Steering Angle Prediction:
The segmented image will be given to a second CNN that will be trained to predict the steering angle. Figure 4 illustrates the technique to predict the steering angle. Before training this CNN, the car will be driven by a human to collect steering angle data and the corresponding image frames. These frames will consists of semantic segmented images on which the roads would appear with a specific color, for example, in blue color as shown in Figure 3. Based on the steering angle data and its corresponding road segmented frames, the CNN will be trained as depicted in Figure 4. The steering angle of the car will be measured using an Optical Shaft Encoder (OSE).
Figure 5 show the inference workflow to predict the steering angle of the car. The input frames from the dash cam will first be given to the trained CNN that will produce new frames in which the road will be segmented from the rest of the scene. The segmented frames will then be given to the trained CNN that will predict the current steering angle required to correctly move the steering wheel ensuring that the car stays on the road. To get a scalar value of the predicted steering angle, a three layer fully connected neural network will be used after the convolution and polling layers of the CNN.
- This project can be a significant milestone towards Pakistan's contribution in developing autonomous driving system using state-of-the-art artificial intelligence technologies.
- This technology can be further developed to autonomous surveillance vehicles or border patrol vehicles.
The final deliverable will include a Driver Assistance System to autonomously steer the Datsun car n a controlled environment. To achieve this goal, the accuracy (mIOU) of the segmentation CNN should be above 90% and the accuracy of steering angle prediction CNN should be above 85%.
Final Deliverable of the Project HW/SW integrated systemCore Industry ManufacturingOther IndustriesCore Technology Artificial Intelligence(AI)Other TechnologiesSustainable Development Goals Industry, Innovation and InfrastructureRequired Resources| Item Name | Type | No. of Units | Per Unit Cost (in Rs) | Total (in Rs) |
|---|---|---|---|---|
| Total in (Rs) | 78800 | |||
| NVIDIA Tx2 Board | Equipment | 1 | 70000 | 70000 |
| Raspberry pi | Miscellaneous | 1 | 6800 | 6800 |
| Wires, fuses, electronic components, etc. | Miscellaneous | 1 | 2000 | 2000 |