RealityScape - Realtime Object Detection with Augmented Reality Annotations

Project Title

RealityScape - Realtime Object Detection with Augmented Reality Annotations

Project Area of Specialization Artificial IntelligenceProject Summary

The overall goal of this application is to make it easier to find information about a product, such as specifications, reviews, and where to buy it. With an AR display, a user merely needs to aim their camera at an object to get interactive information in real time.

We develop this application by combining machine learning technologies with an iOS mobile application. Our mobile application shall take data from a machine learning model using Convolutional Neural Network and will display it on the mobile application using  AR whenever user performs an action.AR  annotation here is done by making use of ARkit.

Project Objectives

• Develop a product to help users make informed purchase decisions. Make it easier to lookup product details with their price.

• Implement real-time object detection on a mobile device that is efficient and gives satisfying results.

• Combine AR technology with Machine Learning.

Project Implementation Method

RealityScape is made up of two parts, object detection and AR annotation. Below we briefly discuss the two components of the iOS application that uses Deep Learning to detect objects.

• Object Detection: The user points the camera at the object. Then the frame is sent to the object detector which in turn returns the name and bounding box of the object. This deep learning model is stored on device which in turn allows the user to be able to detect objects without an internet connection. The modeL uses Swift 5 and CoreML.
• AR Annotation: After the object detection is done the bounding box coordinates are then passed to ARKit which draws an AR element containing the detected object’s information

Benefits of the Project

• Users can make informed purchase decisions.
• Easy to use for users when without the need of typing.

Technical Details of Final Deliverable

• iOS application able to detect object and annotate them in Augmented Reality.
• A Custom Dataset of 12 Products (3000 Images).
• Deep Learning Tiny-YOLO model.
• Augmented Reality graphic element.

Final Deliverable of the Project Software SystemCore Industry ITOther IndustriesCore Technology Artificial Intelligence(AI)Other Technologies Augmented & Virtual RealitySustainable Development GoalsRequired Resources

Item Name	Type	No. of Units	Per Unit Cost (in Rs)	Total (in Rs)
			Total in (Rs)	45000
Mobile Phone	Equipment	1	35000	35000
Training ML Model Service	Miscellaneous	1	5000	5000
AR element Resources	Miscellaneous	1	5000	5000