Determining the Efficacy of Haptic Feedback and Virtual Reality in Flight Simulation Training Devices

Project Title

Determining the Efficacy of Haptic Feedback and Virtual Reality in Flight Simulation Training Devices

Project Area of Specialization Wearables and ImplantableProject Summary

A flight simulator is a miniature version of an aircraft used to simulate the flight experience for any specific aircraft and equip the pilot with sufficient training to reduce the chances of crisis or fatalities. High fidelity is of paramount importance in flight simulation, i.e. it should accurately mimic the actual flight experience. Ensuring high fidelity in such aviation grade equipment give rise to portability problems, developmental complexity, financial constraints, customization and configuration issues, and multiple simulators are needed for multiple versions of the same aircraft. Therefore, despite being effective, flight simulators are still inefficient, and developing an alternative to flight simulators is imperative.

Haptic feedback has been a source of fascination and innovation in the gaming industry ever since possibilities of merging it with Virtual Reality arose. However, unlike the gaming industry, where low fidelity experiences do not compromise on entertainment, higher fidelity increases the training value in flight simulation. In this project, VR replaced the visuals, while Haptic feedback imitated the sensation felt by the pilots. Involving VR meant that the user could switch between aircrafts by customizing the virtual environment at a minimal cost. This project holds the potential to reduce the size of flight simulators, make them portable and allow the pilots to carry their simulator with them to train at their own leisure.

This project investigated three Haptic feedback methods -- vibration motor, gear motor and electric muscle stimulation (EMS) -- to identify which method may serve for high fidelity flight simulators. Leap Motion was used to translate physical hand motion into the virtual environment, developed in Unity 3D. On interaction with an object, the controller triggered a relevant feedback. We identified that vibration-based is the least whereas a hybrid of gear motor and EMS is the most appropriate method for our application.

Project Objectives

Based on the literature review, the problem statement for this project was defined as follows:

Our objective is to determine the role and efficacy of Haptic Feedback and Virtual Reality technologies in the development of Flight Simulation Training Devices (FSTD).

Since this project was pitched by the Pakistan Air Force, the application under consideration was based on flight simulators but a general application could be a training of any sort, in an industrial setting or to help train surgeons. the primary purpose of this project was to investigate if Haptic Feedback can replace conventional flight simulation training methods in any situation or application. Secondly, in the event that Haptic feedback could be used as an alternative, which of the four methods commonly used (vibration, gear motors, muscle stimulation and pneumatics) are more suited for this application. In order to evaluate its effectiveness, the fidelity of the developed Haptics and VR experience also had to be assessed and compared with actual FSTD experience.

The client had listed portability problems, developmental complexity, financial constraints, customization and configuration issues as worth solving. Therefore, our prototype had to be lightweight, portable, re-configurable and simpler to use. The prototype must be able to lock the user's elbow if the user interacts with a rigid object in the VR environment. The prototype must also lock the user's finger joint once the maximum limit of a push button has been reached. Our solution also compared tactile feedback generated by vibration motors with Electric Muscle Stimulation. So when the user interacts with an object they will be able to feel either vibration or current on their fingertips, depending on the prototype they are wearing.

Project Implementation Method

In this project, a push button, toggle switch and a wall were created in 3D using the game engine, Unity 3D. Leap Motion device was used to translate motions from physical world to the digital world because it was compatible with Unity 3D and the SDK was available. It was cost effective and small and could be head mounted. Motion capturing and projection into the 3D environment was almost seamless when the user moved one or both of their hands in space. However, some hardware and software limitations affected the overall prototype performance which were removed when the finger locking structure was mounted on the index finger while the rest of the hand remained uncovered.

To recreate the sensation of touch when the hand is in contact with a virtual object, Electric Muscle Stimulation was applied using the muscle stimulator ComfyStim EV-860, on the user’s index fingertip. A pair of electrodes consisting of an anode and a cathode was placed on the inner side of finger and their size reduced so that the current could be localized to precise regions where the contact sensation is to be produced. 0.0312 A current with 100 Hz frequency and 300 microseconds pulse width was applied on the user’s finger muscles and produce a contact sensation of reasonable fidelity

Finger locking mechanism for the index finger bone, i.e proximal, medial and proximal, and supporting structure was built on SolidWorks and then 3D printed, it also used a micro-servo motor and fish wires. The support was mounted on the back of user’s hand, a small spring was attached from its one end and the micro-servo motor was placed at the other end of lock structure, nearer to the fingers, and a metal pinion gear, with a hole drilled, was glued over the servo pinion to ensure sturdiness. From one end of the spring, a fish wire was tied which passed through the metal pinion hole. The other end of the wire was tied onto the printed distal bone. During interaction of the virtual finger with a push button, the spring simulated the effect of reactive forces on the finger when an actual button is pressed. When the button is fully pressed, the pinion rotates 90 degrees and locks the fish wire in its place, and consequently, the distal bone gets locked, too.

The elbow lock was designed using a disc brake mechanism. This round disc had a hole drilled at every 15 degrees and all holes were at the same radial distance from the center. These 15 degrees would separate two adjacent holes without weakening the disc. A pin, controlled by a high torque servo motor, had two states which would lock the disc depending on the interactions with objects in the virtual world.

Benefits of the Project

Boeing reports that approximately 80% of the airplane accidents occur due to human error while the rest occur due to machine failure. Another report from PlaneCrashInfo.com and The Telegraph UK suggests that from 1950-2009, an average of about 50% of the total fatal accidents were caused by pilot error, in particular. It is safe to assume that a significant number of such accidents and fatalities may have been avoided if the pilot was trained enough to deal with contingency situations. This project focuses on investigating the application and efficacy of haptic feedback technology in flight simulation training devices and hence its primary purpose is to recreate training systems for pilots to minimize risks and casualties. Although this project focused on one tangible outcome, it can be expanded to further human-centric applications largely related to training. Haptic feedback can allow an architect to get the feeling of the structure that they are designing and can enable an interior designer to select from options over virtual reality and move around furniture to suit any preference. It can be used by medical professionals to practice surgery without cadavers or models, and even feel the stiffness of a virtual organ under forceps. It can be used by companies to train new employees to handle expensive equipment or even test equipment before purchase to avoid any accidents during initial usage. It holds the potential to make learning more engaging and interactive for growing kids, and can even be used by vocational training institutions. Additionally, it has the capability to assist the mobility of the visually impaired and help them navigate normally by haptic cues and voice feedback. Furthermore, modern prosthetics paired with haptic technology has the capability to provide a sense of touch to an amputee when the sensors in the artificial tendons fire specified impulses to the user’s central nervous system.

Technical Details of Final Deliverable

For tactile feedback, coin vibration motors were used to provide the sensation of contact on user’s fingertips so that the user can experience a “Hapto-virtual environment”. The vibration intensity and frequency was varied by changing the frequency and PWM of the signal. Later it was noted that the contact sensation was very fragile and felt external regardless of the features of signal and, therefore, it was decided to shift towards other alternative.

The aim was to provide feedback which feels more realistic to the user, the design was switched towards electric muscle stimulation (EMS). For this application small size pair of electrode were used and placed on user’s distal and proximal phalanges of right hand finger. The wires from the electrodes travel from the gap between the index finger to the back of the hand and connects to the circuitry. This technique was preferred as there is much room for further work in EMS as compare to vibration motors.

By varying frequencies and voltage intensities of the provided signal on user’s skin surface  multiple sensations of touch can be simulated but keeping the scope of this project in mind, only one base signal was used for contact sensation with frequency of 100 Hz, pulse width of 300 us, and current of 0.0312 A.  The circuitry for EMS tactile system consists of an electric muscle stimulator and a relay-based switching circuit which was required to automate the process.

For dexterous interaction of virtual objects, the objects need to be virtually modeled according to the predetermined or physical hardness or deformation when force is applied, to ensure a better Haptic feedback response. The finger locking mechanism was based on micro-servo motor in which motor acts directly in the locking process by rotating it’s shaft to 90 degrees when the controller sends a signal. A fish wire was passed through a hole in the metal shaft or pinion of the motor which gets lock as shaft rotates due to torque. The design was inspired by the Novel Digital glove:  Mechanical structure to achieve the physical limitations.The tangential force and the radius of the rotating shaft are inversely proportional to each other so if radius decreases, then force increases. Therefore, the radius is kept very small so that the tangential force on the rotating shaft get magnified enough to lock finger movements.

Locking mechanism system for elbow joint  was needed to produce high-torque, the design needed to be simpler and light in terms of weight. The mechanism was based on ratchet wheel system, controlled by a digital switch which lock the ratchet wheel when the hand interacts with an object in the virtual environment.. After the interaction, the controller would send a signal to the switch which would jam the pawl against the tooth boundary, consequently limiting backward motion of arm.

Final Deliverable of the Project Hardware SystemType of Industry Education , Medical , Manufacturing , Others , Security Technologies Augmented & Virtual Reality, 3D/4D Printing, Robotics, Wearables and ImplantablesSustainable Development Goals Quality Education, Industry, Innovation and InfrastructureRequired Resources

Item Name	Type	No. of Units	Per Unit Cost (in Rs)	Total (in Rs)
			Total in (Rs)	14514
Leap Motion	Equipment	1	4500	4500
Comfy Stim EV-860 with electrodess	Equipment	1	6000	6000
HM-10 Bluetooth module	Equipment	1	850	850
MG-996 Hi-torque motor	Equipment	1	500	500
Vibration motors	Equipment	5	90	450
EMG electrodes	Equipment	1	120	120
solid state relay	Equipment	1	380	380
Arduino cable	Miscellaneous	1	75	75
Digital potentiometer	Equipment	1	459	459
Mosfet driver IC	Equipment	2	250	500
N-channel Mosfet	Equipment	2	210	420
Sliding potentiometer	Equipment	2	60	120
Double tape	Miscellaneous	1	70	70
Daraz Shipping fee	Miscellaneous	1	70	70