When analyzing fighter aircraft, collecting data from head-up display videos can be very time consuming. From personal experience, the data collection process occurs by rewatching these videos during a debrief session and then taking notes at particular time stamps. This process proves to be insufficient when the videos are lengthy enough that it becomes difficult to review in detail, or there are just too many to analyze in the bigger picture. This can be a familiar situation for those who have explored flight tests, such as target location error (TLE) or sensor testing, in the past. In addition, human error comes to play when pilots are tasked with performing data collection during their flights. According to Wilson and others (2019), fatigue was shown to be a contributing factor in at least 20% of accidents that were related to transportation. This is already an indication that there are implications within aviation as well. Drowsiness and fatigue related symptoms have become such a topic of interest that a study was done to collect pilot physiological information, such as photoplethysmogram and electrocardiograms, in flight to confirm the negative effects (Wilson, 2019). Thus, the majority of the data collection work load is on analysts on the ground, which also deal with issues of complacency with redundancy. Therefore, it is necessary to find a new method of data collection that is much more efficient and accurate. In this project, an attempt will be made at performing optical character recognition to collect data from declassified head-up display videos from fighter aircraft.
In other instances, there have been artificial intelligence (AI) related errors in the military theater that have led to tragedies and loss of life. This can, in part, be explained by inaccurate data fed into AI machines that were intended to provide help. In a particular incident on March 22, 2003, the Royal Air Force lost two soldiers because of errors that were attributed to misclassification and the improper definition of rules and autonomous behaviors of a potential enemy weapon (Atherton, 2022). Utilizing other programs, such as OCR, to assist in the reduction of human error in input data can significantly reduce AI related errors, and perform work more efficiently overall. There is very much evidence backing the need for the ability to perform OCR on aircraft data to reduce human error and overall workload.
The main objective of this project is to utilize optical character recognition to recognize text and/or numbers in streaming aircraft HUD videos. Then, this model will be deployed via a web-based platform.
- Review the data source provided on this page. The videos are representative of the input data. The user is able to perform data exploration, visualization, and understand some of the cleaning measures that were taken.
- Review the source code for the OCR model. The preprocessing methods for the video are shown, as well the definitions of each function required to initialize and run the model.
- Simply click the Auto Import Data button. This will load the necessary libraries and packages, as well the video itself into the source code. The user will see a message that reads: "Loading libraries, packages, and data..." This will indicate the desired actions are happening successfully.
- Click the Run OCR button. This will run the rest of the model and print the output on the screen. The user will expect to see a message that reads: "OCR Action in Progress..." Furthermore, the user will see printed frames, as well as some output in a numbers array. There will be additional output that can be viewed in the terminal of the operating system.
Note: Although it is not mandatory, it may be helpful to have the terminal open while using this software.
Note: At any time after running the OCR model, the user will be able to click Stop at the top right corner of the page to stop the model from continuing.
Atherton, K. (2022, May 6). Understanding the errors introduced by military AI applications. Brookings. https://www.brookings.edu/techstream/understanding-the-errors-introduced-by-military-ai-applications/ [DontGetShot]. (2023, February 12). Michigan UFO Declassified F-16 HUD Footage [Video]. YouTube. https://www.youtube.com/watch?v=GZt-lordqBE&ab_channel=DontGetShot Hamad, K. A., & Kaya, M. (2016). A detailed analysis of optical character recognition technology. International Journal of Applied Mathematics, Electronics and Computers, 244-249. https://doi.org/10.18100/ijamec.270374 Wilson, N., Guragain, B., Verma, A., Archer, L., & Tavakolian, K. (2019). Blending human and machine: Feasibility of measuring fatigue through the aviation headset. Human Factors: The Journal of the Human Factors and Ergonomics Society, 62(4). https://doi.org/10.1177/0018720819849783