Developing a computer vision based system for autonomous taxiing of aircraft
Abstract
Authors of this paper propose a computer vision based autonomous system for the taxiing of an aircraft in the real world. The system integrates both lane detection and collision detection and avoidance models. The lane detection component employs a segmentation model consisting of two parallel architectures. An airport dataset is proposed, and the collision detection model is evaluated with it to avoid collision with any ground vehicle. The lane detection model identifies the aircraft’s path and transmits control signals to the steer-control algorithm. The steer-control algorithm, in turn, utilizes a controller to guide the aircraft along the central line with 0.013 cm resolution. To determine the most effective controller, a comparative analysis is conducted, ultimately highlighting the Linear Quadratic Regulator (LQR) as the superior choice, boasting an average deviation of 0.26 cm from the central line. In parallel, the collision detection model is also compared with other state-of-the-art models on the same dataset and proved its superiority. A detailed study is conducted in different lighting conditions to prove the efficacy of the proposed system. It is observed that lane detection and collision avoidance modules achieve true positive rates of 92.59% and 85.19%, respectively.
First published online 4 January 2024
Keyword : autonomous taxi, lane detection, lane navigation, object detection, collision avoidance, airport dataset
How to Cite
Gaikwad, P., Mukhopadhyay, A., Muraleedharan, A., Mitra, M., & Biswas, P. (2023). Developing a computer vision based system for autonomous taxiing of aircraft. Aviation, 27(4), 248–258. https://doi.org/10.3846/aviation.2023.20588
This work is licensed under a Creative Commons Attribution 4.0 International License.
References
Acharya, R. (2014). Understanding satellite navigation. Academic Press. https://doi.org/10.1016/B978-0-12-799949-4.00002-6
Airbus Aircraft. (2022). Global Services Forecast (GSF) | 2022–2041. https://aircraft.airbus.com/en/market/global-services-forecast-gsf-2022-2041
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017). Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12), 2481–2495. https://doi.org/10.1109/TPAMI.2016.2644615
BBC News. (2018). 2017 safest year for air travel as fatalities fall. https://www.bbc.com/news/business-42538053
Boeing. (2023). Statistical summary of commercial jet airplane accidents-worldwide operations 1959–2022. https://www.faa.gov/sites/faa.gov/files/2023-10/statsum_summary_2022.pdf
Chandigarh Traffic Police. (2022). Safe and responsible driving. Chandigarh Traffic Police.
Cheng, V. H., Sharma, V., & Foyle, D. C. (2001). A study of aircraft taxi performance for enhancing airport surface traffic control. IEEE Transactions on Intelligent Transportation Systems, 2(2), 39–54. https://doi.org/10.1109/6979.928715
Cox, J. (2014). Ask the captain: Making time on the taxiways. https://www.usatoday.com/story/travel/columnist/cox/2014/11/23/airport-airplane-taxi-speed/19334661/
Daidzic, N. E. (2017). Determination of taxiing resistances for transport category airplane tractive propulsion. Advances in Aircraft and Spacecraft Science, 4(6), 651–677.
Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05) (Vol. 1, pp. 886–893). IEEE. https://doi.org/10.1109/CVPR.2005.177
Dow, J. H. (2003). U.S. Patent No. 6,600,992. U.S. Patent and Trademark Office.
Felzenszwalb, P. F., Girshick, R. B., McAllester, D., & Ramanan, D. (2009). Object detection with discriminatively trained part-based models. IEEE Transactions on Pattern analysis and Machine Intelligence, 32(9), 1627–1645. https://doi.org/10.1109/TPAMI.2009.167
Girshick, R. (2015). Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) (pp. 1440–1448). IEEE. https://doi.org/10.1109/ICCV.2015.169
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2015). Region-based convolutional networks for accurate object detection and segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(1), 142–158. https://doi.org/10.1109/TPAMI.2015.2437384
Hakkeling-Mesland, M. Y., Beek, B. V., Bussink, F. J. L., Mulder, M., & van Paassen, M. M. (2010). Evaluation of an autonomous taxi solution for airport operations during low visibility conditions. In Proceedings of the 9th USA/Europe Air Traffic Management Research and Development Seminar (ATM 2011). ResearchGate.
Ismail, A. H., Azmi, M. S. M., Hashim, M. A., Ayob, M. N., Hashim, M. M., & Hassrizal, H. B. (2013). Development of a webcam based lux meter. In 2013 IEEE Symposium on Computers & Informatics (ISCI) (pp. 70–74). IEEE. https://doi.org/10.1109/ISCI.2013.6612378
Kang, D. J., Choi, J. W., & Kweon, I. S. (1996). Finding and tracking road lanes using “line-snakes”. In Proceedings of Conference on Intelligent Vehicles (pp. 189–194). IEEE. https://doi.org/10.1109/IVS.1996.566376
LeBlanc, E. L. (2001). U.S. Patent No. 6,305,484. U.S. Patent and Trademark Office.
Lee, J., & Yim, S. (2023). Comparative study of path tracking controllers on low friction roads for autonomous vehicles. Machines, 11(3), 403. https://doi.org/10.3390/machines11030403
Liu, C., & Ferrari, S. (2019). Vision-guided planning and control for autonomous taxiing via convolutional neural networks. In AIAA Scitech 2019 Forum (p. 0928). Aerospace Research Central. https://doi.org/10.2514/6.2019-0928
Mukhopadhyay, A., Mukherjee, I., & Biswas, P. (2019). Comparing shape descriptor methods for different color space and lighting conditions. AI EDAM, 33(4), 389–398. https://doi.org/10.1017/S0890060419000398
Mukhopadhyay, A., Rajshekar Reddy, G. S., Mukherjee, I., Kumar Gopa, G., Pena-Rios, A., & Biswas, P. (2021). Generating synthetic data for deep learning using VR digital twin. In Proceedings of the 2021 5th International Conference on Cloud and Big Data Computing (pp. 52–56). ACM Digital Library. https://doi.org/10.1145/3481646.3481655
Mukhopadhyay, A., Reddy, G. R., Saluja, K. S., Ghosh, S., Peña-Rios, A., Gopal, G., & Biswas, P. (2022a). Virtual-reality-based digital twin of office spaces with social distance measurement feature. Virtual Reality & Intelligent Hardware, 4(1), 55–75. https://doi.org/10.1016/j.vrih.2022.01.004
Mukhopadhyay, A., Murthy, L. R. D., Mukherjee, I., & Biswas, P. (2022b). A hybrid lane detection model for wild road conditions. IEEE Transactions on Artificial Intelligence, 4(6). https://doi.org/10.1109/TAI.2022.3212347
Mukhopadhyay, A., Sharma, V. K., Tatyarao, P. G., Shah, A. K., Rao, A. M., Subin, P. R., & Biswas, P. (2023). A comparison study between XR interfaces for driver assistance in take over request. Transportation Engineering, 11, Article 100159. https://doi.org/10.1016/j.treng.2022.100159
Ogunwa, T. T., & Abdullah, E. J. (2016). Flight dynamics and control modelling of damaged asymmetric aircraft. IOP Conference Series: Materials Science and Engineering, 152(1), Article 012022. https://doi.org/10.1088/1757-899X/152/1/012022
Pizzati, F., Allodi, M., Barrera, A., & García, F. (2020). Lane detection and classification using cascaded CNNs. In Computer Aided Systems Theory – EUROCAST 2019. Lecture Notes in Computer Science (Vol. 12014, pp. 95–103). Springer. https://doi.org/10.1007/978-3-030-45096-0_12
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 779–788). IEEE. https://doi.org/10.1109/CVPR.2016.91
Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster R-CNN: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems, 28.
Sun, T. Y., Tsai, S. J., & Chan, V. (2006). HSI color model based lane-marking detection. In 2006 IEEE Intelligent Transportation Systems Conference (pp. 1168–1172). IEEE.
TurtleBot3. (2023). TurtleBot3. https://www.turtlebot.com/turtlebot3/
Uijlings, J. R., Van De Sande, K. E., Gevers, T., & Smeulders, A. W. (2013). Selective search for object recognition. International Journal of Computer Vision, 104, 154–171. https://doi.org/10.1007/s11263-013-0620-5
Zammit, C., & Zammmit-Mangion, D. (2014). A control technique for automatic taxi in fixed wing. In 52nd Aerospace Sciences Meeting (p. 1163). Aerospace Research Central. https://doi.org/10.2514/6.2014-1163
Zhang, Y., Poupart-Lafarge, G., Teng, H., Wilhelm, J., Jeannin, J. B., Ozay, N., & Scholte, E. (2020). A software architecture for autonomous taxiing of aircraft. In AIAA Scitech 2020 Forum (p. 0139). Aerospace Research Central. https://doi.org/10.2514/6.2020-0139
Airbus Aircraft. (2022). Global Services Forecast (GSF) | 2022–2041. https://aircraft.airbus.com/en/market/global-services-forecast-gsf-2022-2041
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017). Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12), 2481–2495. https://doi.org/10.1109/TPAMI.2016.2644615
BBC News. (2018). 2017 safest year for air travel as fatalities fall. https://www.bbc.com/news/business-42538053
Boeing. (2023). Statistical summary of commercial jet airplane accidents-worldwide operations 1959–2022. https://www.faa.gov/sites/faa.gov/files/2023-10/statsum_summary_2022.pdf
Chandigarh Traffic Police. (2022). Safe and responsible driving. Chandigarh Traffic Police.
Cheng, V. H., Sharma, V., & Foyle, D. C. (2001). A study of aircraft taxi performance for enhancing airport surface traffic control. IEEE Transactions on Intelligent Transportation Systems, 2(2), 39–54. https://doi.org/10.1109/6979.928715
Cox, J. (2014). Ask the captain: Making time on the taxiways. https://www.usatoday.com/story/travel/columnist/cox/2014/11/23/airport-airplane-taxi-speed/19334661/
Daidzic, N. E. (2017). Determination of taxiing resistances for transport category airplane tractive propulsion. Advances in Aircraft and Spacecraft Science, 4(6), 651–677.
Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05) (Vol. 1, pp. 886–893). IEEE. https://doi.org/10.1109/CVPR.2005.177
Dow, J. H. (2003). U.S. Patent No. 6,600,992. U.S. Patent and Trademark Office.
Felzenszwalb, P. F., Girshick, R. B., McAllester, D., & Ramanan, D. (2009). Object detection with discriminatively trained part-based models. IEEE Transactions on Pattern analysis and Machine Intelligence, 32(9), 1627–1645. https://doi.org/10.1109/TPAMI.2009.167
Girshick, R. (2015). Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) (pp. 1440–1448). IEEE. https://doi.org/10.1109/ICCV.2015.169
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2015). Region-based convolutional networks for accurate object detection and segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(1), 142–158. https://doi.org/10.1109/TPAMI.2015.2437384
Hakkeling-Mesland, M. Y., Beek, B. V., Bussink, F. J. L., Mulder, M., & van Paassen, M. M. (2010). Evaluation of an autonomous taxi solution for airport operations during low visibility conditions. In Proceedings of the 9th USA/Europe Air Traffic Management Research and Development Seminar (ATM 2011). ResearchGate.
Ismail, A. H., Azmi, M. S. M., Hashim, M. A., Ayob, M. N., Hashim, M. M., & Hassrizal, H. B. (2013). Development of a webcam based lux meter. In 2013 IEEE Symposium on Computers & Informatics (ISCI) (pp. 70–74). IEEE. https://doi.org/10.1109/ISCI.2013.6612378
Kang, D. J., Choi, J. W., & Kweon, I. S. (1996). Finding and tracking road lanes using “line-snakes”. In Proceedings of Conference on Intelligent Vehicles (pp. 189–194). IEEE. https://doi.org/10.1109/IVS.1996.566376
LeBlanc, E. L. (2001). U.S. Patent No. 6,305,484. U.S. Patent and Trademark Office.
Lee, J., & Yim, S. (2023). Comparative study of path tracking controllers on low friction roads for autonomous vehicles. Machines, 11(3), 403. https://doi.org/10.3390/machines11030403
Liu, C., & Ferrari, S. (2019). Vision-guided planning and control for autonomous taxiing via convolutional neural networks. In AIAA Scitech 2019 Forum (p. 0928). Aerospace Research Central. https://doi.org/10.2514/6.2019-0928
Mukhopadhyay, A., Mukherjee, I., & Biswas, P. (2019). Comparing shape descriptor methods for different color space and lighting conditions. AI EDAM, 33(4), 389–398. https://doi.org/10.1017/S0890060419000398
Mukhopadhyay, A., Rajshekar Reddy, G. S., Mukherjee, I., Kumar Gopa, G., Pena-Rios, A., & Biswas, P. (2021). Generating synthetic data for deep learning using VR digital twin. In Proceedings of the 2021 5th International Conference on Cloud and Big Data Computing (pp. 52–56). ACM Digital Library. https://doi.org/10.1145/3481646.3481655
Mukhopadhyay, A., Reddy, G. R., Saluja, K. S., Ghosh, S., Peña-Rios, A., Gopal, G., & Biswas, P. (2022a). Virtual-reality-based digital twin of office spaces with social distance measurement feature. Virtual Reality & Intelligent Hardware, 4(1), 55–75. https://doi.org/10.1016/j.vrih.2022.01.004
Mukhopadhyay, A., Murthy, L. R. D., Mukherjee, I., & Biswas, P. (2022b). A hybrid lane detection model for wild road conditions. IEEE Transactions on Artificial Intelligence, 4(6). https://doi.org/10.1109/TAI.2022.3212347
Mukhopadhyay, A., Sharma, V. K., Tatyarao, P. G., Shah, A. K., Rao, A. M., Subin, P. R., & Biswas, P. (2023). A comparison study between XR interfaces for driver assistance in take over request. Transportation Engineering, 11, Article 100159. https://doi.org/10.1016/j.treng.2022.100159
Ogunwa, T. T., & Abdullah, E. J. (2016). Flight dynamics and control modelling of damaged asymmetric aircraft. IOP Conference Series: Materials Science and Engineering, 152(1), Article 012022. https://doi.org/10.1088/1757-899X/152/1/012022
Pizzati, F., Allodi, M., Barrera, A., & García, F. (2020). Lane detection and classification using cascaded CNNs. In Computer Aided Systems Theory – EUROCAST 2019. Lecture Notes in Computer Science (Vol. 12014, pp. 95–103). Springer. https://doi.org/10.1007/978-3-030-45096-0_12
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 779–788). IEEE. https://doi.org/10.1109/CVPR.2016.91
Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster R-CNN: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems, 28.
Sun, T. Y., Tsai, S. J., & Chan, V. (2006). HSI color model based lane-marking detection. In 2006 IEEE Intelligent Transportation Systems Conference (pp. 1168–1172). IEEE.
TurtleBot3. (2023). TurtleBot3. https://www.turtlebot.com/turtlebot3/
Uijlings, J. R., Van De Sande, K. E., Gevers, T., & Smeulders, A. W. (2013). Selective search for object recognition. International Journal of Computer Vision, 104, 154–171. https://doi.org/10.1007/s11263-013-0620-5
Zammit, C., & Zammmit-Mangion, D. (2014). A control technique for automatic taxi in fixed wing. In 52nd Aerospace Sciences Meeting (p. 1163). Aerospace Research Central. https://doi.org/10.2514/6.2014-1163
Zhang, Y., Poupart-Lafarge, G., Teng, H., Wilhelm, J., Jeannin, J. B., Ozay, N., & Scholte, E. (2020). A software architecture for autonomous taxiing of aircraft. In AIAA Scitech 2020 Forum (p. 0139). Aerospace Research Central. https://doi.org/10.2514/6.2020-0139