Developing a computer vision based system for autonomous taxiing of aircraft


Authors of this paper propose a computer vision based autonomous system for the taxiing of an aircraft in the real world. The system integrates both lane detection and collision detection and avoidance models. The lane detection component employs a segmentation model consisting of two parallel architectures. An airport dataset is proposed, and the collision detection model is evaluated with it to avoid collision with any ground vehicle. The lane detection model identifies the aircraft’s path and transmits control signals to the steer-control algorithm. The steer-control algorithm, in turn, utilizes a controller to guide the aircraft along the central line with 0.013 cm resolution. To determine the most effective controller, a comparative analysis is conducted, ultimately highlighting the Linear Quadratic Regulator (LQR) as the superior choice, boasting an average deviation of 0.26 cm from the central line. In parallel, the collision detection model is also compared with other state-of-the-art models on the same dataset and proved its superiority. A detailed study is conducted in different lighting conditions to prove the efficacy of the proposed system. It is observed that lane detection and collision avoidance modules achieve true positive rates of 92.59% and 85.19%, respectively.

First published online 4 January 2024

Keyword : autonomous taxi, lane detection, lane navigation, object detection, collision avoidance, airport dataset

How to Cite
Gaikwad, P., Mukhopadhyay, A., Muraleedharan, A., Mitra, M., & Biswas, P. (2023). Developing a computer vision based system for autonomous taxiing of aircraft. Aviation, 27(4), 248–258.
Published in Issue
Dec 29, 2023
Abstract Views
PDF Downloads
Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.


Acharya, R. (2014). Understanding satellite navigation. Academic Press.

Airbus Aircraft. (2022). Global Services Forecast (GSF) | 2022–2041.

Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017). Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12), 2481–2495.

BBC News. (2018). 2017 safest year for air travel as fatalities fall.

Boeing. (2023). Statistical summary of commercial jet airplane accidents-worldwide operations 1959–2022.

Chandigarh Traffic Police. (2022). Safe and responsible driving. Chandigarh Traffic Police.

Cheng, V. H., Sharma, V., & Foyle, D. C. (2001). A study of aircraft taxi performance for enhancing airport surface traffic control. IEEE Transactions on Intelligent Transportation Systems, 2(2), 39–54.

Cox, J. (2014). Ask the captain: Making time on the taxiways.

Daidzic, N. E. (2017). Determination of taxiing resistances for transport category airplane tractive propulsion. Advances in Aircraft and Spacecraft Science, 4(6), 651–677.

Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05) (Vol. 1, pp. 886–893). IEEE.

Dow, J. H. (2003). U.S. Patent No. 6,600,992. U.S. Patent and Trademark Office.

Felzenszwalb, P. F., Girshick, R. B., McAllester, D., & Ramanan, D. (2009). Object detection with discriminatively trained part-based models. IEEE Transactions on Pattern analysis and Machine Intelligence, 32(9), 1627–1645.

Girshick, R. (2015). Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) (pp. 1440–1448). IEEE.

Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2015). Region-based convolutional networks for accurate object detection and segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(1), 142–158.

Hakkeling-Mesland, M. Y., Beek, B. V., Bussink, F. J. L., Mulder, M., & van Paassen, M. M. (2010). Evaluation of an autonomous taxi solution for airport operations during low visibility conditions. In Proceedings of the 9th USA/Europe Air Traffic Management Research and Development Seminar (ATM 2011). ResearchGate.

Ismail, A. H., Azmi, M. S. M., Hashim, M. A., Ayob, M. N., Hashim, M. M., & Hassrizal, H. B. (2013). Development of a webcam based lux meter. In 2013 IEEE Symposium on Computers & Informatics (ISCI) (pp. 70–74). IEEE.

Kang, D. J., Choi, J. W., & Kweon, I. S. (1996). Finding and tracking road lanes using “line-snakes”. In Proceedings of Conference on Intelligent Vehicles (pp. 189–194). IEEE.

LeBlanc, E. L. (2001). U.S. Patent No. 6,305,484. U.S. Patent and Trademark Office.

Lee, J., & Yim, S. (2023). Comparative study of path tracking controllers on low friction roads for autonomous vehicles. Machines, 11(3), 403.

Liu, C., & Ferrari, S. (2019). Vision-guided planning and control for autonomous taxiing via convolutional neural networks. In AIAA Scitech 2019 Forum (p. 0928). Aerospace Research Central.

Mukhopadhyay, A., Mukherjee, I., & Biswas, P. (2019). Comparing shape descriptor methods for different color space and lighting conditions. AI EDAM, 33(4), 389–398.

Mukhopadhyay, A., Rajshekar Reddy, G. S., Mukherjee, I., Kumar Gopa, G., Pena-Rios, A., & Biswas, P. (2021). Generating synthetic data for deep learning using VR digital twin. In Proceedings of the 2021 5th International Conference on Cloud and Big Data Computing (pp. 52–56). ACM Digital Library.

Mukhopadhyay, A., Reddy, G. R., Saluja, K. S., Ghosh, S., Peña-Rios, A., Gopal, G., & Biswas, P. (2022a). Virtual-reality-based digital twin of office spaces with social distance measurement feature. Virtual Reality & Intelligent Hardware, 4(1), 55–75.

Mukhopadhyay, A., Murthy, L. R. D., Mukherjee, I., & Biswas, P. (2022b). A hybrid lane detection model for wild road conditions. IEEE Transactions on Artificial Intelligence, 4(6).

Mukhopadhyay, A., Sharma, V. K., Tatyarao, P. G., Shah, A. K., Rao, A. M., Subin, P. R., & Biswas, P. (2023). A comparison study between XR interfaces for driver assistance in take over request. Transportation Engineering, 11, Article 100159.

Ogunwa, T. T., & Abdullah, E. J. (2016). Flight dynamics and control modelling of damaged asymmetric aircraft. IOP Conference Series: Materials Science and Engineering, 152(1), Article 012022.

Pizzati, F., Allodi, M., Barrera, A., & García, F. (2020). Lane detection and classification using cascaded CNNs. In Computer Aided Systems Theory – EUROCAST 2019. Lecture Notes in Computer Science (Vol. 12014, pp. 95–103). Springer.

Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 779–788). IEEE.

Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster R-CNN: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems, 28.

Sun, T. Y., Tsai, S. J., & Chan, V. (2006). HSI color model based lane-marking detection. In 2006 IEEE Intelligent Transportation Systems Conference (pp. 1168–1172). IEEE.

TurtleBot3. (2023). TurtleBot3.

Uijlings, J. R., Van De Sande, K. E., Gevers, T., & Smeulders, A. W. (2013). Selective search for object recognition. International Journal of Computer Vision, 104, 154–171.

Zammit, C., & Zammmit-Mangion, D. (2014). A control technique for automatic taxi in fixed wing. In 52nd Aerospace Sciences Meeting (p. 1163). Aerospace Research Central.

Zhang, Y., Poupart-Lafarge, G., Teng, H., Wilhelm, J., Jeannin, J. B., Ozay, N., & Scholte, E. (2020). A software architecture for autonomous taxiing of aircraft. In AIAA Scitech 2020 Forum (p. 0139). Aerospace Research Central.