An intelligent construction system based on digital twin and foundation model optimization
DOI: https://doi.org/10.3846/jcem.2026.26154Abstract
Construction sites routinely face multi-trade concurrency, spatiotemporal coupling, and high safety risk; relying solely on manual inspection and heuristic scheduling often leads to lagging detection and inconsistent execution. In response, recent practice has introduced digital twins (DT) to fuse video, sensors, and BIM and thus improve site visibility; however, most implementations remain at monitoring/visualization, lacking a mechanism to convert cognition into executable, verifiable decisions. Meanwhile, Transformer foundation models show strong capabilities in multimodal perception and representation learning, yet they are rarely closed-looped with engineering constraints and on-site execution.
Against this backdrop, taking high-rise self-climbing platform (SCP) operations as a representative scenario, we build a DT×Transformer closed-loop system. We align video/sensor/BIM/text at the component level via “Component-ID + Timestamp”, train a multimodal Transformer for operation-state recognition and short-horizon risk prediction, and then explicitly encode safety, resource, and spatial precedence constraints in a policy module to generate feasible task sequences, which are delivered to crews via AR with acknowledgments to close the loop. The system integrates multisource perception, digital twin, foundation-model reasoning, and AR-assisted execution, and was validated on a highrise self-climbing platform project for its overall improvement of construction performance. The evaluation covered four key aspects – safety management, operational efficiency, communication and execution, and information transparency. Results show that the system significantly extends the lead time of risk warnings, reduces violation rates, stabilizes construction rhythm, shortens decision latency, and markedly improves the consistency between instruction delivery and on-site feedback.
Keywords:
intelligent construction, digital twin, foundation model, high-rise self-climbing platform, AR interactionHow to Cite
Share
License
Copyright (c) 2026 The Author(s). Published by Vilnius Gediminas Technical University.
This work is licensed under a Creative Commons Attribution 4.0 International License.
References
Abioye, S. O., Oyedele, L. O., Akanbi, L., Ajayi, A., Davila Delgado, J. M., Bilal, M., Akinade, O. O., & Ahmed, A. (2021). Artificial intelligence in the construction industry: A review of present status, opportunities and future challenges. Journal of Building Engineering, 44, Article 103299. https://doi.org/10.1016/j.jobe.2021.103299
Alkan, I. B., & Basaga, H. B. (2023). Augmented reality technologies in construction project assembly phases. Automation in Construction, 156, Article 105107. https://doi.org/10.1016/j.autcon.2023.105107
Alsakka, F., Yu, H., El-Chami, I., Hamzeh, F., & Al-Hussein, M. (2024). Digital twin for production estimation, scheduling and real-time monitoring in offsite construction. Computers & Industrial Engineering, 191, Article 110173. https://doi.org/10.1016/j.cie.2024.110173
Amer, F., Jung, Y., & Golparvar-Fard, M. (2021). Transformer machine learning language model for auto-alignment of long-term and short-term plans in construction. Automation in Construction, 132, Article 103929. https://doi.org/10.1016/j.autcon.2021.103929
Ammar, A., Nassereddine, H., AbdulBaky, N., AbouKansour, A., Tannoury, J., Urban, H., & Schranz, C. (2022). Digital twins in the construction industry: A perspective of practitioners and building authority. Frontiers in Built Environment, 8, Article 834671. https://doi.org/10.3389/fbuil.2022.834671
Aroquipa, H., Hurtado, A., Murga, C., De La Cruz, R., & Tarque, N. (2025). Towards smart cities: Foundational methodology for implementing intelligent circular resilience in heritage buildings through structural health monitoring and digital-twins – part a. International Journal of Disaster Risk Reduction, 128, Article 105749. https://doi.org/10.1016/j.ijdrr.2025.105749
Boje, C., Guerriero, A., Kubicki, S., & Rezgui, Y. (2020). Towards a semantic construction digital twin: Directions for future research. Automation in Construction, 114, Article 103179. https://doi.org/10.1016/j.autcon.2020.103179
Bureau of Labor Statistics. (n.d.), Construction deaths due to falls, slips, and trips increased 5.9 percent in 2021. https://www.bls.gov/opub/ted/2023/construction-deaths-due-to-falls-slips-and-trips-increased-5-9-percent-in-2021.htm
Cai, B., Ye, Z., Chen, S., & Liang, X. (2024). Reducing safety risks in construction tower crane operations: A dynamic path planning model. Applied Sciences, 14(22), Article 10599. https://doi.org/10.3390/app142210599
Chen, Q., Long, D., Yang, C., & Xu, H. (2023). Knowledge graph improved dynamic risk analysis method for behavior-based safety management on a construction site. Journal of Management in Engineering, 39(4), Article 4023023. https://doi.org/10.1061/JMENEA.MEENG-5306
Construction Dive. (n.d.). Why construction productivity growth is lagging—And what to do about it | construction dive. https://www.constructiondive.com/news/why-construction-productivity-lags-mckinsey/736082/
Deng, M., Menassa, C. C., & Kamat, V. R. (2021). From BIM to digital twins: A systematic review of the evolution of intelligent building representations in the AEC-FM industry. Journal of Information Technology in Construction, 26, 58–83. https://doi.org/10.36680/j.itcon.2021.005
Eum, I., Kim, J., Wang, S., & Kim, J. (2025). Heavy equipment detection on construction sites using you only look once (YOLO-version 10) with transformer architectures. Applied Sciences, 15(5), Article 2320. https://doi.org/10.3390/app15052320
Fan, C., Zhang, C., Yahja, A., & Mostafavi, A. (2021). Disaster city digital twin: A vision for integrating artificial and human intelligence for disaster management. International Journal of Information Management, 56, Article 102049. https://doi.org/10.1016/j.ijinfomgt.2019.102049
Gharaibeh, L., Matarneh, S., Lantz, B., & Eriksson, K. (2024). Quantifying the influence of BIM adoption: An in-depth methodology and practical case studies in construction. Results in Engineering, 23, Article 102555. https://doi.org/10.1016/j.rineng.2024.102555
Greif, T., Stein, N., & Flath, C. M. (2020). Peeking into the void: Digital twins for construction site logistics. Computers in Industry, 121, Article 103264. https://doi.org/10.1016/j.compind.2020.103264
Grieves, M. (2024). Intelligent digital twins and the development and management of complex systems. Digital Twin, 1(1), Article 8. https://doi.org/10.12688/digitaltwin.17574.1
Han, K. K., & Golparvar-Fard, M. (2017). Potential of big visual data and building information modeling for construction performance analytics: An exploratory study. Automation in Construction, 73, 184–198. https://doi.org/10.1016/j.autcon.2016.11.004
Huang, Y., Tao, J., Sun, G., Wu, T., Yu, L., & Zhao, X. (2023). A novel digital twin approach based on deep multimodal information fusion for aero-engine fault diagnosis. Energy, 270, Article 126894. https://doi.org/10.1016/j.energy.2023.126894
Jiang, F., Ma, L., Broyd, T., & Chen, K. (2021). Digital twin and its implementations in the civil engineering sector. Automation in Construction, 130, Article 103838. https://doi.org/10.1016/j.autcon.2021.103838
Jiang, Y., Su, S., Zhao, S., Zhong, R. Y., Qiu, W., Skibniewski, M. J., Brilakis, I., & Huang, G. Q. (2024). Digital twin-enabled synchronized construction management: A roadmap from construction 4.0 towards future prospect. Developments in the Built Environment, 19, Article 100512. https://doi.org/10.1016/j.dibe.2024.100512
Jung, Y., Hockenmaier, J., & Golparvar-Fard, M. (2024). Transformer language model for mapping construction schedule activities to uniformat categories. Automation in Construction, 157, Article 105183. https://doi.org/10.1016/j.autcon.2023.105183
Li, F., Lang, S., Tian, Y., Hong, B., Rolf, B., Noortwyck, R., Schulz, R., & Reggelin, T. (2025). A transformer-based deep reinforcement learning approach for dynamic parallel machine scheduling problem with family setups. Journal of Intelligent Manufacturing, 36(7), 4735–4768. https://doi.org/10.1007/s10845-024-02470-8
Lu, Q., Parlikad, A. K., Woodall, P., Don Ranasinghe, G., Xie, X., Liang, Z., Konstantinou, E., Heaton, J., & Schooling, J. (2020). Developing a digital twin at building and city levels: Case study of West Cambridge campus. Journal of Management in Engineering, 36(3), Article 5020004. https://doi.org/10.1061/(ASCE)ME.1943-5479.0000763
Madubuike, O. C., Anumba, C. J., & Khallaf, R. (2022). A review of digital twin applications in construction. Journal of Information Technology in Construction, 27, 145–172. https://doi.org/10.36680/j.itcon.2022.008
McKinsey. (n.d.). Challenges in the construction industry. Retrieved June 29, 2025, from https://www.linkedin.com/pulse/mckinsey-challenges-construction-industry-buromatei-uttrc
Linesight. (n.d.). Is efficiency eluding the construction industry? https://www.linesight.com/en-us/insights/is-efficiency-eluding-the-construction-industry
Nguyen, T. D., & Adhikari, S. (2023). The role of BIM in integrating digital twin in building construction: A literature review. Sustainability, 15(13), Article 10462. https://doi.org/10.3390/su151310462
Oesterreich, T. D., & Teuteberg, F. (2016). Understanding the implications of digitisation and automation in the context of industry 4.0: A triangulation approach and elements of a research agenda for the construction industry. Computers in Industry, 83, 121–139. https://doi.org/10.1016/j.compind.2016.09.006
Pan, Y., & Zhang, L. (2021). A BIM-data mining integrated digital twin framework for advanced project management. Automation in Construction, 124, Article 103564. https://doi.org/10.1016/j.autcon.2021.103564
Peng, J., & Liu, X. (2023). Automated code compliance checking research based on BIM and knowledge graph. Scientific Reports, 13(1), Article 7065. https://doi.org/10.1038/s41598-023-34342-1
Reja, V. K., Varghese, K., & Ha, Q. P. (2022). Computer vision-based construction progress monitoring. Automation in Construction, 138, Article 104245. https://doi.org/10.1016/j.autcon.2022.104245
Sacks, R., Brilakis, I., Pikas, E., Xie, H. S., & Girolami, M. (2020). Construction with digital twin information systems. Data-Centric Engineering, 1, Article e14. https://doi.org/10.1017/dce.2020.16
Shi, T., & Shide, K. (2026). A comparative analysis of LSTM, GRU, and transformer models for construction cost prediction with multidimensional feature integration. Journal of Asian Architecture and Building Engineering, 25(1), 634–649. https://doi.org/10.1080/13467581.2025.2455034
Tian, C., Chen, Y., Feng, Y., & Zhang, J. (2024). Fine-tuning vision transformer (ViT) to classify highway construction workers’ activities. In Proceedings of Construction Research Congress 2024 (pp. 1140–1148). ASCE. https://doi.org/10.1061/9780784485262.116
Vassena, G. P. M., Perfetti, L., Comai, S., Mastrolembo Ventura, S., & Ciribini, A. L. C. (2023). Construction progress monitoring through the integration of 4D BIM and SLAM-based mapping devices. Buildings, 13(10), Article 10. https://doi.org/10.3390/buildings13102488
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. In 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
Wang, C. C., & Chien, O. (2014). The use of BIM in project planning and scheduling in the australian construction industry. In Proceedings of ICCREM 2014 (pp. 126–133). ASCE. https://doi.org/10.1061/9780784413777.015
Wang, X., Truijens, M., Hou, L., Wang, Y., & Zhou, Y. (2014). Integrating augmented reality with building information modeling: Onsite construction process controlling for liquefied natural gas industry. Automation in Construction, 40, 96–105. https://doi.org/10.1016/j.autcon.2013.12.003
Wang, C. C., Wang, M., Sun, J., & Mojtahedi, M. (2021). A safety warning algorithm based on axis aligned bounding box method to prevent onsite accidents of mobile construction machineries. Sensors, 21(21), Article 7075. https://doi.org/10.3390/s21217075
Wang, M., Wang, C. C., Zlatanova, S., Shen, X., & Brilakis, I. (2024). A streamlined laser scanning verticality check method for installation of prefabricated wall panels. Journal of Construction Engineering and Management, 150(11), Article 4024159. https://doi.org/10.1061/JCEMD4.COENG-14989
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., & Fedus, W. (2022). Emergent abilities of large language models. arXiv. https://doi.org/10.48550/arXiv.2206.07682
Wu, J., Peng, L., Sheng, W., Wang, C. C., & Sun, J. (2023). Track gauge measurement based on model matching using UAV image. Automation in Construction, 155, Article 105070. https://doi.org/10.1016/j.autcon.2023.105070
Xie, X., Lu, Q., Rodenas-Herraiz, D., Parlikad, A. K., & Schooling, J. M. (2020). Visualised inspection system for monitoring environmental anomalies during daily operation and maintenance. Engineering, Construction and Architectural Management, 27(8), 1835–1852. https://doi.org/10.1108/ECAM-11-2019-0640
Xu, P., Zhu, X., & Clifton, D. A. (2023). Multimodal learning with transformers: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10), 12113–12132. https://doi.org/10.1109/TPAMI.2023.3275156
Yang, M., Wu, C., Guo, Y., Jiang, R., Zhou, F., Zhang, J., & Yang, Z. (2023). Transformer-based deep learning model and video dataset for unsafe action identification in construction projects. Automation in Construction, 146, Article 104703. https://doi.org/10.1016/j.autcon.2022.104703
Yang, Z., Tang, C., Zhang, T., Zhang, Z., & Doan, D. T. (2024). Digital twins in construction: Architecture, applications, trends and challenges. Buildings, 14(9), Article 2616. https://doi.org/10.3390/buildings14092616
Yao, Y., Tam, V. W. Y., Wang, J., Le, K. N., & Butera, A. (2024). Automated construction scheduling using deep reinforcement learning with valid action sampling. Automation in Construction, 166, Article 105622. https://doi.org/10.1016/j.autcon.2024.105622
Yitmen, I., Kovacic, I., & Tagliabue, L. C. (2023). Cognitive digital twins for facilitating construction 4.0: Challenges and opportunities for implementation. Frontiers in Built Environment, 9, Article 1130115. https://doi.org/10.3389/fbuil.2023.1130115
Yoo, B., Kim, J., Park, S., Ahn, C. R., & Oh, T. (2024). Harnessing generative pre-trained transformers for construction accident prediction with saliency visualization. Applied Sciences, 14(2), Article 664. https://doi.org/10.3390/app14020664
Zulu, S. L., Saad, A. M., & Omotayo, T. (2023). The mediators of the relationship between digitalisation and construction productivity: A systematic literature review. Buildings, 13(4), Article 4. https://doi.org/10.3390/buildings13040839
View article in other formats
CrossMark check
Published
Issue
Section
Copyright
Copyright (c) 2026 The Author(s). Published by Vilnius Gediminas Technical University.
License
This work is licensed under a Creative Commons Attribution 4.0 International License.