Evaluating knowledge graph enhanced retrieval augmented generation for automated functional requirements extraction
DOI: https://doi.org/10.3846/ntcs.2026.27121Abstract
The automation of requirements engineering using large language models offers significant potential for efficiency but struggles with hallucinations and a lack of domain-specific precision in highly regulated fields such as healthcare. While retrieval augmented generation (RAG) addresses some of these issues, standard vector-based retrieval often fails to capture complex semantic relationships required for strict compliance. This research assesses the effectiveness of automated functional requirement extraction by comparing two distinct retrieval architectures: a baseline vector-only RAG versus a knowledge graph-enhanced RAG with three distinct prompt strategies. We implemented an end-to-end automated extraction pipeline applied to a corpus of heterogeneous healthcare documents. The study constructed two separate knowledge bases to perform a comparative analysis. Under the experimental conditions (seven healthcare documents, three security-focused testing areas, and Claude3-Haiku at temperature zero), knowledge base curation was found to be a stronger determinant of extraction quality than retrieval architecture, though the relative impact may vary with corpus scale, domain, and base LLM. With a curated knowledge base, GraphRAG eliminated the systematic relevance collapse observed under standard preprocessing, achieved significant structural traceability validity in independent Neo4j cross-checking, and produced prompt-stable performance across three prompt strategies. Without rigorous preprocessing, graph augmentation amplified rather than mitigated retrieval noise
Keywords:
requirements extraction, Large Language Models (LLMs), prompt strategies, retrieval augmented generation, GraphRAGHow to Cite
Share
License
Copyright (c) 2026 The Author(s). Published by Vilnius Gediminas Technical University.
This work is licensed under a Creative Commons Attribution 4.0 International License.
References
Abualhaija, S., Basak Aydemir, F., Dalpiaz, F., Dell’Anna, D., Ferrari, A., Franch, X., & Fucci, D. (2024). Replication in requirements engineering: The NLP for RE case. ACM Transactions on Software Engineering and Methodology, 33(6), Article 151. https://doi.org/10.1145/3658669
Aishwarya, V. (2023). A prompt engineering approach for structured data extraction from unstructured text using conversational LLMs. In Proceedings of the 2023 6th International Conference on Algorithms, Computing and Artificial Intelligence (ACAI ‘23) (pp. 183–189). Association for Computing Machinery. https://doi.org/10.1145/3639631.3639663
Alhoshan, W., Ferrari, A., & Zhao, L. (2023). Zero-shot learning for requirements classification: An exploratory study. Information and Software Technology, 159, Article 107202. https://doi.org/10.1016/j.infsof.2023.107202
Alhoshan, W., Ferrari, A., & Zhao, L. (2025). How effective are generative large language models in performing requirements classification? arXiv. https://doi.org/10.48550/arXiv.2504.16768
Arora, C., Herda, T., & Homm, V. (2024). Generating test scenarios from NL requirements using retrieval-augmented LLMs: An industrial study. In Proceedings of the IEEE International Conference on Requirements Engineering (pp. 240–251). IEEE. https://doi.org/10.1109/RE59067.2024.00031
Arvidsson, S., & Axell, J. (2023). Prompt engineering guidelines for LLMs in Requirements Engineering (pp. 1–19). Göteborgs universitet. https://gupea.ub.gu.se/handle/2077/77967
Barnett, S., Kurniawan, S., Thudumu, S., Brannelly, Z., & Abdelrazek, M. (2024). Seven failure points when engineering a retrieval augmented generation system. In Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering – Software Engineering for AI (CAIN ‘24) (pp. 194–199). Association for Computing Machinery. https://doi.org/10.1145/3644815.3644945
Chen, B., Guo, Z., Yang, Z., Chen, Y., Chen, J., Liu, Z., Shi, C., & Yang, C. (2026). PathRAG: Pruning graph-based retrieval augmented generation with relational paths. Proceedings of the AAAI Conference on Artificial Intelligence, 40(36), 30183–30191. https://doi.org/10.1609/aaai.v40i36.40268
Edge, D., Trinh, H., Cheng, N., Bradley, J., Chao, A., Mody, A., Truitt, S., Metropolitansky, D., Ness, R. O., & Larson, J. (2025). From local to global: A graph RAG approach to query-focused summarization. arXiv. https://doi.org/10.48550/arXiv.2404.16130
Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2024). RAGAS: Automated evaluation of retrieval augmented generation. In EACL 2024 – 18th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of System Demonstrations (pp. 150–158). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.eacl-demo.16
Feng, N., Marsso, L., Yaman, S. G., Standen, I., Baatartogtokh, Y., Ayad, R., De Mello, V. O., Townsend, B., Bartels, H., Cavalcanti, A., Calinescu, R., & Chechik, M. (2024). Normative requirements operationalization with large language models. In Proceedings of the IEEE International Conference on Requirements Engineering (pp. 129–141). IEEE. https://doi.org/10.1109/RE59067.2024.00022
Guo, Z., Xia, L., Yu, Y., Ao, T., & Huang, C. (2025). LightRAG: Simple and Fast retrieval-augmented generation. In Findings of the Association for Computational Linguistics: EMNLP 2025 (pp. 10746–10761). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.findings-emnlp.568
IEEE Computer Society. (1998). IEEE Recommended Practice for Software Requirements Specifications (Standard No. IEEE Std 830-1998). IEEE. https://doi.org/10.1109/IEEESTD.1998.88286
International Organization for Standardization. (2011). Systems and software engineering – Life cycle processes – Requirements engineering (Standard No. ISO/IEC/IEEE 29148:2011). https://doi.org/10.1109/IEEESTD.2011.6146379
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y., Chen, D., Dai, W., Chan, H. S., Madotto, A., & Fung, P. (2024). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), Article 248. https://doi.org/10.1145/3571730
Krishna, M., Gaur, B., Verma, A., & Jalote, P. (2024). Using LLMs in Software Requirements Specifications: An Empirical Evaluation. In Proceedings of the IEEE International Conference on Requirements Engineering (pp. 475–483). IEEE. https://doi.org/10.1109/RE59067.2024.00056
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W. T., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems (NIPS ‘20) (pp. 9459–9474). Curran Associates Inc.
Marron, J. A. (2024). Implementing the Health Insurance Portability and Accountability Act (HIPAA) security rule: A cybersecurity resource guide (NIST Special Publication 800-66r2). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.SP.800-66r2
Motger, Q., & Franch, X. (2024). NLP-based Relation Extraction Methods in Requirements Engineering. arXiv. https://doi.org/10.48550/arXiv.2401.12075
Pasquale, L., Ragone, A., Piemontese, E., & Darban, A. A. (2025). Exploring the Use of LLMs for Requirements Specification in an IT Consulting Company. In Proceedings of the IEEE International Conference on Requirements Engineering (pp. 389–399). IEEE. https://doi.org/10.1109/RE63999.2025.00045
Sarmah, B., Hall, B., Rao, R., Patel, S., Pasquali, S., & Mehta, D. (2024). HybridRAG: Integrating knowledge graphs and vector retrieval augmented generation for efficient information extraction. In Proceedings of the 5th ACM International Conference on AI in Finance (pp. 608–616). Association for Computing Machinery. https://doi.org/10.48550/arXiv.2408.04948
Vogelsang, A., & Fischbach, J. (2024). Using large language models for natural language processing tasks in requirements engineering: A systematic guideline. arXiv. https://doi.org/10.48550/arXiv.2402.13823
Wei, B. (2024). Requirements are all you need: From requirements to code with LLMs. In Proceedings of the IEEE International Conference on Requirements Engineering (pp. 416–422). IEEE. https://doi.org/10.1109/RE59067.2024.00049
View article in other formats
CrossMark check
Published
Issue
Section
Copyright
Copyright (c) 2026 The Author(s). Published by Vilnius Gediminas Technical University.
License
This work is licensed under a Creative Commons Attribution 4.0 International License.