The efficiency of machine learning algorithms in classifying non-functional requirements
Abstract
Machine learning (ML) algorithms are more and more widely applied in various types of systems, so the research related to them is also increasing. One of the areas of research under consideration is the classification of non-functional requirements (NFRs) using ML algorithms. This area of research is important because the automatic classification of NFRs using high-performance ML algorithms and corresponding features helps requirements engineers classify non-functional requirements more accurately. This paper examines ML algorithms suitable for solving classification problems and their effectiveness in classifying non-functional requirements. Based on the described stages of the research methodology ML algorithms models were compared using the accuracy, precision, recall, and F-score metrics. A majority voting classifier model was created using Support Vector Machine, Naïve Bayes and K Nearest Neighbor Algorithm algorithms. After K-Fold cross validation were obtained these results: accuracy – 0.710 (scale from 0 to 1), precision – 0.845, recall – 0.814 and F-score – 0.815.
Keyword : machine learning, non-functional requirements, classification, support vector machine, ensemble models, K-Fold cross validation
How to Cite
Maciejauskaitė, M., & Miliauskaitė, J. (2024). The efficiency of machine learning algorithms in classifying non-functional requirements. New Trends in Computer Sciences, 2(1), 46–56. https://doi.org/10.3846/ntcs.2024.21574
This work is licensed under a Creative Commons Attribution 4.0 International License.
References
Abad, Z. S., Karras, O., Ghazi, P., Glinz, M., Ruhe, G., & Schneider, K. (2017). What works better? A study of classifying requirements. In 2017 IEEE 25th International Requirements Engineering Conference (RE), (pp. 496–501). Lisbon. https://doi.org/10.1109/RE.2017.36
Alashqar, A. M. (2022). Studying the commonalities, mappings and relationships between non-functional requirements using machine learning. Science of Computer Programming, 218, Article 102806. https://doi.org/10.1016/j.scico.2022.102806
Bajaj, A. (2023, April 27). Ensemble models: How to make better predictions by combining multiple models with Python codes (explained). https://aryanbajaj13.medium.com/ensemble-models-how-to-make-better-predictions-by-combining-multiple-models-with-python-codes-6ac54403414e
Baker, C., Deng, L., Chakraborty, S., & Dehlinger, J. (2019). Automatic multi-class non-functional software requirements classification using neural networks. In 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC), (pp. 610–615). Milwaukee. https://doi.org/10.1109/COMPSAC.2019.10275
Bao, W., Lianju, N., & Yue, K. (2019). Integration of unsupervised and supervised machine learning algorithms for credit risk assessment. Expert Systems with Applications, 128, 301–315. https://doi.org/10.1016/j.eswa.2019.02.033
Binkhonain, M., & Zhao, L. (2019). A review of machine learning algorithms for identification and classification of non-functional requirements. Expert Systems with Applications. https://doi.org/10.1016/j.eswa.2019.02.031
Carta, S. (2022). What is machine learning? Wiley-Blackwell. https://doi.org/10.1002/9781119815075.ch18
Ghoneim, S. (2019, April 02). Accuracy, recall, precision, f-score & specificity, which to optimize on? https://medium.com/towards-data-science/accuracy-recall-precision-f-score-specificity-which-to-optimize-on-867d3f11124
Habibullah, K. M., & Horkoff, J. (2021, September). Non-functional requirements for machine learning: Understanding current use and challenges in industry. In 2021 IEEE 29th International Requirements Engineering Conference (RE) (pp. 13–23). IEEE. https://doi.org/10.1109/RE51729.2021.00009
Haque, M. A., Rahman, M. A., & Siddik, M. S. (2019). Non-functional requirements classification with feature extraction and machine learning: An empirical study. In 2019 1st International Conference on Advances in Science, Engineering and Robotics Technology (ICASERT), (pp. 1–5). Dhaka, Bangladesh. https://doi.org/10.1109/ICASERT.2019.8934499
Harikrishnan, N. B. (2019, December 10). Confusion matrix, accuracy, precision, recall, F1 score. https://medium.com/analytics-vidhya/confusion-matrix-accuracy-precision-recall-f1-score-ade299cf63cd
Hendricks, R. (n.d.). What is a good accuracy score in Machine Learning? https://deepchecks.com/question/what-is-a-good-accuracy-score-in-machine-learning/
Ho, W. K., Tang, B.-S., & Wong, S. W. (2021). Predicting property prices with machine learning algorithms. Journal of Property Research, 38(1), 48–70. https://doi.org/10.1080/09599916.2020.1832558
Ibrahim, I. M., & Abdulazeez, A. M. (2021). The role of machine learning algorithms for diagnosing diseases. Journal of Applied Science and Technology Trends (JASTT), 2(1), 10–19. https://doi.org/10.38094/jastt20179
Imam, T., & Ananda, J. (2022). Machine learning for characterizing growth in tourism employment in developing economies: an assessment of tourism employment in Sri Lanka. Current Issues in Tourism, 25(16), 2695–2716. https://doi.org/10.1080/13683500.2021.1991895
Janiesch, C., Zschech, P., & Heinrich, K. (2021). Machine learning and deep learning. Electronic Markets, 31, 685–695. https://doi.org/10.1007/s12525-021-00475-2
Kanade, V. (2022). What is logistic regression? Equation, assumptions, types, and best practices. https://www.spiceworks.com/tech/artificial-intelligence/articles/what-is-logistic-regression/
Karthiban, R., Ambika, M., & Kannammal, K. E. (2019, January). A review on machine learning classification technique for bank loan approval. In 2019 International Conference on Computer Communication and Informatics (ICCCI) (pp. 1–6). IEEE. https://doi.org/10.1109/ICCCI.2019.8822014
Khurshid, I., Imtiaz, S., Boulila, W., Khan, Z., & Abbasi, A. (2022). Classification of non-functional requirements from IoT oriented healthcare requirement document. Frontiers Public Health, 10, Article 860536. https://doi.org/10.3389/fpubh.2022.860536
Koehrsen, W. (2018, March 03). Beyond accuracy: Precision and recall. https://medium.com/towards-data-science/beyond-accuracy-precision-and-recall-3da06bea9f6c
Kumar, R. (2023, August 12). VotingClassifier. https://medium.com/@ranjankumar_29097/votingclassifier-3f85ba8e4580
Kurtanović, Z., & Maalej, W. (2017). Automatically classifying functional and non-functional requirements using supervised machine learning. In 2017 IEEE 25th International Requirements Engineering Conference (RE), (pp. 490–495). Lisbon. https://doi.org/10.1109/RE.2017.82
Mahesh, B. (2020). Machine learning algorithms – A review. International Journal of Science and Research (IJSR), 9(1), 381–386.
Miller, M. I., Shih, L. C., & Kolachalama, V. B. (2023). Machine learning in clinical trials: A primer with applications to neurology. Neurotherapeutics, 20(4), 1066–1080. https://doi.org/10.1007/s13311-023-01384-2
Mohd, T., Masrom, S., & Johari, N. (2019). Machine learning housing price prediction in Petaling Jaya, Selangor, Malaysia. International Journal of Recent Technology and Engineering, 8(2S11), 542–546. https://doi.org/10.35940/ijrte.B1084.0982S1119
Nelson, D. (2020, October 26). Kas yra ansamblinis mokymasis? https://www.unite.ai/lt/kas-yra-ansamblinis-mokymasis/
Rajaguru, H., & Chakravarthy, S. (2019). Analysis of decision tree and K-Nearest neighbor algorithm in the classification of breast cancer. Asian Pacific Journal Cancer Prevention, 20(12), 3777–3781. https://doi.org/10.31557/APJCP.2019.20.12.3777
Rymarczyk, T., Kozłowski, E., Kłosowski, G., & Niderla, K. (2019). Logistic regression for machine learning in process tomography. Sensors, 19(15), Article 3400. https://doi.org/10.3390/s19153400
Sarker, I. H., Kayes, A. S., Badsha, S., Alqahtani, H., Watters, P., & Ng, A. (2020). Cybersecurity data science: An overview from machine learning perspective. Journal of Big Data, 7, Article 41. https://doi.org/10.1186/s40537-020-00318-5
Shukla, V. (2023, February). Software requirements dataset. https://www.kaggle.com/datasets/iamvaibhav100/software-requirements-dataset?resource=download
Shung, K. P. (2018, March 15). Accuracy, precision, recall or F1? https://medium.com/towards-data-science/accuracy-precision-recall-or-f1-331fb37c5cb9
Silwal, D. (2022, January 05). Confusion matrix, accuracy, precision, recall & F1 score: Interpretation of performance measures. https://www.linkedin.com/pulse/confusion-matrix-accuracy-precision-recall-f1-score-measures-silwal
Singh, A. (2023, November 22). A comprehensive guide to ensemble learning (with Python codes). https://www.analyticsvidhya.com/blog/2018/06/comprehensive-guide-for-ensemble-models/
Sruthi, E. R. (2023, April 26). Understand random forest algorithms with examples (updated 2023). https://www.analyticsvidhya.com/blog/2021/06/understanding-random-forest/
Uddin, S., Khan, A., Hossain, M. E., & Moni, M. A. (2019). Comparing different supervised machine learning algorithms for disease prediction. BMC Medical Informatics and Decision Making, 19, Article 281. https://doi.org/10.1186/s12911-019-1004-8
Wickramasinghe, I., & Kalutarage, H. (2021). Naive Bayes: Applications, variations and vulnerabilities: a review of literature with code snippets for implementation. Soft Computing, 25, 2277–2293. https://doi.org/10.1007/s00500-020-05297-6
Yang, L., & Shami, A. (2020). On hyperparameter optimization of machine learning algorithms: Theory and practice. Neurocomputing, 415, 295–316. https://doi.org/10.1016/j.neucom.2020.07.061
Alashqar, A. M. (2022). Studying the commonalities, mappings and relationships between non-functional requirements using machine learning. Science of Computer Programming, 218, Article 102806. https://doi.org/10.1016/j.scico.2022.102806
Bajaj, A. (2023, April 27). Ensemble models: How to make better predictions by combining multiple models with Python codes (explained). https://aryanbajaj13.medium.com/ensemble-models-how-to-make-better-predictions-by-combining-multiple-models-with-python-codes-6ac54403414e
Baker, C., Deng, L., Chakraborty, S., & Dehlinger, J. (2019). Automatic multi-class non-functional software requirements classification using neural networks. In 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC), (pp. 610–615). Milwaukee. https://doi.org/10.1109/COMPSAC.2019.10275
Bao, W., Lianju, N., & Yue, K. (2019). Integration of unsupervised and supervised machine learning algorithms for credit risk assessment. Expert Systems with Applications, 128, 301–315. https://doi.org/10.1016/j.eswa.2019.02.033
Binkhonain, M., & Zhao, L. (2019). A review of machine learning algorithms for identification and classification of non-functional requirements. Expert Systems with Applications. https://doi.org/10.1016/j.eswa.2019.02.031
Carta, S. (2022). What is machine learning? Wiley-Blackwell. https://doi.org/10.1002/9781119815075.ch18
Ghoneim, S. (2019, April 02). Accuracy, recall, precision, f-score & specificity, which to optimize on? https://medium.com/towards-data-science/accuracy-recall-precision-f-score-specificity-which-to-optimize-on-867d3f11124
Habibullah, K. M., & Horkoff, J. (2021, September). Non-functional requirements for machine learning: Understanding current use and challenges in industry. In 2021 IEEE 29th International Requirements Engineering Conference (RE) (pp. 13–23). IEEE. https://doi.org/10.1109/RE51729.2021.00009
Haque, M. A., Rahman, M. A., & Siddik, M. S. (2019). Non-functional requirements classification with feature extraction and machine learning: An empirical study. In 2019 1st International Conference on Advances in Science, Engineering and Robotics Technology (ICASERT), (pp. 1–5). Dhaka, Bangladesh. https://doi.org/10.1109/ICASERT.2019.8934499
Harikrishnan, N. B. (2019, December 10). Confusion matrix, accuracy, precision, recall, F1 score. https://medium.com/analytics-vidhya/confusion-matrix-accuracy-precision-recall-f1-score-ade299cf63cd
Hendricks, R. (n.d.). What is a good accuracy score in Machine Learning? https://deepchecks.com/question/what-is-a-good-accuracy-score-in-machine-learning/
Ho, W. K., Tang, B.-S., & Wong, S. W. (2021). Predicting property prices with machine learning algorithms. Journal of Property Research, 38(1), 48–70. https://doi.org/10.1080/09599916.2020.1832558
Ibrahim, I. M., & Abdulazeez, A. M. (2021). The role of machine learning algorithms for diagnosing diseases. Journal of Applied Science and Technology Trends (JASTT), 2(1), 10–19. https://doi.org/10.38094/jastt20179
Imam, T., & Ananda, J. (2022). Machine learning for characterizing growth in tourism employment in developing economies: an assessment of tourism employment in Sri Lanka. Current Issues in Tourism, 25(16), 2695–2716. https://doi.org/10.1080/13683500.2021.1991895
Janiesch, C., Zschech, P., & Heinrich, K. (2021). Machine learning and deep learning. Electronic Markets, 31, 685–695. https://doi.org/10.1007/s12525-021-00475-2
Kanade, V. (2022). What is logistic regression? Equation, assumptions, types, and best practices. https://www.spiceworks.com/tech/artificial-intelligence/articles/what-is-logistic-regression/
Karthiban, R., Ambika, M., & Kannammal, K. E. (2019, January). A review on machine learning classification technique for bank loan approval. In 2019 International Conference on Computer Communication and Informatics (ICCCI) (pp. 1–6). IEEE. https://doi.org/10.1109/ICCCI.2019.8822014
Khurshid, I., Imtiaz, S., Boulila, W., Khan, Z., & Abbasi, A. (2022). Classification of non-functional requirements from IoT oriented healthcare requirement document. Frontiers Public Health, 10, Article 860536. https://doi.org/10.3389/fpubh.2022.860536
Koehrsen, W. (2018, March 03). Beyond accuracy: Precision and recall. https://medium.com/towards-data-science/beyond-accuracy-precision-and-recall-3da06bea9f6c
Kumar, R. (2023, August 12). VotingClassifier. https://medium.com/@ranjankumar_29097/votingclassifier-3f85ba8e4580
Kurtanović, Z., & Maalej, W. (2017). Automatically classifying functional and non-functional requirements using supervised machine learning. In 2017 IEEE 25th International Requirements Engineering Conference (RE), (pp. 490–495). Lisbon. https://doi.org/10.1109/RE.2017.82
Mahesh, B. (2020). Machine learning algorithms – A review. International Journal of Science and Research (IJSR), 9(1), 381–386.
Miller, M. I., Shih, L. C., & Kolachalama, V. B. (2023). Machine learning in clinical trials: A primer with applications to neurology. Neurotherapeutics, 20(4), 1066–1080. https://doi.org/10.1007/s13311-023-01384-2
Mohd, T., Masrom, S., & Johari, N. (2019). Machine learning housing price prediction in Petaling Jaya, Selangor, Malaysia. International Journal of Recent Technology and Engineering, 8(2S11), 542–546. https://doi.org/10.35940/ijrte.B1084.0982S1119
Nelson, D. (2020, October 26). Kas yra ansamblinis mokymasis? https://www.unite.ai/lt/kas-yra-ansamblinis-mokymasis/
Rajaguru, H., & Chakravarthy, S. (2019). Analysis of decision tree and K-Nearest neighbor algorithm in the classification of breast cancer. Asian Pacific Journal Cancer Prevention, 20(12), 3777–3781. https://doi.org/10.31557/APJCP.2019.20.12.3777
Rymarczyk, T., Kozłowski, E., Kłosowski, G., & Niderla, K. (2019). Logistic regression for machine learning in process tomography. Sensors, 19(15), Article 3400. https://doi.org/10.3390/s19153400
Sarker, I. H., Kayes, A. S., Badsha, S., Alqahtani, H., Watters, P., & Ng, A. (2020). Cybersecurity data science: An overview from machine learning perspective. Journal of Big Data, 7, Article 41. https://doi.org/10.1186/s40537-020-00318-5
Shukla, V. (2023, February). Software requirements dataset. https://www.kaggle.com/datasets/iamvaibhav100/software-requirements-dataset?resource=download
Shung, K. P. (2018, March 15). Accuracy, precision, recall or F1? https://medium.com/towards-data-science/accuracy-precision-recall-or-f1-331fb37c5cb9
Silwal, D. (2022, January 05). Confusion matrix, accuracy, precision, recall & F1 score: Interpretation of performance measures. https://www.linkedin.com/pulse/confusion-matrix-accuracy-precision-recall-f1-score-measures-silwal
Singh, A. (2023, November 22). A comprehensive guide to ensemble learning (with Python codes). https://www.analyticsvidhya.com/blog/2018/06/comprehensive-guide-for-ensemble-models/
Sruthi, E. R. (2023, April 26). Understand random forest algorithms with examples (updated 2023). https://www.analyticsvidhya.com/blog/2021/06/understanding-random-forest/
Uddin, S., Khan, A., Hossain, M. E., & Moni, M. A. (2019). Comparing different supervised machine learning algorithms for disease prediction. BMC Medical Informatics and Decision Making, 19, Article 281. https://doi.org/10.1186/s12911-019-1004-8
Wickramasinghe, I., & Kalutarage, H. (2021). Naive Bayes: Applications, variations and vulnerabilities: a review of literature with code snippets for implementation. Soft Computing, 25, 2277–2293. https://doi.org/10.1007/s00500-020-05297-6
Yang, L., & Shami, A. (2020). On hyperparameter optimization of machine learning algorithms: Theory and practice. Neurocomputing, 415, 295–316. https://doi.org/10.1016/j.neucom.2020.07.061