Fine-Tuning IndoBERT for Indonesian Exam Question Classification Based on Bloom's Taxonomy
Downloads
Background: The learning assessment of elementary schools has recently incorporated Bloom's Taxonomy, a structure in education that categorizes different levels of cognitive learning and thinking skills, as a fundamental framework. This assessment now includes High Order Thinking Skill (HOTS) questions, with a specific focus on Indonesian topics. The implementation of this system has been observed to require teachers to manually categorize or classify questions, and this process typically requires more time and resources. To address the associated difficulty, automated categorization and classification are required to streamline the process. However, despite various research efforts in questions classification, there is still room for improvement in terms of performance, particularly in precision and accuracy. Numerous investigations have explored the use of Deep Learning Natural Language Processing models such as BERT for classification, and IndoBERT is one such pre-trained model for text analysis.
Objective: This research aims to build classification system that is capable of classifying Indonesian exam questions in multiple-choice form based on Bloom's Taxonomy using IndoBERT pre-trained model.
Methods: The methodology used includes hyperparameter fine-tuning, which was carried out to identify the optimal model performance. This performance was subsequently evaluated based on accuracy, F1 Score, Precision, Recall, and the time required for the training and validation of the model.
Results: The proposed Fine Tuned IndoBERT Model showed that the accuracy rate was 97%, 97% F1 Score, 97% Recall, and 98% Precision with an average training time per epoch of 1.55 seconds and an average validation time per epoch of 0.38 seconds.
Conclusion: Fine Tuned IndoBERT model was observed to have a relatively high classification performance, and based on this observation, the system was considered capable of classifying Indonesian exam questions at the elementary school level.
Keywords: IndoBERT, Fine Tuning, Indonesian Exam Question, Model Classifier, Natural Language Processing, Bloom's Taxonomy
N. Omar et al., "Automated analysis of exam questions according to Bloom's taxonomy,” Procedia-Social and Behavioral Sciences, vol. 59, pp. 297–303, 2012.
W.-C. Chang and M.-S. Chung, "Automatic applying Bloom's taxonomy to classify and analysis the cognition level of English question items,” in 2009 Joint Conferences on Pervasive Computing (JCPC), IEEE, 2009, pp. 727–734.
S. U. Monrad et al., "What faculty write versus what students see? Perspectives on multiple-choice questions using Bloom's taxonomy,” Med Teach, vol. 43, no. 5, pp. 575–582, 2021.
B. S. Bloom and D. R. Krathwohl, Taxonomy of educational objectives: The classification of educational goals. Book 1, Cognitive domain. longman, 2020.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
F. Koto, A. Rahimi, J. H. Lau, and T. Baldwin, "IndoLEM and IndoBERT: A benchmark dataset and pre-trained language model for Indonesian NLP,” arXiv preprint arXiv:2011.00677, 2020.
S. R. Anggraeni, N. A. Ranggianto, I. Ghozali, C. Fatichah, and D. Purwitasari, "Deep Learning Approaches for Multi-Label Incidents Classification from Twitter Textual Information.,” Journal of Information Systems Engineering & Business Intelligence, vol. 8, no. 1, 2022.
A. F. Abdillah, P. Putra, C. Bagus, S. Juanita, and D. Purwitasari, "Ensemble-based Methods for Multi-label Classification on Biomedical Question-Answer Data.,” Journal of Information Systems Engineering & Business Intelligence, vol. 8, no. 1, 2022.
M. Mohammed and N. Omar, "Question classification based on Bloom's taxonomy cognitive domain using modified TF-IDF and word2vec,” PLoS One, vol. 15, no. 3, p. e0230442, 2020.
F. Baharuddin and A. Tjahyanto, "Peningkatan Performa Klasifikasi Machine Learning Melalui Perbandingan Metode Machine Learning dan Peningkatan Dataset,” Jurnal Sisfokom (Sistem Informasi dan Komputer), vol. 11, no. 1, pp. 25–31, 2022.
S. Paul and S. Saha, "CyberBERT: BERT for cyberbullying identification: BERT for cyberbullying identification,” Multimed Syst, vol. 28, no. 6, pp. 1897–1904, 2022.
R. K. Kaliyar, A. Goswami, and P. Narang, "FakeBERT: Fake news detection in social media with a BERT-based deep learning approach,” Multimed Tools Appl, vol. 80, no. 8, pp. 11765–11788, 2021.
A. Rahmawati, A. Alamsyah, and A. Romadhony, "Hoax News Detection Analysis using IndoBERT Deep Learning Methodology,” in 2022 10th International Conference on Information and Communication Technology (ICoICT), IEEE, 2022, pp. 368–373.
X. Chen, P. Cong, and S. Lv, "A long-text classification method of Chinese news based on BERT and CNN,” IEEE Access, vol. 10, pp. 34046–34057, 2022.
R. Rahutomo and B. Pardamean, "Finetunning IndoBERT to Understand Indonesian Stock Trader Slang Language,” in 2021 1st International Conference on Computer Science and Artificial Intelligence (ICCSAI), IEEE, 2021, pp. 42–46.
L. Khan, A. Amjad, N. Ashraf, and H.-T. Chang, "Multi-class sentiment analysis of urdu text using multilingual BERT,” Sci Rep, vol. 12, no. 1, p. 5436, 2022.
X. Qiu, T. Sun, Y. Xu, Y. Shao, N. Dai, and X. Huang, "Pre-trained models for natural language processing: A survey,” Sci China Technol Sci, vol. 63, no. 10, pp. 1872–1897, 2020, doi: 10.1007/s11431-020-1647-3.
H. Lu, L. Ehwerhemuepha, and C. Rakovski, "A comparative study on deep learning models for text classification of unstructured medical notes with various levels of class imbalance,” BMC Med Res Methodol, vol. 22, no. 1, p. 181, 2022.
A. Esmaeilzadeh and K. Taghva, "Text classification using neural network language model (nnlm) and bert: An empirical comparison,” in Intelligent Systems and Applications: Proceedings of the 2021 Intelligent Systems Conference (IntelliSys) Volume 3, Springer, 2022, pp. 175–189.
A. Kulkarni, M. Mandhane, M. Likhitkar, G. Kshirsagar, J. Jagdale, and R. Joshi, "Experimental evaluation of deep learning models for marathi text classification,” in Proceedings of the 2nd International Conference on Recent Trends in Machine Learning, IoT, Smart Cities and Applications: ICMISC 2021, Springer, 2022, pp. 605–613.
D. Kakwani et al., "IndicNLPSuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages,” in Findings of the Association for Computational Linguistics: EMNLP 2020, 2020, pp. 4948–4961.
F. Baharuddin, "Elementary School Indonesian Exam Questions with Bloom's Taxonomy Dataset,” Sep. 2023, doi: 10.5281/ZENODO.8331563.
I. M. Arievitch, "The vision of Developmental Teaching and Learning and Bloom's Taxonomy of educational objectives,” Learn Cult Soc Interact, vol. 25, p. 100274, Jun. 2020, doi: 10.1016/J.LCSI.2019.01.007.
L. W. Anderson and D. R. Krathwohl, A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives. Longman, 2021.
Y. HaCohen-Kerner, D. Miller, and Y. Yigal, "The influence of preprocessing on text classification using a bag-of-words representation,” PLoS One, vol. 15, no. 5, p. e0232525, 2020.
A. Kathuria, A. Gupta, and R. K. Singla, "A Review of Tools and Techniques for Preprocessing of Textual Data,” in Computational Methods and Data Engineering, V. Singh, V. K. Asari, S. Kumar, and R. B. Patel, Eds., Singapore: Springer Singapore, 2021, pp. 407–422.
M. Munikar, S. Shakya, and A. Shrestha, "Fine-grained sentiment classification using BERT,” in 2019 Artificial Intelligence for Transforming Business and Society (AITB), IEEE, 2019, pp. 1–5.
R. Qasim, W. H. Bangyal, M. A. Alqarni, and A. Ali Almazroi, "A fine-tuned BERT-based transfer learning approach for text classification,” J Healthc Eng, vol. 2022, 2022.
M. Bilal and A. A. Almazroi, "Effectiveness of fine-tuned BERT model in classification of helpful and unhelpful online customer reviews,” Electronic Commerce Research, pp. 1–21, 2022.
O. A. Montesinos López, A. Montesinos López, and J. Crossa, "Overfitting, Model Tuning, and Evaluation of Prediction Performance,” in Multivariate Statistical Machine Learning Methods for Genomic Prediction, O. A. Montesinos López, A. Montesinos López, and J. Crossa, Eds., Cham: Springer International Publishing, 2022, pp. 109–139. doi: 10.1007/978-3-030-89010-0_4.
Copyright (c) 2023 The Authors. Published by Universitas Airlangga.
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
All accepted papers will be published under a Creative Commons Attribution 4.0 International (CC BY 4.0) License. Authors retain copyright and grant the journal right of first publication. CC-BY Licenced means lets others to Share (copy and redistribute the material in any medium or format) and Adapt (remix, transform, and build upon the material for any purpose, even commercially).
