Comparative Study of TF-IDF-SVM and IndoBERT for Imbalanced Indonesian Hoax Detection

Abstract

Indonesian hoax detection remains challenging because classification performance is influenced by text representation, class imbalance, and experimental design. This study comparatively evaluates a conventional Term Frequency–Inverse Document Frequency with Support Vector Machine (TF-IDF–SVM) approach and the transformer-based IndoBERT model under the same experimental protocol on a naturally imbalanced Indonesian hoax dataset. News titles and narratives were combined as textual input, duplicate records were removed, and the resulting dataset was stratified into training, validation, and held-out test sets. Model configurations were selected exclusively using validation Macro F1-score to prevent test-set information from influencing model selection. The final SVM employed unigram–bigram TF-IDF features with balanced LinearSVC, whereas the selected IndoBERT configuration used unweighted cross-entropy loss with a maximum sequence length of 128 tokens. On the test set, TF-IDF–SVM achieved an accuracy of 0.8299, Macro Precision of 0.7123, Macro Recall of 0.7065, and Macro F1-score of 0.7093, while IndoBERT achieved 0.8047, 0.6665, 0.6573, and 0.6616, respectively. The stronger numerical performance of TF-IDF–SVM suggests that discriminative lexical patterns remained highly informative in this dataset, while the contextual advantages of IndoBERT may not have been fully realized under the substantial class imbalance and the investigated training configuration. However, the exact McNemar test produced a p-value of 0.1371, indicating that the difference in paired classification correctness was not statistically significant at the 0.05 level. These findings demonstrate that TF-IDF–SVM remains a competitive and computationally simpler baseline for Indonesian hoax detection under imbalanced data conditions, while the benefit of more complex contextual models remains dependent on dataset characteristics and experimental settings.