Rizky Dwi Purnomo, Tirana Noor Fatyanosa, Mahendra Data
Transformer-based models, such as Bidirectional Encoder Representations from Transformers (BERT), have achieved significant advancements in natural language processing by understanding deeper connections between words and their context. This advancement has relied on the model's architecture and its capacity to capture linguistic nuances, which are essential for tasks like text classification. Fine-tuning on domain-specific datasets has been crucial for optimal performance in text classification. Additionally, data preprocessing has significantly improved BERT's performance by emphasizing relevant information. This research aimed to assess how data preprocessing and fine-tuning affect BERT's performance in text classification. Specifically, preprocessing techniques enhance performance by effectively reducing noise and highlighting critical features in complex datasets, while fine-tuning adjusts the model's parameters to capture domain-specific nuances, thereby improving accuracy on tailored tasks. The study involved several stages: collecting benchmark datasets (SST-2, AG News, Enron Spam), designing preprocessing methods, fine-tuning BERT using domain-specific models, hyperparameter tuning, feature-based approaches, and classifier algorithms. The findings revealed that pre-processing improved performance for complex datasets while reducing it for simpler ones. Domain-specific BERT models outperformed base and large versions. The optimal learning rate was found to be 2 e-5, and the last encoder layer provided the best feature representation to be fed to BERT's classification head. © 2025 IEEE.
Faculty of Computer Science, Universitas Brawijaya, Malang, Indonesia
