MACHINE LEARNING MODELS FOR HAUSA-BASED LANGUAGE (WORDS) LEMMATIZATION

Abstract

Lemmatization, the process of reducing inflected word forms to their canonical dictionary form (lemma), is a foundational task in Natural Language Processing (NLP) that significantly enhances the performance of downstream applications like information retrieval, machine translation, and sentiment analysis. For morphologically rich, low-resource languages like Hausa—spoken by over 50 million people—this task is particularly challenging due to the absence of annotated datasets and the limitations of traditional rule-based systems, which are labor-intensive and fail to generalize. This paper presents the first comparative study of classical supervised machine learning models, Support Vector Machine (SVM) and Random Forest (RF), for Hausa word lemmatization. To address the critical data scarcity, we manually curated and linguistically validated a novel dataset of 4,530 Hausa word-lemma pairs sourced from diverse corpora including BBC Hausa, VOA Hausa, and BUK FM radio transcripts. Our methodology involved comprehensive text preprocessing (normalization of diacritics, clitic handling, stopword removal) and sophisticated feature engineering, extracting character trigrams, word length, prefix/suffix flags, and reduplication indicators. Models were trained on an 80:20 split and optimized using GridSearchCV. Our results demonstrate that the Random Forest model outperformed SVM significantly, achieving an accuracy of 63.25% and an F1-score of 60.26%, compared to SVM’s 56.73% accuracy and 54.75% F1-score. Crucially, RF was also six times faster to train (180 seconds vs. 1,094 seconds), making it far more practical for deployment. Feature importance analysis revealed that character trigrams and word length were the most predictive features, highlighting the efficacy of subword morphological cues.