Developing modified ALBERT Model for Under-Represented Classes in Depression Detection
| dc.contributor.author | Sanwar, Sk Nahid | |
| dc.date.accessioned | 2026-07-28T10:13:36Z | |
| dc.date.issued | 2025-10-25 | |
| dc.description | Supervised by Dr. Hasan Mahmud, Professor, Department of Computer Science and Engineering (CSE) Islamic University of Technology (IUT) Board Bazar, Gazipur, Bangladesh This thesis is submitted in partial fulfillment of the requirement for the degree of Master of Science in Computer Science and Engineering, 2025 | |
| dc.description.abstract | In the aftermath of the COVID-19 pandemic, global reliance on communication plat forms has surged, presenting unique opportunities to analyze mental health trends at scale. Early and accurate detection of severe depression via social media content can facilitate timely interventions. However, accurately identifying varying levels of de pression severity from textual data remains a significant challenge. Human language is nuanced and severe depression cases are often under-represented in datasets. Lan guage models such as ALBERT, BERT,andDistilBERT have become prominent tools in sentiment analysis and mental health diagnostics, yet achieving consistently high accuracy remains challenging due to class imbalances and the complexity of natural language. In this study, a modified version of the ALBERT Base v2 model was intro duced incorporating an additional self-attention mechanism designed to enhance the detection of depression in text, which is often most underrepresented in datasets and contains complex structures. The performance of the original ALBERT Base v2, the modified ALBERT, BERT, and DistilBERT models were compared in terms of classi fication accuracy and parameter efficiency. Our comparative analysis demonstrates that the ALBERT Base v2 model outperforms BERT and DistilBERT overall, and our modifiedALBERTmodelshows significantimprovement in classification tasks to de tect severe depression, by focusing on long and complex dependency within words, using a self-attention mechanism. These findings suggest that the use of ALBERT models, particularly with modifications to the attention mechanisms, can improve classification performance in mental health analysis, offering an optimized classifi cation model, utilizing the parameter sharing nature of ALBERT. This work provides valuable insights into the effectiveness of different language model architectures for mental health diagnostics and highlights the impact of attention mechanisms in the mental health domain. | |
| dc.identifier.citation | [1] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” CoRR, vol. abs/1409.0473, 2015. [2] F. Barbieri, J. Camacho-Collados, L. Espinosa Anke, and L. Neves, “Tweete val: Unified benchmark and comparative evaluation for tweet classification,” in FindingsoftheAssociationforComputationalLinguistics:EMNLP2020,2020, pp. 1644–1654. [3] A. T. Beck, Depression: Clinical, Experimental, and Theoretical Aspects. New York: Harper & Row, 1967. [4] C.M.Bishop, Pattern Recognition and Machine Learning. Springer, 2006. [5] S.Bucci, M.Schwannauer, and N.Berry, “The digital revolution and its impact on mental health care,” Psychology and Psychotherapy: Theory, Research and Practice, vol. 92, no. 2, pp. 277–297, 2019. doi: https://doi.org/10.1111/ papt.12222. [Online]. Available: https://bpspsychub.onlinelibrary. wiley.com/doi/abs/10.1111/papt.12222. [6] M.Buda,A.Maki,andM.A.Mazurowski,“Asystematic study of the class im balance problem in convolutional neural networks,” Neural Networks, vol. 106, pp. 249–259, 2018. doi: 10.1016/j.neunet.2018.07.011. [7] A.L.CalearandH.Christensen,“Systematicreviewofschool-basedprevention and early intervention programs for depression,” Journal of adolescence, vol. 33 3, pp. 429–38, 2010. [8] R.A.Calvo,D.N.Milne,M.-A.Hussain,andH.Christensen,“Naturallanguage processing in mental health: Applications, challenges and ethical questions,” Current Opinion in Psychology, vol. 9, pp. 1–6, 2017. [9] R.A.CalvoandD.N.Milne,“Natural language processing in mental health: A scoping review,” Current Opinion in Psychology, vol. 36, pp. 18–24, 2020. doi: 10.1016/j.copsyc.2020.04.005. 62 [10] A. Cohan, B. Desmet, A. Yates, L. Soldaini, S. MacAvaney, and N. Goharian, “Smhd:Alarge-scale resource for exploring online language usage for multiple mental health conditions,” arXiv preprint arXiv:1806.05258, 2018. [11] G. Coppersmith, M. Dredze, and C. Harman, “Quantifying mental health sig nals in twitter,” Proceedings of the Workshop on Computational Linguistics and Clinical Psychology, pp. 51–60, 2014. [12] C.Delahunty,M.O’Neill,andR.Soricut,“Efficiencyandfairnessintransformer based nlp models: Balancing accuracy and deployment cost,” Journal of Artifi cial Intelligence Research, vol. 77, pp. 123–146, 2023. [13] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformersforlanguageunderstanding,”arXivpreprintarXiv:1810.04805, 2018. [14] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in NAACL, 2019. [15] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformersforlanguageunderstanding,”inProceedingsofNAACL HLT, 2019, pp. 4171–4186. [16] G.2.Diseases and I. Collaborators, “Global burden of 369 diseases and injuries in 204 countries and territories, 1990–2019: A systematic analysis for the global burden of disease study 2019,” The Lancet, vol. 396, no. 10258, pp. 1204–1222, 2020, Specifically, depression is highlighted as a leading cause of years lived with disability (YLDs). doi: 10.1016/S0140-6736(20)30925-9. [17] S. K. Ernala and et al., “Methodological biases in social media mental health research,” Journal of Medical Internet Research, 2019. [18] S. K. Ernala et al., “Methodological Gaps in Predicting Mental Health States from SocialMedia:TriangulatingDiagnosticSignals,”inProceedingsofthe2019 CHI Conference on Human Factors in Computing Systems, New York, NY, USA: Association for Computing Machinery, 2019, pp. 1–16, isbn: 9781450359702. doi: 10.1145/3290605.3300364. [Online]. Available: https://dl.acm.org/ doi/abs/10.1145/3290605.3300364. [19] J. Gao, J. Zhang, and Y. Liu, “A survey of attention mechanisms in natural lan guage processing,” Journal of Artificial Intelligence Research, vol. 71, pp. 301 340, 2021. doi: 10.1613/jair.1.12749. [20] I.H.GotlibandJ.Joormann,“Conceptualizingandmeasuringdepression:The role of heterogeneity,” Current Opinion in Psychiatry, vol. 30, pp. 3–10, 1 2017. doi: 10.1097/YCO.0000000000000305. 63 [21] S.C.Guntukuetal.,“Trackingmentalhealthandsymptommentionsontwitter during covid-19,” Journal of General Internal Medicine, vol. 35, pp. 2798–2800, 2020. [22] K. Harrigian, C. Aguirre, and M. Dredze, “On the state of social media data for mental health research: Availability, limitations, and challenges,” in Find ings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 2021, pp. 3467–3479. doi: 10.18653/v1/2021.findings-acl.305. [23] A. Husseini Orabi, P. Buddhitha, M. Husseini Orabi, and D. Inkpen, “Deep learning for depression detection of twitterusers,”ProceedingsoftheFifthWork shop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic, 2018. doi: https://doi.org/10.18653/v1/w18-0609. [24] S.IoffeandC.Szegedy,“Batchnormalization:Acceleratingdeepnetworktrain ingbyreducing internalcovariateshift,”inProceedingsofthe32ndInternational Conference on Machine Learning, 2015, pp. 448–456. [25] M.N.Islam, M. R. Islam, M. H. Kabir, and M. M. Rahman, “A comprehensive review on multimodal depression detection using machine learning and deep learning techniques,” IEEE Access, vol. 10, pp. 48301–48325, 2022. doi: 10. 1109/ACCESS.2022.3172820. [26] R. Johnson, S. Lee, and D. Miller, “The peril of overall accuracy: Misclassifi cation costs in medical diagnostic ai for rare but severe conditions,” Journal of Medical Internet Research, vol. 23, e27890, 7 2021. doi: 10.2196/27890. [27] M.Kabiretal.,“Deptweet:Atypologyforsocialmediatextstodetectdepression severities,” Computers in Human Behavior, vol. 135, p. 107503, 2022. [28] R. Kavuluru, M. Ramos-Morales, T. Holaday, A. G. Williams, L. Haye, and J. Cerel, “Classification of helpful comments on online suicide watch forums,” Proceedings of the 7th ACM International Conference on Bioinformatics, Com putational Biology, and Health Informatics, 2016. [29] J.Kim,J.Lee,E.Park, andJ.Han,“Adeeplearningmodelfordetecting mental illness from user content on socialmedia,”Scientificreports,vol.10,no.1,pp.1 6, 2020. [30] D.P. KingmaandJ. Ba, “Adam: AMethodforStochastic Optimization,” in 3rd International Conference on Learning Representations (ICLR), Y. Bengio and Y. LeCun, Eds., 2015. [Online]. Available: http://arxiv.org/abs/1412.6980. [31] K. Kroenke, R. L. Spitzer, and J. B. W. Williams, “The PHQ-9,” Journal of Gen eral Internal Medicine, vol. 16, no. 9, pp. 606–613, 2001. doi: 10.1046/j.1525 64 1497.2001.016009606.x. [Online]. Available: https://onlinelibrary. wiley.com/doi/abs/10.1046/j.1525-1497.2001.016009606.x. [32] Z.Lan,M.Chen,S.Goodman,K.Gimpel,P.Sharma,andR.Soricut,“Albert:A lite bert forself-supervisedlearningoflanguagerepresentations,”arXivpreprint arXiv:1909.11942, 2019. [33] J. L. Leevy, T. M. Khoshgoftaar, R. A. Bauder, and N. Seliya, “A survey on ad dressing high-class imbalance in big data,” Journal of Big Data, vol. 5, no. 1, pp. 1–30, 2018. doi: 10.1186/s40537- 018- 0151- 6. [Online]. Available: https://link.springer.com/article/10.1186/s40537-018-0151-6. [34] Y.Liuet al., “Roberta: A robustly optimized bert pretraining approach,” ArXiv, vol. abs/1907.11692, 2019. [35] I. Loshchilov and F. Hutter, “Decoupled Weight Decay Regularization,” in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net, 2019. [Online]. Available: https: //openreview.net/forum?id=Bkg6RiCqY7. [36] D. M. Low, L. Rumker, T. Talkar, J. B. Torous, G. A. Cecchi, and S. S. Ghosh, “Natural language processing reveals vulnerable mental health support groups and heightened health anxiety on reddit duringcovid-19:Observationalstudy,” Journal of Medical Internet Research, vol. 22, 2020. [37] S. Parvez, Multiclass sentiment analysis dataset, urlhttps://huggingface.co/datasets/Sp1786/multiclass-sentiment-analysis-dataset, 2023. [38] J. W. Pennebaker, M. R. Mehl, and K. G. Niederhoffer, “The social, linguistic, andpsychologicalcorrelates ofdepression,”JournalofLanguageandSocialPsy chology, vol. 22,no.2,pp.207–226,2003.doi:10.1177/0261927X03022002007. [39] A.G.Reece,A.J.Reagan, K. L. Lix, P. S. Dodds, and C. M. Danforth, “Forecast ing the onset and course of mental illness with twitter data,” Scientific Reports, vol. 7, no. 1, p. 13006, 2017. [40] N.ReimersandI.Gurevych,“Sentence-bert:Sentenceembeddingsusingsiamese bert-networks,” arXiv preprint arXiv:1908.10084, 2019. [41] A.Saif,K.W.Wong,andN.Yahya,“Handlingimbalanceddata:Areview,”Inter nationalJournal ofAdvancedComputerScienceandApplications,vol.11,no.11, pp. 531–540, 2020. doi: 10.14569/IJACSA.2020.0111164. 65 [42] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter,” arXiv preprint arXiv:1910.01108, 2019. [43] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter,” ArXiv, vol. abs/1910.01108, 2019. [44] J. Shen, Y. Wang, X. Zhang, and J. Li, “A survey of publicly available datasets for depression detection and analysis,” Journal of Biomedical and Health Infor matics, vol. 24, pp. 3535–3547, 12 2020. doi: 10.1109/JBHI.2020.2974987. [45] S. Singh, D. Roy, K. Sinha, S. Parveen, G. Sharma, and G. Joshi, “Impact of COVID-19 and lockdown on mental health of children and adolescents: A nar rativereviewwithrecommendations,”en,PsychiatryResearch,vol.293,p.113429, Nov. 2020, issn: 01651781. doi: 10.1016/j.psychres.2020.113429. Ac cessed: Jan. 28, 2022. [Online]. Available: https://linkinghub.elsevier. com/retrieve/pii/S016517812031725X. [46] J.Smith,L.Chen,andR.Gupta,“Limitationsofgeneral-purposelanguagemod els in detecting critical medicalconditions from clinicalnotes,”JournalofBiomed ical Informatics, vol. 123, p. 103987, 2022. doi: 10.1016/j.jbi.2022.103987. [47] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929–1958, 2014. [48] Y. Sun, A. K. Wong, and M. S. Kamel, “Classification of Imbalanced Data: A Review,” International Journal of Pattern Recognition and Artificial Intelligence, vol. 23, no. 04, pp. 687–719, 2009. doi: 10.1142/S0218001409007326. [On line]. Available: https://www.worldscientific.com/doi/abs/10.1142/ S0218001409007326. [49] J. C. Tolentino and S. L. Schmidt, “DSM-5 Criteria and Depression Severity: Implications for Clinical Practice,” Frontiers in Psychiatry, vol. 9, p. 450, Oct. 2018, issn: 1664-0640. doi: 10.3389/fpsyt.2018.00450. Accessed: Jan. 21, 2022. [Online]. Available: https://www.frontiersin.org/article/10. 3389/fpsyt.2018.00450/full. [50] A.Vaswanietal.,“Attentionisallyouneed,”inAdvancesinNeuralInformation Processing Systems, 2017, pp. 5998–6008. [51] A.Vaswaniet al., “Attention is all you need,” ArXiv, vol. abs/1706.03762, 2017. [52] J.Vincent,FacebookisusingAItospotuserswithsuicidalthoughtsandsendthem help, en, Nov. 2017. Accessed: Jan. 21, 2022. [Online]. Available: https://www. 66 theverge.com/2017/11/28/16709224/facebook-suicidal-thoughts-ai help. [53] D. Watson and L. A. Clark, “The panas scales: Development and validation of brief measures of positive and negative affect,” JournalofPersonalityandSocial Psychology, vol. 54, no. 6, pp. 1063–1070, 1994. doi: 10.1037/0022-3514.54. 6.1063. [54] T. Wolf et al., “Transformers: State-of-the-art natural language processing,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020, pp. 38–45. [55] Y. Wu et al., “Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation,” CoRR, vol. abs/1609.08144, 2016. [Online]. Available: http://arxiv.org/abs/1609.08144. [56] S. Yadav, J. Chauhan, J. P. Sain, K. Thirunarayan, A. Sheth, and J. A. Schumm, “Identifying depressive symptoms from tweets: Figurative language enabled multitask learning framework,” in COLING, 2020. [57] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le, “Xlnet: Generalized autoregressive pretraining for language understanding,” Advances in Neural Information Processing Systems, vol. 32, 2019. [58] Y. Zhang, Q. Yang, and H. Sun, “Data augmentation for natural language pro cessing: A survey,” Artificial Intelligence Review, vol. 55, pp. 4281–4330, 2022. [59] Y. Zhang, X. Liu, and X. Chen, “Enhancing depression detection in social me dia text via attention-based deep models,” in Proceedings of the International Conference on Computational Linguistics (COLING), 2022, pp. 2503–2512 | |
| dc.identifier.uri | https://repository.iutoic-dhaka.edu/handle/123456789/2764 | |
| dc.language.iso | en | |
| dc.publisher | Department of Computer Science and Engineering (CSE), Islamic University of Technology (IUT), Board Bazar, Gazipur, Bangladesh | |
| dc.title | Developing modified ALBERT Model for Under-Represented Classes in Depression Detection | |
| dc.type | Thesis |
Files
Original bundle
License bundle
1 - 1 of 1
Loading...
- Name:
- license.txt
- Size:
- 1.71 KB
- Format:
- Item-specific license agreed upon to submission
- Description: