A Framework for Multi-Turn Mental Health Dialogue Expansion and LLM-Based Evaluation

Abstract

In this present world struggling with mental well-being, why would AI stay behind from playing a crucial role in our lives to better manage our mentalhealth. Thispaper introduces a pipeline for building and evaluating multi-turn mental health conversa tion using limited single-turn datasets. Initially, publicly available mental health QA pairs are expanded into realistic multi-turn conversations via prompt-based genera tion. These enriched dialogues are used to fine-tune a LLaMA-based model, which is trained to play the role of a virtual psychiatrist. Tosimulatediverseinteractionscenar ios, the chatbot engages in conversations with other large language models acting as synthetic patients. The resulting dialogues are evaluated by three independent LLMs, each assessing the chatbot’s performance across mental health support metrics such as reliability, bias, sensibility, specifity and interestingness(SSI), safety and security, empathy, robustness and human likeness. Final scores are weighted averaged to en sure balanced evaluation keeping a confidence. Other than these, a sparse human evaluation based comparison among the LLM ratings is also done in this work to see how well the LLMs can produce judgments with respect to human evaluators. This workhighlightsanovelapproachthatcombinessyntheticdataexpansion,LLM-based simulation, and automatedevaluationtoimprovethedevelopmentandassessmentof mental health dialogue systems.