LLM-BasedAuto-Labeling of Developer Discussions AComparative Study of Zero-Shot, Sampling Methods, Ensembles and Judge-Guided Strategies

Abstract

Software bugs have long posed challenges to the delivery of reliable digital services, promptingextensiveresearch intoautomatedbuglabeling. Whilesignificantadvance ments have been made, existing approaches often struggle with high false positive rates and face difficulties in practical deployment due to reliance on structured bug reports. Most contemporary studies utilize structured datasets containing developer generated bug reports, typically written in natural language. These reports require manual or semi-automated extraction of relevant inputs, a process that is both time consuming and error-prone. With the emergence of Large Language Models (LLMs), a new research opportu nity arises: can LLMs effectively extract failure-inducing inputs from unstructured, community-driven sources such as GitHub, Stack Overflow, and other developer fo rums? In this study, we propose a novel end-to-end pipeline that leverages LLMs for bug labeling directly from raw, unstructured text. Our methodology focuses on au tomated labeling, utilizing prompt-based approaches to optimize the performance of generative models. Wecuratedandannotateda datasetcomprising1885StackOverflow questions posted between 2023 and2025,andfurthervalidatedourapproachusingadatasetofGitHub issue reports. Through extensive experimentation, we assess the accuracy and ro bustness of our pipeline across diverse input formats. Unlike existing solutions, our proposed framework emphasizes simplicity, scalability, and cost-effectiveness, mak ing it well-suited for integration into real-world software development workflows.