Study Types
This list of study types is currently a DRAFT and based on a discussion sessions with researchers at the 2024 International Software Engineering Research Network (ISERN) meeting and at the 2nd Copenhagen Symposium on Human-Centered Software Engineering AI.
The development of empirical guidelines for studies involving large language models (LLMs) in software engineering is crucial for ensuring the validity and reproducibility of results. However, these guidelines must be tailored for different study types as they may pose unique challenges. Therefore, understanding the classification of these studies is essential for developing appropriate guidelines. We envision that a mature set of guidelines provides specific guidance for each of these study types, addressing their individual methodological idiosyncrasies.
As outlined on the Motivation and Scope page, we currently focus on large language models, that is, on natural language processing. In the future, we might extend our focus to multimodal foundation models. Moreover, we require an active involvement of LLMs in data collection, processing, or analysis. We consider LLMs supporting writing or LLMs supporting study design to be out-of-scope for our guidelines.
Overview
The development of empirical guidelines for software engineering (SE) studies involving large language models (LLMs) is crucial for ensuring the validity and reproducibility of results. However, these guidelines must be tailored for different study types that pose unique challenges. Therefore, understanding the classification of studies involving LLMs is essential for developing appropriate guidelines. Note that we currently focus on large language models, that is, on natural language processing. In the future, we might extend and generalize our focus to include multimodal foundation models.
Our Guidelines use the following study types to contextualize the recommendations we provide. We present the study types independent of specific guidelines, that is, the guidelines refer to the study types but not the other way around.
Introduction: LLMs as Tools for Software Engineering Researchers
LLMs can be leveraged as powerful tools to assist researchers conducting empirical studies. They can automate various tasks such as data collection, preprocessing, and analysis. For example, LLMs can apply pre-defined coding guides to large qualitative datasets (LLMs as Annotators), assess the quality of software artifacts (LLMs as Judges), generate summaries of research papers (LLMs for Synthesis), and even simulate human behavior in empirical studies (LLMs as Participants). This can significantly reduce the time and effort required by researchers, allowing them to focus on other aspects of their studies. However, besides the many advantages that all these applications bring, they also come with challenges such as potential threats to validity and implications for the reproducibility of study results.
LLMs as Annotators
Description
Just like human annotators, LLMs can label artifacts based on a pre-defined coding guide. However, they can label data much faster than any human could. In qualitative data analysis, manually annotating (“coding”) natural language text, e.g. in software artifacts, open-ended survey responses, or interview transcripts, is a time-consuming manual process [1]. LLMs can be used to augment or even replace human annotations, provide suggestions for new codes (see Section LLMs for Synthesis), or even automate the entire process.
Example(s)
Recent work in software engineering has begun exploring the use of LLMs for annotation tasks. Huang et al. [2] proposed an approach leveraging multiple LLMs for joint annotation of mobile application reviews. They used three models of comparable size with an absolute majority voting rule (i.e., a label is only accepted if it receives more than half of the total votes from the models). Accordingly, the annotations fell into three categories: exact matches (where all models agreed), partial matches (where a majority agreed), and non-matches (where no majority was reached).
The study by Ahmed et al. [3] examined LLMs-as-annotators in software engineering research across five datasets and ten annotation tasks. Using six state-of-the-art LLMs to perform tasks previously done by humans, they found that model-model agreement strongly correlates with human-model agreement, suggesting situations in which LLMs could effectively replace human annotators. Their research showed that for tasks where humans themselves disagree significantly (such as evaluating code conciseness), models also perform poorly. Conversely, if multiple LLMs reach similar solutions independently, then LLMs are likely suitable for the annotation task. They proposed using model confidence scores to identify specific samples that could be safely delegated to LLMs, potentially reducing human annotation effort without compromising inter-rater agreement.
Advantages
Recent research demonstrates several key advantages of using LLMs as annotators, in particular their cost-effectiveness as well as efficiency and accuracy benefits. LLM-based annotation dramatically reduces costs compared to human labeling, with studies showing cost reductions of 50-96% across various natural language tasks [4]. For example, He et al. found in their study that GPT-4 annotation cost only $122.08 compared to $4,508 for a comparable MTurk pipeline. Moreover, the LLM-based approach resulting in a completion time of just 2 days versus several weeks for the crowdsourced approach [5]. Moreover, LLMs consistently demonstrate strong performance, with ChatGPT’s accuracy exceeding crowd workers by approximately 25% on average [6] and achieving impressive results in specific tasks such as sentiment analysis (65% accuracy) and counterspeech detection (79% precision) [7]. They also show remarkably high inter-coder agreement, surpassing both crowd workers and trained annotators [6].
Challenges
Challenges of using LLMs as annotators include reliability issues, human-LLM interaction challenges, biases, errors, and resource considerations. Studies suggest that while LLMs show promise as annotation tools in SE research, their optimal use may be in augmenting rather than replacing human annotators [4], [5]. LLMs can negatively impact human judgment when their labels are incorrect [8], and their overconfidence requires careful verification [9]. Moreover, in previous studies, LLMs have shown significant variability in annotation quality based on the dataset and the annotation task [10]. Studies have shown LLMs to be especially unreliable for high-stakes labeling tasks [11] and have demonstrated that LLMs can have notable performance disparities across label categories [7]. Recent empirical evidence indicates that LLM consistency in text annotation often falls below scientific reliability thresholds, with outputs being sensitive to minor prompt variations [12]. Context-dependent annotations pose a specific challenge, as LLMs show difficulty in correctly interpreting text segments that require broader contextual understanding [5]. While pooling multiple outputs can improve reliability, this approach necessitates additional computational resources and still requires validation against human-annotated data. While generally cost-effective, LLM annotation requires careful management of per-token charges, particularly for longer texts [4]. Furthermore, achieving reliable annotations may require multiple runs of the same input to enable majority voting [12], although exact cost comparisons between LLM-based and human annotation is controversial [5]. Finally, research has identified consistent biases in label assignment, including tendencies to overestimate certain labels and misclassify neutral content, particularly in stance detection tasks [7].
LLMs as Judges
Description
LLMs can act as judges or raters to evaluate properties of software artifacts. For instance, LLMs can be used to assess code readability, adherence to coding standards, or the quality of code comments. Judgment is distinct from the more qualitative task of assigning a code or label to, typically unstructured, text (see Section LLMs as Annotators). It is also distinct from using LLMs for general software engineering tasks, as we discuss in Section LLMs as Tools for Software Engineering Researchers.
Example(s)
Lubos et al. [13] leveraged Llama-2 to evaluate the quality of software requirements statements. They prompted the LLM with the text below, where the words in brackets reflect parameters for the study:
Your task is to evaluate the quality of a software requirement.
Evaluate whether the following requirement is {quality_characteristic}.
{quality_ characteristic} means: {quality_characteristic_explanation}
The evaluation result must be: ‘yes’ or ‘no’.
Request: Based on the following description of the project: {project_description}
Evaluate the quality of the following requirement: {requirement}.
Explain your decision and suggest an improved version.
The authors then evaluated the LLM’s output against human raters, to assess how well the LLM matched experts. Agreement can be measured in many ways; this study used Cohen’s kappa measure [14] and found moderate agreement for simple requirements, and poor agreement for more complex requirements. However, evaluating machine judgment may not be the same as human-human judgments, and is an open area of research [15].
One crucial decision in such studies is the number of examples provided to the LLM. This might involve no tuning (zero-shot), several examples (few-shot), or providing many examples (closer to traditional training data). In the example above, Lubos et al. chose zero-shot tuning, providing no specific guidance besides the project’s context, i.e., they did not show the LLM what a yes or no answer might look like.
Advantages
By providing, depending on the model configuration, relatively consistent and evaluations, LLMs can help mitigate human biases and the variability that human judges might introduce. This may lead to more reliable and reproducible results in empirical studies, to the extent these models can be reproduced or checkpointed. LLMs can be much more efficient, and scale far more easily, than the equivalent human approach. With LLM automation, entire datasets could be assessed, as opposed to subsets. The main constraint, which varies by model and budget, is the input context size, i.e., the number of tokens one can pass into a model. For example, the upper bound on context that can be passed into OpenAi’s o1-mini model is 32k tokens.
Challenges
When relying on the judgment of LLMs, researchers must build a reliable process for generating judgment labels that considers the non-deterministic nature of LLMs and report the intricacies of that process transparently [16]. For example, the order of options has been shown to affect LLM outputs in multiple-choice settings [17]. In addition to reliability, other quality attributes include accuracy of the labels, and the speed and scalability of the LLM tool. A reliable LLM might be reliably inaccurate and wrong. Evaluating and judging large numbers of items—for example, to perform fault localization on the thousands of bugs big open-source projects deal with—comes with costs in clock time, compute time, and environmental sustainability.
Evidence shows LLMs can behave differently, for instance, when reviewing their own outputs [18]. In more human-oriented datasets (such as discussions of pull requests) LLMs may suffer from well documented biases and issues with fairness [19]. For tasks where human judges themselves disagree significantly, it is not clear if an LLM judge should reflect the majority opinion or act as an independent judge. The underlying statistical framework of an LLM usually pushes outputs towards the most likely (majority) answer.
Research is ongoing as to how suitable LLMs are as standalone judges. Questions around bias, accuracy, and trust remain [20]. There is reason for concern about LLMs judging student assignments or doing peer review of scientific papers [21]. Even beyond questions of technical capacity, ethical questions remain, particularly if there is some implicit expectation that a human is judging the output, such as in a quiz. Involving a human in the judgment loop—for example, to contextualize the scoring—is one approach [22]. A lack of large-scale ground truth datasets for benchmarking LLM performance on judgment studies is hindering progress on evaluating research in this area.
LLMs for Synthesis
Description
LLMs can support synthesis tasks in software engineering research by processing and distilling information from qualitative data sources. In this context, synthesis refers to the process of integrating and interpreting information from multiple sources to generate higher-level insights, identify patterns across datasets, and develop conceptual frameworks or theories. Unlike annotation (see Section LLMs as Annotators), which focuses on categorizing or labeling individual data points, synthesis involves connecting and interpreting these annotations to develop a cohesive understanding of the phenomenon being studied.
Example(s)
While published examples of applying LLMs for synthesis in the software engineering domain are still scarce, recent work has explored the use of LLMs for qualitative synthesis in other domains, enabling a reflection on how LLMs can be applied for this purpose in software engineering [1]. Barros et al. [23] conducted a systematic mapping study on the use of LLMs for qualitative research. They identified examples in domains such as healthcare and social sciences (see, e.g., [24], [25]) in which LLMs were used to support different methodologies for qualitative analysis, including grounded theory and thematic analysis. Overall, the findings highlight the successful generation of preliminary coding schemes from interview transcripts, later refined by human researchers, along with support for pattern identification. This approach was reported not only to expedite the initial coding process but also to allow researchers to focus more on higher-level analysis and interpretation. However, they emphasize that effective use of LLMs requires structured prompts and careful human oversight. This particular paper suggests using LLMs to support tasks such as initial coding and theme identification while conservatively reserving interpretative or creative processes for human analysts. Similarly, Lecça et al. [26] conducted a systematic mapping study to investigate how LLMs are used in qualitative analysis and how they can be applied in software engineering research. Consistent with the study by Barros et al. [23], they identified that LLMs are applied primarily in tasks like coding, thematic analysis, and data categorization, providing efficiency by reducing the time, cognitive demands, and resources often required for these processes.
Advantages
LLMs offer promising support for synthesis in SE research by helping to process artifacts such as interview transcripts, survey responses, and literature reviews. Qualitative research in software engineering traditionally faces several challenges, including limited scalability due to the manual nature of the analysis, inconsistencies in coding across researchers, difficulties in generalizing findings from small or context-specific samples, and the influence of researcher subjectivity on data interpretation [1]. The use of LLMs for synthesis can offer advantages in addressing these classical challenges [1], [23], [26]. LLMs can reduce manual effort and subjectivity, improve consistency and generalizability, and assist researchers in deriving codes and developing coding guides during the early stages of qualitative data analysis [1], [27]. The codes generated by LLMs can then be used to annotate additional data, with the models identifying emerging themes and generating candidate insights, potentially automating the entire synthesis process across large qualitative datasets. LLMs can enable researchers to analyze larger datasets, identifying patterns across broader contexts than traditional qualitative methods typically allow. Additionally, they can help mitigate the effects of human subjectivity. Nevertheless, while LLMs streamline many aspects of qualitative synthesis, careful oversight remains essential to ensure nuanced interpretation and contextual accuracy.
Challenges
While LLMs have the potential to automate synthesis, concerns about overreliance remain, especially due to discrepancies between AI- and human-generated insights, particularly in capturing contextual nuances [28]. Bano et al. [28] found that while LLMs can provide structured summaries and qualitative coding frameworks, they may misinterpret nuanced qualitative data due to their lack of contextual understanding. Other studies have echoed similar concerns [1], [23], [26]. In particular, LLMs cannot independently assess argument validity, and critical thinking remains a human responsibility in qualitative synthesis. Peters and Chin-Yee have shown that popular LLMs and LLM-based tools tend to overgeneralize results when summarizing scientific articles, that is they “produce broader generalizations of scientific results than those in the original.” Compared to human-authored summaries, “LLM summaries were nearly five times more likely to contain broad generalizations” [29].
Moreover, LLMs may produce biased results, reinforcing existing prejudices or omitting key perspectives, making human oversight essential to ensure accurate interpretation, mitigate biases, and maintain quality control. The proprietary nature of many LLMs limits transparency. In particular, it is unknown how the inclusion of certain training data might affect the synthesis process. Furthermore, reproducibility issues persist due to inconsistencies across inferences, model versions, and prompt variations.
LLMs as Subjects
Description
In empirical studies, data is collected from participants through methods such as surveys, interviews, or controlled experiments. LLMs can serve as subjects in empirical studies by simulating human behavior and interactions. We use the term “subject” since “participant” implies being human.[1] In this capacity, LLMs generate responses that approximate those of human participants, making them particularly valuable for research involving user interactions, collaborative coding environments, and software usability assessments. This approach enables data collection that closely reflects human reactions while avoiding the need for direct human involvement. To achieve this, prompt engineering techniques are widely employed, with a common approach being the use of the Personas Pattern [30], which involves tailoring LLM responses to align with predefined profiles or roles that emulate specific user archetypes. Zhao et al. outline opportunities and challenges of using LLMs as research subjects in detail [31]. Furthermore, recent sociological studies have emphasized that, to be effectively utilized in this capacity, LLMs—including their agentic versions tailored through prompt engineering—should meet four criteria of algorithmic fidelity [32]; generated responses should be: indistinguishable from human-produced texts (e.g., LLM-generated code reviews should be comparable to those from real developers); consistent with the attitudes and sociodemographic information of the conditioning context (e.g., LLMs simulating junior developers should exhibit different confidence levels, vocabulary, and concerns compared to senior engineers); naturally aligned with the form, tone, and content of the provided context (e.g., responses in an agile stand-up meeting simulation should be concise, task-focused, and aligned with sprint objectives rather than long, formal explanations); and reflective of patterns in relationships between ideas, demographics, and behavior observed in comparable human data (e.g., discussions on software architecture decisions should capture trade-offs typically debated by human developers, such as maintainability versus performance, rather than abstract theoretical arguments).
Example(s)
LLMs can be used as subjects in various types of empirical studies, enabling researchers to simulate human participants. The broader applicability of LLM-based studies beyond software engineering has been compiled by Xu et al. [33], who examined various uses in social science research. Given the socio-technical nature of software development, some of these approaches should be transferable to empirical software engineering research.
For example, LLMs can be applied in survey and interview studies to impersonate developers responding to survey questionnaires or interviews, allowing researchers to test the clarity and effectiveness of survey items or to simulate responses under varying conditions, such as different expertise levels or cultural contexts. For instance, Gerosa et al. [34] explored persona-based interviews and multi-persona focus groups, demonstrating how LLMs can emulate human responses and behaviors while addressing ethical concerns, biases, and methodological challenges. Another example are usability studies, in which LLMs can simulate end-user feedback, providing insights into potential usability issues and offering suggestions for improvement based on predefined user personas. This aligns with the work of Bano et al. [35], who investigated biases in LLM-generated candidate profiles in software engineering recruitment processes. Their study, which analyzed both textual and visual inputs, revealed biases favoring male, Caucasian candidates, lighter skin tones, and slim physiques, particularly for senior roles.
Advantages
Using LLMs as subjects offers valuable insights while significantly reducing the need to recruit human participants—a process that is often time-consuming and costly [36]—by augmenting existing datasets or, in some cases, even replacing human participants. Their use has potential applications across a wide range of empirical research methods. In interviews and survey studies, they can simulate diverse respondent profiles, thus enhancing scalability and enabling access to underrepresented populations, which can strengthen the generalizability of findings. In data mining studies, LLMs can generate synthetic data to fill gaps where real-world data is unavailable or underrepresented.
Challenges
However, it is important that researchers are aware of LLMs’ inherent biases [37] and limitations [38], [39] when using them as study subjects. One critical concern is construct validity. LLMs have been shown to misrepresent demographic group perspectives, failing to capture the diversity of opinions and experiences within a group. The use of identity-based prompts can reduce identities to fixed and innate characteristics, amplifying perceived differences between groups. These biases introduce a risk that studies relying on LLM-generated responses may inadvertently reinforce stereotypes or misrepresent real-world social dynamics. To mitigate these issues, encoded identity names—statistically representative person names associated with specific demographic groups—can be used instead of explicit labels (e.g., using “Mario Rossi” instead of the label “Italian white man”), the temperature setting can be increased to enhance response diversity, and alternatives to demographic prompts can be employed when the goal is to broaden response coverage [39]. All three strategies aim to reduce the risk of stereotyped, flattened, or overly deterministic outputs by introducing indirectness or variability into the prompt, thereby encouraging the model to generate responses that better reflect the internal diversity and nuance of human perspectives. Beyond construct validity, internal validity must also be considered, particularly regarding causal conclusions based on studies relying on LLM-simulated responses. External validity also remains a challenge, as findings based on LLMs may not generalize to humans.
Introduction: LLMs as Tools for Software Engineers
LLM-based assistants have become an essential tool for software engineers, supporting them in various tasks such as code generation, summarization, and repair. Researchers have studied how software engineers use LLMs (Studying LLM Usage in Software Engineering), developed new tools that integrate LLMs (LLMs for New Software Engineering Tools), and benchmarked LLMs for software engineering tasks (Benchmarking LLMs for Software Engineering Tasks). Like in the previous section, we outline the advantages that studying LLMs in these contexts brings, but also point to specific challenges.
Studying LLM Usage in Software Engineering
Description
Studying how software engineers use LLMs in their workflows is crucial for understanding the current state of practice in software engineering. Researchers can observe software engineers’ usage of LLM-based tools in the field, or study if and how they adopt such tools, their usage patterns, as well as perceived benefits and challenges. Surveys, interviews, observational studies, or the analysis of usage logs, can provide insights into how LLMs are integrated into development processes, how they influence decision-making, and what factors affect their acceptance and effectiveness. Such studies can inform improvements for existing LLM-based tools, motivate the design of novel tools, or derive best practices for LLM-assisted software engineering. However, they can also uncover risks or deficiencies of the tools.
Example(s)
Khojah et al. investigated the use of ChatGPT by professional software engineers in a week-long observational study [40] (GPT-3.5, data collection between March and July 2023). They found that most developers do not use code generated by ChatGPT directly, but instead used the outputs as guidance to implement their own solutions. Furthermore, they recommend that future research investigate the use of ChatGPT in non code-related SE tasks and for training and learning SE concepts. Ananza et al. conducted a case study evaluating the impact of introducing LLMs to the onboarding process of new software developers [41] (GPT-3.5, published 2024). Their study identified potential in the use of LLMs for onboarding, as it may allow newcomers to seek information on their own, without the need to “bother” senior colleagues. However, they also highlight significant privacy concerns with the use of proprietary, third-party LLMs. Surveys can help researchers quickly provide a wider overview of the current perceptions of LLM use. For example, Jahic and Sami surveyed participants from 15 software companies regarding their practices on LLMs in software engineering [42] (published 2024). They found that the majority of study participants had already adopted AI for software engineering tasks, with most of them using ChatGPT. Multiple participants cited copyright and privacy issues, as well as the inconsistent or low quality outputs as barriers for adoption. Retrospective studies analyzing the data generated during the use of LLMs by software engineers can provide additional insights into human LLM interactions. For example, researchers can employ data mining methods to build large-scale conversation datasets, such as the DevGPT dataset introduced by Xiao et al [43] (data collection started in August 2023). Conversations can then be analyzed using quantitative [44] and qualitative [45] analysis methods (both studies use the DevGPT dataset).
Advantages
Studying the real-world usage of LLM-based tools allows researchers to understand the state of practice and guide future research directions. In field studies, researchers may uncover usage patterns, adoption rates, and influence of contextual factors on the usage behavior. Outside of controlled study environment, researchers may uncover nuanced information about LLM-assisted SE workflows beyond the specific LLMs being evaluated. This may, for example, help researchers in generating hypotheses about how LLMs impact developer productivity, collaboration, and decision-making processes.
In controlled laboratory studies, researchers can study specific phenomena related to LLM usage under carefully regulated, but potentially artificial, conditions. Specifically, they may isolate individual tasks in the software engineering workflow and investigate how LLM-based tools may support the task completion. Furthermore, controlled experiments allow for direct comparisons between different LLM-based tools. Results from these experiments can then be used validate general hypotheses about human LLM interactions in software engineering.
Challenges
When conducting field studies in real-world environments, researchers have to ensure that their study results are “dependable” [46] beyond the traditional validity criteria such as internal or construct validity. The usage environment and choice of LLMs in a real-world context can often be extremely diverse. In practice, the process integration of LLMs can range from the use of specific LLMs based on company policy to the unregulated use of any available LLM. Both extremes may influence the use of LLMs by software engineers in different ways, and as such should be addressed differently by the study methodology. Further, in longitudinal case studies, the timing of the study may have a significant impact on its result, as LLMs are quickly being developed and replaced by newer versions. This difficulty is exacerbated by the relative novelty of LLMs in the SE process. Developers are still learning how to best make use of the new technology and best practices are being established. Additionally, the predominance of proprietary commercial LLM-based tools in the market poses a significant barrier to research. Limited access to telemetry data or other usage metrics restricts the ability of researchers to conduct comprehensive analyses of real-world tool utilization. To nevertheless make study results dependable, researchers need to establish that their results are trustworthy and rigorous, e.g., by using triangulation, to understand and minimize potential biases [46].
When it comes to studying LLM usage in controlled laboratory studies, researchers may struggle with the inherent variability of LLM outputs. Since reproducibility and precision are essential for hypothesis testing, the stochastic nature of LLM responses—where identical prompts may yield different outputs across participants—can complicate the interpretation of experimental results and affect the reliability of study findings.
LLMs for New Software Engineering Tools
Description
LLMs are being integrated into new tools supporting software engineers in their daily tasks, e.g., code comprehension [47] and test cases generation [48]. Such integration is important to tailor the tools to the specific needs of a development team, enhance their capabilities, and influence their behavior in accordance with users’ needs. An evolution of such integration are GenAI Agents. Unlike simple applications powered by LLMs, these agents are capable of acting autonomously and proactively, are often tailored to meet specific user needs, and can interact with external environments [49], [50]. From an architectural perspective, GenAI Agents can be implemented in various ways [50]. However, they generally share three key components: a reasoning mechanism that guides the LLM (often enabled by advanced prompt engineering), a set of tools for interfacing with external systems (e.g., APIs or databases), and a user communication interface that extends beyond traditional chat-based interactions [51], [52], [53]. Other than propose tools, researcher can also evaluate the effectiveness of these tools in improving developer experience (for example, in terms of productivity, artefact quality, and developer satisfaction) and testing with different implementations of the above mentioned architecture in order to improve the tools.
Example(s)
As an example of LLMs-powered tool, Yan et al. proposed IVIE, a tool integrated into the VS Code graphical interface that generates and explains code using LLMs [47]. In this case, the authors focused more on the presentation part of the architecture, providing a user-friendly interface to interact with the LLM. Furthermore, Schäfer et al. [48] presented a large-scale empirical evaluation on the effectiveness of LLMs for automated unit test generation. Concretely, they presented TestPilot: a tool implementing an approach where the LLM is provided with prompts that include the signature and implementation of a function under test, along with usage examples extracted from documentation. More on the agentic side, Richards and Wessel introduced a preliminary GenAI agent designed to assist developers in understanding source code by incorporating a reasoning component grounded in the theory of mind [51]. Their work specifically emphasized the reasoning aspect of the architecture, leveraging the theory of mind to develop tailored agents while also demonstrating their effectiveness.
Advantages
From an engineering perspective, developing LLM-based tools is easier than implementing many traditional SE approaches such as static analysis or symbolic execution. Depending on the capabilities of the underlying model, it is also easier to build tools that are independent of a specific programming language. This enables researchers to build tools for a more diverse set of tasks. It furthermore allows them to test their tools in a wider range of contexts.
Challenges
Traditional approaches such as static analysis are deterministic. LLMs are not. Although the non-determinism of LLMs can be mitigated using parameters and prompting strategies, this poses a major challenge. It can be challenging for researchers to evaluate the effectiveness of their tool, as minor changes in the input can lead to major differences in the performance. When using commercial LLMs, researchers face challenges because the training data is unknown, hindering a reliable assessment of tool performance for unknown data. From an engineering perspective, while open models are available, the most performant ones require substantial hardware resources. Resorting to cloud-based APIs or relying on third-party providers for hosting, while seemingly a potential solution, introduces new concerns related to data privacy and security.
Benchmarking LLMs for Software Engineering Tasks
Description
Benchmarking is the process of evaluating the LLM output for standardized tasks using standardized metrics. Hence, high-quality reference datasets are necessary to perform evaluation across studies. For example, HumanEval [54] is increasingly being used for the code generation, establishing it as de-facto standard for this task. LLM output is compared against a ground truth in the benchmark dataset using general metrics for text generation, such as ROUGE, BLEU, or METEOR [55], or task-specific metrics, such as CodeBLEU for code generation.
Example(s)
In software engineering, benchmarking may include the evaluation of an LLM’s ability to produce accurate and reliable outputs for a given input, usually a task description which may be accompanied by data obtained from curated real-world projects or from synthetic SE-specific datasets (e.g., for few-shot prompting). Typical tasks include code generation, code summarization, code completion, and code repair, but also natural-language processing tasks, such as anaphora resolution (i.e., the task of identifying the referring expression of a word or phrase occurring earlier in the text), interesting for subfields such a Requirements Engineering.
RepairBench [56], for example, contains 574 buggy Java methods and their corresponding fixed versions, which can be used to evaluate the performance of LLMs in code repair tasks. The metrics used are Plausible@1 (i.e., the probability that the first generated patch passes all test cases) and AST Match@1 (i.e., the probability that the abstract syntax tree of the first generated patch matches the one of the ground truth patch). SWE-Bench [57] is a more generic benchmark that contains 2,294 SE Python tasks extracted from GitHub pull requests. For scoring the performance of the LLMs on the tasks, the authors report whether the generated patch is applicable (i.e., it fails compilation) and, for successful patches, the percentage of test cases passed.
Advantages
Properly-built benchmarks provide objective, reproducible evaluation across different tasks, enabling a comparison between different models (and versions). Moreover, benchmarks built for specific SE tasks can help identify LLM weaknesses and support their optimization/fine-tuning for such tasks. Benchmark built using real-world data can also help legitimize research results for practitioners, supporting industry-academia collaboration. Benchmarks are, however, subject to the issues outlined below. Finally, benchmarks can foster open science practices by providing a common ground for sharing data (e.g., as part of the benchmark itself) and results (e.g., of models run against a benchmark).
Challenges
Benchmark contamination, the inclusion of the benchmark in the training dataset of the LLM, [58] has recently been identified as an issue. The careful selection of samples and building of corresponding input prompts is particularly important, as correlations between prompts may bias benchmark results [59]. While LLMs might perform well on a specific benchmark, such as HumanEval, it does not necessarily perform well on another benchmark. Benchmark metrics such as perplexity or BLEU-N do not always reflect human judgment. Recently, Cao et al. [60] has proposed guidelines for building benchmarks for LLMs related to coding tasks, grounded in a systematic survey of existing benchmarks. In this process, they highlight current shortcomings related to reliability, transparency irreproducibility, low data quality, and inadequate validation measures. Comment: Previous comment not addressed yet: I like the last reference to guildelines for building a benchmark. Should we include some challenges from the paper, too?
References
[1] M. Bano, R. Hoda, D. Zowghi, and C. Treude, “Large language models for qualitative research in software engineering: Exploring opportunities and challenges,” Autom. Softw. Eng., vol. 31, no. 1, p. 8, 2024, doi: 10.1007/S10515-023-00407-8.
[2] J. Huang et al., “Enhancing review classification via LLM-based data annotation and multi-perspective feature representation learning,” SSRN Electronic Journal, pp. 1–15, 2024, doi: 10.2139/ssrn.5002351.
[3] T. Ahmed, P. Devanbu, C. Treude, and M. Pradel, “Can LLMs replace manual annotation of software engineering artifacts?” arXiv, Feb. 04, 2025. doi: 10.48550/arXiv.2408.05534.
[4] S. Wang, Y. Liu, Y. Xu, C. Zhu, and M. Zeng, “Want to reduce labeling cost? GPT-3 can help,” in Findings of the association for computational linguistics: EMNLP 2021, virtual event / punta cana, dominican republic, 16-20 november, 2021, M.-F. Moens, X. Huang, L. Specia, and S. W. Yih, Eds., Association for Computational Linguistics, 2021, pp. 4195–4205. doi: 10.18653/V1/2021.FINDINGS-EMNLP.354.
[5] Z. He, C.-Y. Huang, C.-K. C. Ding, S. Rohatgi, and T.-H. K. Huang, “If in a crowdsourced data annotation pipeline, a GPT-4,” in Proceedings of the CHI conference on human factors in computing systems, CHI 2024, honolulu, HI, USA, may 11-16, 2024, F. ’Floyd’Mueller, P. Kyburz, J. R. Williamson, C. Sas, M. L. Wilson, P. O. T. Dugas, and I. Shklovski, Eds., ACM, 2024, pp. 1040:1–1040:25. doi: 10.1145/3613904.3642834.
[6] F. Gilardi, M. Alizadeh, and M. Kubli, “ChatGPT outperforms crowd-workers for text-annotation tasks,” CoRR, vol. abs/2303.15056, 2023, doi: 10.48550/ARXIV.2303.15056.
[7] Y. Zhu, P. Zhang, E. ul Haq, P. Hui, and G. Tyson, “Can ChatGPT reproduce human-generated labels? A study of social computing tasks,” CoRR, vol. abs/2304.10145, 2023, doi: 10.48550/ARXIV.2304.10145.
[8] F. Huang, H. Kwak, and J. An, “Is ChatGPT better than human annotators? Potential and limitations of ChatGPT in explaining implicit hate speech,” in Companion proceedings of the ACM web conference 2023, WWW 2023, austin, TX, USA, 30 april 2023 - 4 may 2023, Y. Ding, J. Tang, J. F. Sequeda, L. Aroyo, C. Castillo, and G.-J. Houben, Eds., ACM, 2023, pp. 294–297. doi: 10.1145/3543873.3587368.
[9] M. Wan et al., “TnT-LLM: Text mining at scale with large language models,” in Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, KDD 2024, barcelona, spain, august 25-29, 2024, R. Baeza-Yates and F. Bonchi, Eds., ACM, 2024, pp. 5836–5847. doi: 10.1145/3637528.3671647.
[10] N. Pangakis, S. Wolken, and N. Fasching, “Automated annotation with generative AI requires validation,” CoRR, vol. abs/2306.00176, 2023, doi: 10.48550/ARXIV.2306.00176.
[11] X. Wang, H. Kim, S. Rahman, K. Mitra, and Z. Miao, “Human-LLM collaborative annotation through effective verification of LLM labels,” in Proceedings of the CHI conference on human factors in computing systems, CHI 2024, honolulu, HI, USA, may 11-16, 2024, F. ’Floyd’Mueller, P. Kyburz, J. R. Williamson, C. Sas, M. L. Wilson, P. O. T. Dugas, and I. Shklovski, Eds., ACM, 2024, pp. 303:1–303:21. doi: 10.1145/3613904.3641960.
[12] M. V. Reiss, “Testing the reliability of ChatGPT for text annotation and classification: A cautionary remark,” CoRR, vol. abs/2304.11085, 2023, doi: 10.48550/ARXIV.2304.11085.
[13] S. Lubos et al., “Leveraging LLMs for the quality assurance of software requirements,” in 32nd IEEE international requirements engineering conference, RE 2024, reykjavik, iceland, june 24-28, 2024, G. Liebel, I. Hadar, and P. Spoletini, Eds., IEEE, 2024, pp. 389–397. doi: 10.1109/RE59067.2024.00046.
[14] J. Cohen, “A coefficient of agreement for nominal scales,” Educational and Psychological Measurement, vol. 20, no. 1, pp. 37–46, Apr. 1960, doi: 10.1177/001316446002000104.
[15] A. Elangovan et al., “Beyond correlation: The impact of human uncertainty in measuring the effectiveness of automatic evaluation and LLM-as-a-judge,” CoRR, vol. abs/2410.03775, 2024, doi: 10.48550/ARXIV.2410.03775.
[16] K. Schroeder and Z. Wood-Doughty, “Can you trust LLM judgments? Reliability of LLM-as-a-judge,” CoRR, vol. abs/2412.12509, 2024, doi: 10.48550/ARXIV.2412.12509.
[17] P. Pezeshkpour and E. Hruschka, “Large language models sensitivity to the order of options in multiple-choice questions,” in Findings of the association for computational linguistics: NAACL 2024, mexico city, mexico, june 16-21, 2024, K. Duh, H. Gómez-Adorno, and S. Bethard, Eds., Association for Computational Linguistics, 2024, pp. 2006–2017. doi: 10.18653/V1/2024.FINDINGS-NAACL.130.
[18] A. Panickssery, S. Bowman, and S. Feng, “LLM evaluators recognize and favor their own generations,” in Advances in neural information processing systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, Eds., Curran Associates, Inc., 2024, pp. 68772–68802. Available: https://proceedings.neurips.cc/paper_files/paper/2024/file/7f1f0218e45f5414c79c0679633e47bc-Paper-Conference.pdf
[19] I. O. Gallegos et al., “Bias and fairness in large language models: A survey,” Computational Linguistics, vol. 50, pp. 1097–1179, 2024, doi: 10.1162/coli_a_00524.
[20] A. Bavaresco et al., “LLMs instead of human judges? A large scale empirical study across 20 NLP evaluation tasks,” CoRR, vol. abs/2406.18403, 2024, doi: 10.48550/ARXIV.2406.18403.
[21] R. Zhou, L. Chen, and K. Yu, “Is LLM a reliable reviewer? A comprehensive evaluation of LLM on automatic paper reviewing tasks,” in Proceedings of the 2024 joint international conference on computational linguistics, language resources and evaluation, LREC/COLING 2024, 20-25 may, 2024, torino, italy, N. Calzolari, M.-Y. Kan, V. Hoste, A. Lenci, S. Sakti, and N. Xue, Eds., ELRA; ICCL, 2024, pp. 9340–9351. Available: https://aclanthology.org/2024.lrec-main.816
[22] Q. Pan et al., “Human-Centered Design Recommendations for LLM-as-a-Judge,” in ACL 2024 Workshop HuCLLM, arXiv, Jul. 2024. doi: 10.48550/arXiv.2407.03479.
[23] C. F. Barros et al., “Large language model for qualitative research – a systematic mapping study.”
- Available: https://arxiv.org/abs/2411.14473</span>
[24] S. De Paoli, “Performing an inductive thematic analysis of semi-structured interviews with a large language model: An exploration and provocation on the limits of the approach,” Social Science Computer Review, vol. 42, no. 4, pp. 997–1019, 2024.
[25] W. S. Mathis, S. Zhao, N. Pratt, J. Weleff, and S. De Paoli, “Inductive thematic analysis of healthcare qualitative interviews using open-source large language models: How does it compare to traditional methods?” Computer Methods and Programs in Biomedicine, vol. 255, p. 108356, 2024.
[26] M. de Morais Leça, L. Valença, R. Santos, and R. de Souza Santos, “Applications and implications of large language models in qualitative analysis: A new frontier for empirical software engineering.” 2024. Available: https://arxiv.org/abs/2412.06564
[27] C. Byun, P. Vasicek, and K. D. Seppi, “Dispensing with humans in human-computer interaction research,” in Extended abstracts of the 2023 CHI conference on human factors in computing systems, CHI EA 2023, hamburg, germany, april 23-28, 2023, A. Schmidt, K. Väänänen, T. Goyal, P. O. Kristensson, and A. Peters, Eds., ACM, 2023, pp. 413:1–413:26. doi: 10.1145/3544549.3582749.
[28] M. Bano, D. Zowghi, and J. Whittle, “Exploring qualitative research using LLMs.” 2023. Available: https://arxiv.org/abs/2306.13298
[29] U. Peters and B. Chin-Yee, “Generalization bias in large language model summarization of scientific research,” R. Soc. Open Sci., vol. 12, no. 12241776, 2025, doi: 10.1098/rsos.241776.
[30] A. Kong et al., “Better zero-shot reasoning with role-play prompting,” CoRR, vol. abs/2308.07702, 2023, doi: 10.48550/ARXIV.2308.07702.
[31] C. Zhao, M. Habule, and W. Zhang, “Large language models (LLMs) as research subjects: Status, opportunities and challenges,” New Ideas in Psychology, vol. 79, p. 101167, 2025, doi: https://doi.org/10.1016/j.newideapsych.2025.101167.
[32] L. P. Argyle, E. C. Busby, N. Fulda, J. Gubler, C. M. Rytting, and D. Wingate, “Out of one, many: Using language models to simulate human samples,” CoRR, vol. abs/2209.06899, 2022, doi: 10.48550/ARXIV.2209.06899.
[33] R. Xu et al., “AI for social science and social science of AI: A survey,” Inf. Process. Manag., vol. 61, no. 2, p. 103665, 2024, doi: 10.1016/J.IPM.2024.103665.
[34] M. A. Gerosa, B. Trinkenreich, I. Steinmacher, and A. Sarma, “Can AI serve as a substitute for human subjects in software engineering research?” Autom. Softw. Eng., vol. 31, no. 1, p. 13, 2024, doi: 10.1007/S10515-023-00409-6.
[35] M. Bano, H. Gunatilake, and R. Hoda, “What does a software engineer look like? Exploring societal stereotypes in LLMs.” 2025. Available: https://arxiv.org/abs/2501.03569
[36] K. Madampe, J. Grundy, R. Hoda, and H. O. Obie, “The struggle is real! The agony of recruiting participants for empirical software engineering studies,” in 2024 IEEE symposium on visual languages and human-centric computing (VL/HCC), liverpool, UK, september 2-6, 2024, IEEE, 2024, pp. 417–422. doi: 10.1109/VL/HCC60511.2024.00065.
[37] R. Crowell, “Why AI’s diversity crisis matters, and how to tackle it,” Nature Career Feature, 2023, doi: 10.1038/d41586-023-01689-4.
[38] J. Harding, W. D’Alessandro, N. G. Laskowski, and R. Long, “AI language models cannot replace human research participants,” AI Soc., vol. 39, no. 5, pp. 2603–2605, 2024, doi: 10.1007/S00146-023-01725-X.
[39] A. Wang, J. Morgenstern, and J. P. Dickerson, “Large language models cannot replace human participants because they cannot portray identity groups,” CoRR, vol. abs/2402.01908, 2024, doi: 10.48550/ARXIV.2402.01908.
[40] R. Khojah, M. Mohamad, P. Leitner, and F. G. de Oliveira Neto, “Beyond code generation: An observational study of ChatGPT usage in software engineering practice,” Proc. ACM Softw. Eng., vol. 1, no. FSE, pp. 1819–1840, 2024, doi: 10.1145/3660788.
[41] M. Azanza, J. Pereira, A. Irastorza, and A. Galdos, “Can LLMs facilitate onboarding software developers? An ongoing industrial case study,” in 36th international conference on software engineering education and training, CSEE&t 2024, würzburg, germany, july 29 - aug. 1, 2024, IEEE, 2024, pp. 1–6. doi: 10.1109/CSEET62301.2024.10662989.
[42] J. Jahic and A. Sami, “State of practice: LLMs in software engineering and software architecture,” in 21st IEEE international conference on software architecture, ICSA 2024 - companion, hyderabad, india, june 4-8, 2024, IEEE, 2024, pp. 311–318. doi: 10.1109/ICSA-C63560.2024.00059.
[43] T. Xiao, C. Treude, H. Hata, and K. Matsumoto, “DevGPT: Studying developer-ChatGPT conversations,” in 21st IEEE/ACM international conference on mining software repositories, MSR 2024, lisbon, portugal, april 15-16, 2024, D. Spinellis, A. Bacchelli, and E. Constantinou, Eds., ACM, 2024, pp. 227–230. doi: 10.1145/3643991.3648400.
[44] Md. F. Rabbi, A. I. Champa, M. F. Zibran, and Md. R. Islam, “AI writes, we analyze: The ChatGPT python code saga,” in 21st IEEE/ACM international conference on mining software repositories, MSR 2024, lisbon, portugal, april 15-16, 2024, D. Spinellis, A. Bacchelli, and E. Constantinou, Eds., ACM, 2024, pp. 177–181. doi: 10.1145/3643991.3645076.
[45] S. Mohamed, A. Parvin, and E. Parra, “Chatting with AI: Deciphering developer conversations with ChatGPT,” in 21st IEEE/ACM international conference on mining software repositories, MSR 2024, lisbon, portugal, april 15-16, 2024, D. Spinellis, A. Bacchelli, and E. Constantinou, Eds., ACM, 2024, pp. 187–191. doi: 10.1145/3643991.3645078.
[46] G. M. Sullivan and J. Sargeant, “Qualities of qualitative research: Part I,” J Grad Med Educ, vol. 3, no. 4, pp. 449–452, Dec. 2011.
[47] L. Yan, A. Hwang, Z. Wu, and A. Head, “Ivie: Lightweight anchored explanations of just-generated code,” in Proceedings of the CHI conference on human factors in computing systems, CHI 2024, honolulu, HI, USA, may 11-16, 2024, F. ’Floyd’Mueller, P. Kyburz, J. R. Williamson, C. Sas, M. L. Wilson, P. O. T. Dugas, and I. Shklovski, Eds., ACM, 2024, pp. 140:1–140:15. doi: 10.1145/3613904.3642239.
[48] M. Schäfer, S. Nadi, A. Eghbali, and F. Tip, “An empirical evaluation of using large language models for automated unit test generation,” IEEE Trans. Software Eng., vol. 50, no. 1, pp. 85–105, 2024, doi: 10.1109/TSE.2023.3334955.
[49] W. Takerngsaksiri et al., “Human-in-the-loop software development agents,” arXiv preprint arXiv:2411.12924, 2024.
[50] J. Wiesinger, P. Marlow, and V. Vuskovic, “Agents.” Google DeepMind; Whitepaper, 2025. Available: https://gemini.google.com
[51] J. Richards and M. Wessel, “What you need is what you get: Theory of mind for an LLM-based code understanding assistant,” in IEEE international conference on software maintenance and evolution, ICSME 2024, flagstaff, AZ, USA, october 6-11, 2024, IEEE, 2024, pp. 666–671. doi: 10.1109/ICSME58944.2024.00070.
[52] T. R. Sumers, S. Yao, K. Narasimhan, and T. L. Griffiths, “Cognitive architectures for language agents,” Trans. Mach. Learn. Res., vol. 2024, 2024, Available: https://openreview.net/forum?id=1i6ZCvflQJ
[53] W. Zhou et al., “Agents: An open-source framework for autonomous language agents,” CoRR, vol. abs/2309.07870, 2023, doi: 10.48550/ARXIV.2309.07870.
[54] M. Chen et al., “Evaluating large language models trained on code,” CoRR, vol. abs/2107.03374, 2021, Available: https://arxiv.org/abs/2107.03374
[55] X. Hou et al., “Large language models for software engineering: A systematic literature review,” ACM Trans. Softw. Eng. Methodol., vol. 33, no. 8, Dec. 2024, doi: 10.1145/3695988.
[56] A. Silva and M. Monperrus, “RepairBench: Leaderboard of frontier models for program repair,” arXiv preprint arXiv:2409.18952, 2024.
[57] C. E. Jimenez et al., “SWE-bench: Can language models resolve real-world github issues?” in The twelfth international conference on learning representations, ICLR 2024, vienna, austria, may 7-11, 2024, OpenReview.net, 2024. Available: https://openreview.net/forum?id=VTF8yNQM66
[58] S. Ahuja, V. Gumma, and S. Sitaram, “Contamination report for multilingual benchmarks,” CoRR, vol. abs/2410.16186, 2024, doi: 10.48550/ARXIV.2410.16186.
[59] C. Siska, K. Marazopoulou, M. Ailem, and J. Bono, “Examining the robustness of LLM evaluation to the distributional assumptions of benchmarks,” in Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), ACL 2024, bangkok, thailand, august 11-16, 2024, L.-W. Ku, A. Martins, and V. Srikumar, Eds., Association for Computational Linguistics, 2024, pp. 10406–10421. doi: 10.18653/V1/2024.ACL-LONG.560.
[60] J. Cao et al., “How should i build a benchmark?” arXiv preprint arXiv:2501.10711, 2025.