This page compares the paper body, appendices, and references. Deletions are struck through; additions are underlined. Sections follow the order of the newer release.
16 changed sections.
Introduction Changed
2026.03
Introduction
Since the release of ChatGPT in November 2022, large language models (LLMs) have rapidly become a major focus of software engineering (SE) research (Hou et al. 2024), yet the reproducibility and replicability of empirical studies involving LLMs remains uncertain. Recent findings indicate a low reproducibility of SE studies involving LLMs (Angermeir et al. 2025). Three characteristics of LLMs make this particularly challenging: their inherent non-determinism causes variability across runs, with slight changes leading to significantly different results (Schroeder and Wood-Doughty 2024; Song et al. 2025; Agarwal et al. 2021; Bjarnason, Silva, and Monperrus 2026); commercial models evolve beyond version identifiers, so reported performance can change over time (L. Chen, Zaharia, and Zou 2024); and even for “open” models, training data and fine-tuning details often remain undisclosed (Gibney 2024). Moreover, prompt formatting choices alone can shift accuracy by tens of percentage points (Sclar et al. 2024), and configured parameters such as temperature affect output variability (Renze 2024). Hence, not reporting these settings directly affects reproducibility.
Although the SE research community has developed guidelines for conducting and reporting specific types of empirical studies such as controlled experiments (e.g., (Wohlin et al. 2024; Shull, Singer, and Sjøberg 2008)), their replications (e.g., (Santos et al. 2021)), or empirical studies in general (e.g., the ACM SIGSOFT Empirical Standards(Ralph et al. 2021)), none of these address the LLM-specific aspects described above. Previously, a position paper highlighted these issues (Wagner et al. 2025), but there was no comprehensive community-developed guidance for designing and reporting empirical studies involving LLMs in SE.
Therefore, we present community-developed guidelines for conducting and reporting studies involving LLMs in SE research, co-developed by 22 researchers. After outlining our Scope, we introduce a taxonomy of Study Types, then present eight Guidelines. We complement these with an applicability matrix mapping guidelines to study types and a reporting checklist for authors and reviewers. For each study type and guideline, we identify relevant examples, both within and outside of SE research. We maintain the guidelines online as a living resource for the community to use and shape.1
2026.04
Introduction
Since the release of ChatGPT in November 2022, large language models (LLMs) have rapidly become a major focus of software engineering (SE) research (Hou et al. 2024), yet the reproducibility and replicability of empirical studies involving LLMs remains uncertain. Recent findings indicate a low reproducibility of SE studies involving LLMs (Angermeir et al. 2025). Three characteristics of LLMs make this particularly challenging: their inherent non-determinism causes variability across runs, with slight changes leading to significantly different results (Schroeder and Wood-Doughty 2024; Song et al. 2025; Agarwal et al. 2021; Bjarnason, Silva, and Monperrus 2026); commercial models evolve beyond version identifiers, so reported performance can change over time (L. Chen, Zaharia, and Zou 2024); and even for “open” models, training data and fine-tuning details often remain undisclosed (Gibney 2024). Moreover, prompt formatting choices alone can shift accuracy by tens of percentage points (Sclar et al. 2024), and configured parameters such as temperature affect output variability (Renze 2024). Hence, not reporting these settings directly affects reproducibility.
Although the SE research community has developed guidelines for conducting and reporting specific types of empirical studies such as controlled experiments (e.g., (Wohlin et al. 2024; Shull, Singer, and Sjøberg 2008)), their replications (e.g., (Santos et al. 2021)), or empirical studies in general (e.g., the ACM SIGSOFT Empirical Standards(Ralph et al. 2021)), none of these address the LLM-specific aspects described above. Previously, a position paper highlighted these issues (Wagner et al. 2025), but there was no comprehensive community-developed guidance for designing and reporting empirical studies involving LLMs in SE.
Therefore, we present community-developed guidelines for conducting and reporting studies involving LLMs in SE research, co-developed by 22 researchers. After outlining our Scope, we introduce a taxonomy of Study Types, then present eight Guidelines. We complement these with an applicability matrix mapping guidelines to study types and a reporting checklist for authors and reviewers. For each study type and guideline, we identify relevant examples, both within and outside of SE research. We maintain the guidelines online as a living resource for the community to use and shape (llm-guidelines.org).
Scope of Guidelines Changed
2026.03
Scope of Guidelines
Software Engineering as our Target Discipline
We target SE research because existing cross-disciplinary guidelines do not address its needs. For instance, Gallifant et al.’s (Gallifant et al. 2025) guidelines for LLM use in healthcare research include discipline-specific items irrelevant to most SE studies (e.g., clinical-care usability) while lacking guidance on tool architectures, benchmarking, and code-specific evaluation that SE research requires. Moreover, SE research employs a wide variety of empirical methods (Ralph et al. 2021; Wohlin et al. 2024). Our work builds on Sallou, Durieux, and Panichella (2024)’s vision paper on mitigating validity threats in LLM-based SE research (Sallou, Durieux, and Panichella 2024) and our own position paper (Wagner et al. 2025), providing more detailed advice for a wider variety of research methods organized into a taxonomy of Study Types (Section 4).
… Unchanged text
Focus on Text-Based Use Cases
While multi-modal foundation models that use or generate images, audio, or video may also support SE research and practice, we focus on textual use cases of LLMs (e.g., in natural language or programming languages). Many of our guidelines—particularly those concerning model reporting, prompt documentation, and reproducibility—are likely applicable to multi-modal settings as well, but we leave their explicit validation to future work.
Focus on Direct Development or Research Support
While researchers may use LLMs for many peripheral tasks (e.g., proof-reading, spell-checking, translation), our guidelines focus on their direct role in empirical research and engineering practice. For engineers, we focus on the use of LLMs to automate SE tasks, that is, artificial intelligence (AI) for software engineering (AI4SE) (see LLMs for SE ). This includes agentic systems that autonomously plan and execute multi-step tasks using LLMs (see ArchitectureG3: Architecture ). For researchers, we focus on the use of LLMs to automate empirical research tasks such as data collection, processing, or analysis (see LLMs for Research ).
Researchers as our Target Audience
Our guidelines are intended to help SE researchers design, plan, conduct, and report empirical studies involving LLMs, and to support scholarly peer review of such studies. Each guideline includes an Advice for Reviewers subsection with targeted assessment suggestions. Our guidelines focus on what to report and how; they complement but do not replace methodological guidance for designing specific types of empirical studies.
… Unchanged text
How to Navigate this Paper
This paper is structured to support different reading strategies depending on the reader’s goal. Researchers planning a new study may start with the taxonomy of study types (see Study Types ) to identify which types apply to their planned work, then consult Table 3 to determine which guidelines are requirements (MUST ) and which are recommendations (SHOULD ) for those study types. Each guideline section opens with a tl;dr summary in a shaded box, allowing readers to quickly assess relevance before reading the full text. Researchers writing up results may prefer to start with the checklist in Appendix 7, which organizes actionable items by typical paper sections (Introduction, Research Design and Methods, Results, etc.). Table 2 provides a quick reference to all eight guidelines, and Table 4 maps each guideline’s rationale to its key recommendations. Reviewers can use the Advice for Reviewers subsection at the end of each guideline for targeted guidance on assessing manuscripts. Table 3 helps reviewers identify which guidelines apply to the study type under review.
2026.04
Scope of Guidelines
Software Engineering as our Target Discipline
We target SE research because existing cross-disciplinary guidelines do not address its needs. For instance, Gallifant et al.’s (Gallifant et al. 2025) guidelines for LLM use in healthcare research include discipline-specific items irrelevant to most SE studies (e.g., clinical-care usability) while lacking guidance on tool architectures, benchmarking, and code-specific evaluation that SE research requires. Moreover, SE research employs a wide variety of empirical methods (Ralph et al. 2021; Wohlin et al. 2024). Our work builds on Sallou, Durieux, and Panichella (2024)’s vision paper on mitigating validity threats in LLM-based SE research (Sallou, Durieux, and Panichella 2024) and our own position paper (Wagner et al. 2025), providing more detailed advice for a wider variety of research methods organized into a taxonomy of Study Types (Section 4).
… Unchanged text
Focus on Text-Based Use Cases
While multi-modal foundation models that use or generate images, audio, or video may also support SE research and practice, we focus on textual use cases of LLMs (e.g., in natural language or programming languages). Many of our guidelines—particularly those concerning model reporting, prompt documentation, and reproducibility—are likely applicable to multi-modal settings as well, but we leave their explicit validation to future work.
Focus on Direct Development or Research Support
While researchers may use LLMs for many peripheral tasks (e.g., proof-reading, spell-checking, translation), our guidelines focus on their direct role in empirical research and engineering practice. For engineers, we focus on the use of LLMs to automate SE tasks, that is, artificial intelligence (AI) for software engineering (AI4SE) (see LLMs for SE ). This includes agentic systems that autonomously plan and execute multi-step tasks using LLMs (see System and Prompt DesignG3: System and Prompt Design ). For researchers, we focus on the use of LLMs to automate empirical research tasks such as data collection, processing, or analysis (see LLMs for Research ).
Researchers as our Target Audience
Our guidelines are intended to help SE researchers design, plan, conduct, and report empirical studies involving LLMs, and to support scholarly peer review of such studies. Each guideline includes an Advice for Reviewers subsection with targeted assessment suggestions. Our guidelines focus on what to report and how; they complement but do not replace methodological guidance for designing specific types of empirical studies.
Reporting Locations
We use PAPER to refer to the manuscript PDF, including any appendices bound with it. We use SUPPLEMENTARY MATERIAL to refer to any artifact, replication package, dataset, or other resource external to the manuscript PDF (e.g., hosted on Zenodo or Figshare). What belongs in the main body versus an appendix is a presentation choice for authors and venues; what belongs inside the manuscript versus outside it is the reporting-location decision the guidelines make. When a recommendation does not specify a location, either PAPER or SUPPLEMENTARY MATERIAL is acceptable.
… Unchanged text
How to Navigate this Paper
This paper is structured to support different reading strategies depending on the reader’s goal. Researchers planning a new study may start with the taxonomy of study types (see Study Types ) to identify which types apply to their planned work, then consult Table 3 to determine which guidelines are requirements (MUST ) and which are recommendations (SHOULD ) for those study types. Each guideline section opens with a tl;dr summary in a shaded box, allowing readers to quickly assess relevance before reading the full text. Researchers writing up results may prefer to start with the checklist in Appendix 7, which organizes actionable items by typical paper sections (Introduction, Research Design and Methods, Results, etc.). Table 2 provides a quick reference to all eight guidelines, and Table 4 maps each guideline’s rationale to its key recommendations. Reviewers can use the Advice for Reviewers subsection at the end of each guideline for targeted guidance on assessing manuscripts. Table 3 helps reviewers identify which guidelines apply to the study type under review.
Methodology Changed
2026.03
Methodology
This project was initiated at the 2024 meeting of the International Software Engineering Research Network (ISERN) in Barcelona, Spain, where the first and last author organized a session on guidelines for empirical studies involving LLMs. These discussions led to a position paper at WSESE 2025(Wagner et al. 2025), which was presented at the Copenhagen Symposium on Human-Centered Software Engineering AI (CHASEAI 2024), forming a workstream to collaboratively refine and extend the preliminary guidelines. Our process follows an established tradition of developing reporting guidelines through expert collaboration, as exemplified by CONSORT for clinical trials (Begg et al. 1996) and, more recently, TRIPOD-LLM for LLM-based prediction studies in medicine (Gallifant et al. 2025).
Then, at least two co-authors were assigned to each study type and guideline to refine and extend the content, followed by a peer review and refinement phase. Afterwards, the first author reviewed all sections and discussed potential changes with the topic leads. A requirement for co-authorship was that each author contributed or reviewed content and, most importantly, read the complete guidelines and confirmed that they stand behind all recommendations, not only their own contributions.
… Unchanged text
To select illustrative examples for each study type and guideline, co-author pairs independently searched for relevant papers using academic databases. This initial pool was extended based on suggestions from other co-authors. Identified papers were assessed by one author and subsequently reviewed by a second author; papers that did not add new insights beyond those already covered were not included. Multiple review rounds were then carried out on the guidelines as a whole, during which selected papers were cross-checked by additional authors and, where necessary, further examples were added or replaced. The examples are intended to be illustrative, not the result of a systematic review; the involvement of multiple authors at different stages helped mitigate individual selection bias.
After completing the internal review, we invited external experts in empirical SE to review the study types and guidelines. During the review process of the Empirical Software Engineering journal, the structure and content were improved based on reviewers’ suggestions. We self-assessed our guidelines against the quality dimensions of the SIGSOFT Empirical Standards(Ralph et al. 2021); this assessment is reported in the conclusion.
2026.04
Methodology
This project was initiated at the 2024 meeting of the International Software Engineering Research Network (ISERN) in Barcelona, Spain, where the first and last author organized a session on guidelines for empirical studies involving LLMs. These discussions led to a position paper at WSESE 2025(Wagner et al. 2025), which was presented at the Copenhagen Symposium on Human-Centered Software Engineering AI (CHASEAI 2024), forming a workstream to collaboratively refine and extend the preliminary guidelines. Our process follows an established tradition of developing reporting guidelines through expert collaboration, as exemplified by CONSORT for clinical trials (Begg et al. 1996) and, more recently, TRIPOD-LLM for LLM-based prediction studies in medicine (Gallifant et al. 2025).
Then, at least two co-authors were assigned to each study type and guideline to refine and extend the content, followed by a peer review and refinement phase. Afterwards, the first author reviewed all sections and discussed potential changes with the topic leads. A requirement for co-authorship was that each author contributed or reviewed content and, most importantly, read the complete guidelines and confirmed that they stand behind all recommendations, not only their own contributions.
… Unchanged text
To select illustrative examples for each study type and guideline, co-author pairs independently searched for relevant papers using academic databases. This initial pool was extended based on suggestions from other co-authors. Identified papers were assessed by one author and subsequently reviewed by a second author; papers that did not add new insights beyond those already covered were not included. Multiple review rounds were then carried out on the guidelines as a whole, during which selected papers were cross-checked by additional authors and, where necessary, further examples were added or replaced. The examples are intended to be illustrative, not the result of a systematic review; the involvement of multiple authors at different stages helped mitigate individual selection bias.
After completing the internal review, we invited external experts in empirical SE to review the study types and guidelines. During the review process of the Empirical Software Engineering journal, the structure and content were improved based on reviewers’ suggestions. We self-assessed our guidelines against the quality dimensions of the SIGSOFT Empirical Standards(Ralph et al. 2021); this assessment is reported in the conclusion.
Advantages and Challenges Changed
2026.03
Advantages and Challenges
Using LLMs in empirical SE research, whether as tools for researchers or for software engineers, offers several potential advantages but also raises fundamental challenges that cut across the study types described above.
… Unchanged text
Advantages
The primary advantage of using LLMs for research tasks is speed, cost reduction, and scalability. LLMs can annotate, judge, synthesize, and simulate faster and at lower cost than human researchers, with studies showing cost reductions of 50–96% on various natural language tasks (S. Wang et al. 2021) (e.g., Z. He et al. (2024) found that GPT-4 annotation required only two days and 122.08 USD compared to several weeks and 4,508 USD for a comparable MTurk pipeline (Z. He et al. 2024)). Similarly, augmenting or replacing human participants with LLM-generated virtual participants would reduce recruitment effort (Madampe et al. 2024). This efficiency unlocks scalability: qualitative research traditionally does not scale well to large samples, but LLMs allow researchers to analyze larger datasets (Bano et al. 2024; Barros et al. 2025; Morais Leça et al. 2025) and support larger judgment datasets.
LLMs can also automate tasks such as coding qualitative data, assessing artifact quality, and generating summaries, reducing cognitive demands and resources required for qualitative research. Some studies suggest that LLMs may improve consistency: ChatGPT’s accuracy exceeded crowd workers by approximately 25%, and LLMs can achieve higher inter-rater agreement than crowd workers and trained annotators (Gilardi, Alizadeh, and Kubli 2023). However, no compelling evidence supports claims that LLMs are less biased than human annotators (Gilardi, Alizadeh, and Kubli 2023).
LLMs could potentially provide access to otherwise-inaccessible research contexts. If virtual participants’ behavior is sufficiently similar to human participants, LLMs could access underrepresented and hard-to-reach populations, strengthen generalizability and inclusiveness, impute missing data (see LLMs for SynthesisS3: LLMs for Synthesis ), and enable research that is ethically problematic with real humans (e.g., questions that would force a human to relive past trauma do not harm an LLM).
Studying real-world LLM-based tools allows researchers to understand the state of practice, uncovering usage patterns, adoption rates, and contextual factors, and generate hypotheses about how LLMs affect developer productivity, collaboration, and decision-making. From an engineering perspective, developing LLM-based tools requires less task-specific engineering than traditional SE approaches such as static analysis or symbolic execution, because a single model can handle diverse inputs without language-specific parsers or hand-crafted rules. Good benchmarks provide standardized evaluation and model comparison, reducing research effort and fostering open science by providing common ground for sharing data and results. Benchmarks built for specific SE tasks can help identify LLM weaknesses and support optimization and fine-tuning.
Challenges
The stochastic nature of LLM responses, where identical prompts may yield different outputs, undermines replicability across all study types, complicating interpretation of experimental results and test-retest reliability. More broadly, reliability is a persistent concern: conceptualizing an LLM-as-judge or LLM-as-annotator as a measurement instrument, researchers should expect validity, test-retest reliability, inter-rater reliability, minimal error, and measurement invariance (Ralph et al. 2024). However, LLMs show significant variability depending on the dataset and task (Pangakis, Wolken, and Fasching 2023), are sensitive to prompt variations (Reiss 2023) and option order (Pezeshkpour and Hruschka 2024), can behave differently when reviewing their own outputs (Panickssery, Bowman, and Feng 2024), and can be unreliable for high-stakes labeling (X. Wang et al. 2024). Crucially, reliability does not imply validity: a reliable LLM might be reliably inaccurate (R. Zhou, Chen, and Yu 2024), and for tasks with no single correct answer, the statistical framework pushes outputs toward the most likely answer, which may not be the best one (Bavaresco et al. 2024). See Version and ConfigurationG2: Version and Configuration and Limitations and MitigationsG8: Limitations and Mitigations for reporting guidance on replicability and reliability.
The rapid evolution of LLMs and LLM-based tools, combined with professionals rapidly learning how to employ them, complicates longitudinal comparisons and may quickly render study findings obsolete (e.g., GPT-4’s accuracy on a simple mathematical task dropped from 84% to 51% within three months (L. Chen, Zaharia, and Zou 2024)).
The prevalence of proprietary tools and opaque training data limits researchers’ ability to assess and mitigate biases, and enables benchmark contamination (Ahuja, Gumma, and Sitaram 2024). LLMs exhibit bias in multiple forms: tendencies to overestimate certain labels (Zhu et al. 2023), well-documented fairness issues (Gallegos et al. 2024), and the potential to reinforce prejudices. When simulating human participants, LLMs are “likely to both misportray and flatten the representations of demographic groups” (A. Wang, Morgenstern, and Dickerson 2025). See Open LLMsG6: Open LLMs and Limitations and MitigationsG8: Limitations and Mitigations for mitigation strategies.
Evaluation difficulty is inherent in studying LLM-based tools and benchmarks. Benchmarks may lack construct validity (Ralph et al. 2024), usually do not capture the full complexity of software engineering work (Chandra 2025), and may lead to overconfidence and overfitting (Wan et al. 2024). LLMs are also susceptible to adversarial manipulation where semantics-preserving changes may deceive them into accepting flawed artifacts (J. He et al. 2025). See Benchmarks and MetricsG7: Benchmarks and Metrics for detailed guidance on benchmark design, including Cao et al. (2025)’s (Cao et al. 2025) guidelines for coding task benchmarks.
A fundamental challenge is philosophical and methodological incongruence with qualitative research. In a recent open letter, 419 experienced qualitative researchers argued “that analytic approaches such as reflexive thematic analysis are human research practices requiring a subjective, positioned, and reflexive researcher and therefore the use of GenAI in such approaches is not methodologically congruent” (Jowsey et al. 2025). LLMs lack the capacity for genuinely reflexive qualitative analysis because they operate on statistical prediction without understanding the meaning of the data being analyzed (Jowsey et al. 2025; Bano, Zowghi, and Whittle 2023; Montes et al. 2025). Effective use requires structured prompts and careful human oversight (Barros et al. 2025); see Human ValidationG5: Human Validation and Limitations and MitigationsG8: Limitations and Mitigations for guidance.
There is insufficient evidence for the effectiveness of LLMs in most research roles. No compelling evidence exists that LLMs can accurately simulate human participants (Schröder et al. 2025) or reliably judge most relevant properties of SE artifacts; gathering such evidence for each specific usage may be quite difficult (Harding et al. 2024). LLMs may be better suited for augmenting rather than replacing human researchers (S. Wang et al. 2021; Z. He et al. 2024), but even then, limited evidence exists that augmentation increases effectiveness as well as efficiency(Schröder et al. 2025). When LLM outputs are incorrect, they can negatively affect human judgment(F. Huang, Kwak, and An 2023; Pan et al. 2024); see Human ValidationG5: Human Validation for mitigation strategies.
Pooling multiple outputs for majority voting improves reliability but increases cost and environmental impact(Reiss 2023; S. Wang et al. 2021). While open models are available, the most capable ones require substantial hardware; relying on cloud-based APIs introduces concerns related to data privacy, security, and replicability. See Limitations and MitigationsG8: Limitations and Mitigations for sustainability considerations.
Field study findings face generalizability challenges because outcomes may be highly sensitive to individual differences, usage patterns, goals, and contexts. Field studies must be “dependable” (Sullivan and Sargeant 2011) beyond traditional validity criteria, which complicates methodology; see Limitations and MitigationsG8: Limitations and Mitigations for a detailed threat taxonomy.
2026.04
Advantages and Challenges
Using LLMs in empirical SE research, whether as tools for researchers or for software engineers, offers several potential advantages but also raises fundamental challenges that cut across the study types described above.
… Unchanged text
Advantages
The primary advantage of using LLMs for research tasks is speed, cost reduction, and scalability. LLMs can annotate, judge, synthesize, and simulate faster and at lower cost than human researchers, with studies showing cost reductions of 50–96% on various natural language tasks (S. Wang et al. 2021) (e.g., Z. He et al. (2024) found that GPT-4 annotation required only two days and 122.08 USD compared to several weeks and 4,508 USD for a comparable MTurk pipeline (Z. He et al. 2024)). Similarly, augmenting or replacing human participants with LLM-generated virtual participants would reduce recruitment effort (Madampe et al. 2024). This efficiency unlocks scalability: qualitative research traditionally does not scale well to large samples, but LLMs allow researchers to analyze larger datasets (Bano et al. 2024; Barros et al. 2025; Morais Leça et al. 2025) and support larger judgment datasets.
LLMs can also automate tasks such as coding qualitative data, assessing artifact quality, and generating summaries, reducing cognitive demands and resources required for qualitative research. Some studies suggest that LLMs may improve consistency: ChatGPT’s accuracy exceeded crowd workers by approximately 25%, and LLMs can achieve higher inter-rater agreement than crowd workers and trained annotators (Gilardi, Alizadeh, and Kubli 2023). However, no compelling evidence supports claims that LLMs are less biased than human annotators (Gilardi, Alizadeh, and Kubli 2023).
LLMs could potentially provide access to otherwise-inaccessible research contexts. If virtual participants’ behavior is sufficiently similar to human participants, LLMs could access underrepresented and hard-to-reach populations, strengthen generalizability and inclusiveness, impute missing data (see LLMs for SynthesisS3: LLMs for Synthesis ), and enable research that is ethically problematic with real humans (e.g., questions that would force a human to relive past trauma do not harm an LLM).
Studying real-world LLM-based tools allows researchers to understand the state of practice, uncovering usage patterns, adoption rates, and contextual factors, and generate hypotheses about how LLMs affect developer productivity, collaboration, and decision-making. From an engineering perspective, developing LLM-based tools requires less task-specific engineering than traditional SE approaches such as static analysis or symbolic execution, because a single model can handle diverse inputs without language-specific parsers or hand-crafted rules. Good benchmarks provide standardized evaluation and model comparison, reducing research effort and fostering open science by providing common ground for sharing data and results. Benchmarks built for specific SE tasks can help identify LLM weaknesses and support optimization and fine-tuning.
Challenges
The stochastic nature of LLM responses, where identical prompts may yield different outputs, undermines replicability across all study types, complicating interpretation of experimental results and test-retest reliability. More broadly, reliability is a persistent concern: conceptualizing an LLM-as-judge or LLM-as-annotator as a measurement instrument, researchers should expect validity, test-retest reliability, inter-rater reliability, minimal error, and measurement invariance (Ralph et al. 2024). However, LLMs show significant variability depending on the dataset and task (Pangakis, Wolken, and Fasching 2023), are sensitive to prompt variations (Reiss 2023) and option order (Pezeshkpour and Hruschka 2024), can behave differently when reviewing their own outputs (Panickssery, Bowman, and Feng 2024), and can be unreliable for high-stakes labeling (X. Wang et al. 2024). Crucially, reliability does not imply validity: a reliable LLM might be reliably inaccurate (R. Zhou, Chen, and Yu 2024), and for tasks with no single correct answer, the statistical framework pushes outputs toward the most likely answer, which may not be the best one (Bavaresco et al. 2024). See Version and ConfigurationG2: Version and Configuration and Limitations and MitigationsG8: Limitations and Mitigations for reporting guidance on replicability and reliability.
The rapid evolution of LLMs and LLM-based tools, combined with professionals rapidly learning how to employ them, complicates longitudinal comparisons and may quickly render study findings obsolete (e.g., GPT-4’s accuracy on a simple mathematical task dropped from 84% to 51% within three months (L. Chen, Zaharia, and Zou 2024)).
The prevalence of proprietary tools and opaque training data limits researchers’ ability to assess and mitigate biases, and enables benchmark contamination (Ahuja, Gumma, and Sitaram 2024). LLMs exhibit bias in multiple forms: tendencies to overestimate certain labels (Zhu et al. 2023), well-documented fairness issues (Gallegos et al. 2024), and the potential to reinforce prejudices. When simulating human participants, LLMs are “likely to both misportray and flatten the representations of demographic groups” (A. Wang, Morgenstern, and Dickerson 2025). See Open LLMsG6: Open LLMs and Limitations and MitigationsG8: Limitations and Mitigations for mitigation strategies.
Evaluation difficulty is inherent in studying LLM-based tools and benchmarks. Benchmarks may lack construct validity (Ralph et al. 2024), usually do not capture the full complexity of software engineering work (Chandra 2025), and may lead to overconfidence and overfitting (Wan et al. 2024). LLMs are also susceptible to adversarial manipulation where semantics-preserving changes may deceive them into accepting flawed artifacts (J. He et al. 2025). See Benchmarks and MetricsG5: Benchmarks and Metrics for detailed guidance on benchmark design, including Cao et al. (2025)’s (Cao et al. 2025) guidelines for coding task benchmarks.
A fundamental challenge is philosophical and methodological incongruence with qualitative research. In a recent open letter, 419 experienced qualitative researchers argued “that analytic approaches such as reflexive thematic analysis are human research practices requiring a subjective, positioned, and reflexive researcher and therefore the use of GenAI in such approaches is not methodologically congruent” (Jowsey et al. 2025). LLMs lack the capacity for genuinely reflexive qualitative analysis because they operate on statistical prediction without understanding the meaning of the data being analyzed (Jowsey et al. 2025; Bano, Zowghi, and Whittle 2023; Montes et al. 2025). Effective use requires structured prompts and careful human oversight (Barros et al. 2025); see Human ValidationG7: Human Validation and Limitations and MitigationsG8: Limitations and Mitigations for guidance.
There is insufficient evidence for the effectiveness of LLMs in most research roles. No compelling evidence exists that LLMs can accurately simulate human participants (Schröder et al. 2025) or reliably judge most relevant properties of SE artifacts; gathering such evidence for each specific usage may be quite difficult (Harding et al. 2024). LLMs may be better suited for augmenting rather than replacing human researchers (S. Wang et al. 2021; Z. He et al. 2024), but even then, limited evidence exists that augmentation increases effectiveness as well as efficiency(Schröder et al. 2025). When LLM outputs are incorrect, they can negatively affect human judgment(F. Huang, Kwak, and An 2023; Pan et al. 2024); see Human ValidationG7: Human Validation for mitigation strategies.
Pooling multiple outputs for majority voting improves reliability but increases cost and environmental impact(Reiss 2023; S. Wang et al. 2021). While open models are available, the most capable ones require substantial hardware; relying on cloud-based APIs introduces concerns related to data privacy, security, and replicability. See Limitations and MitigationsG8: Limitations and Mitigations for sustainability considerations.
Field study findings face generalizability challenges because outcomes may be highly sensitive to individual differences, usage patterns, goals, and contexts. Field studies must be “dependable” (Sullivan and Sargeant 2011) beyond traditional validity criteria, which complicates methodology; see Limitations and MitigationsG8: Limitations and Mitigations for a detailed threat taxonomy.
Guidelines Changed
2026.03
Guidelines
A primary goal of our guidelines is to enable reproducibility and replicability of empirical SE studies involving LLMs. As repeating LLM-focused research to verify results lies somewhere between the ACM’s definitions of reproducibility (different team, same research artifacts) and repeatability (different team, different research artifacts) (ACM Publications Board 2020) due to potential model changes and imperfect research artifacts, we follow Angermeir et al. (2025)(Angermeir et al. 2025) and use the terms interchangeably. While previous guidelines regarding open science and empirical studies still apply, LLM-specific characteristics (e.g. inherent non-determinism (Song et al. 2025; Yuan et al. 2025), opaque and proprietary models) present additional replicability challenges, which, in turn, demand new guidance.
… Unchanged text
Each guideline below begins with a brief tl;dr summary, followed by its rationale, recommendations, examples, benefits, challenges, links to the relevant study types. The rationale articulates the underlying principle, i.e., why the guideline matters, while the recommendations provide concrete, actionable practices.
The guidelines further contain advice for reviewers. Broadly, reviewers should use our guidelines to help them interrogate the extent to which a manuscript’s authors have done what was practically possible to improve reproducibility. We must neither accept research absent reasonable efforts to improve reproducibility, nor reject research for failing to obtain an impossible goal. We borrow this principle of “reasonable efforts” from the SIGSOFT Empirical Standards (Ralph et al. 2021), where it applies to methodological rigor more generally. Of course, papers should acknowledge their limitations, but to determine whether these limitations are reasonable, reviewers should ask “have the authors done what they could to minimize limitations?” Our guidelines attempt to capture what “reasonable efforts” practically means for LLM-based studies.
To distinguish essential criteria from recommendations, our guidelines use two tiers. A MUST criterion is a requirement. Studies that intend to follow these guidelines are expected to meet all MUST criteria. A SHOULD criterion represents a desired practice that strengthens a study’s rigor or transparency. However, there may be valid reasons to deviate in particular circumstances (e.g., resource constraints, inapplicability to a specific study context or type). Nonetheless, studies that deviate from a SHOULD criterion should briefly justify the deviation and discuss its potential impact (e.g., on validity or reproducibility).
The following sections indicate which information we expect researchers to report, and whether it should be in the PAPER or SUPPLEMENTARY MATERIAL. Where a publication venue’s page limits hinder reporting all expected elements in the PAPER, it is better to report essential information in the SUPPLEMENTARY MATERIAL than not at all. The SUPPLEMENTARY MATERIAL should be published according to the ACM SIGSOFT Open Science Policies(Graziotin 2024).
A primary goal of our guidelines is to enable reproducibility and replicability of empirical SE studies involving LLMs. As repeating LLM-focused research to verify results lies somewhere between the ACM’s definitions of reproducibility (different team, same research artifacts) and repeatability (different team, different research artifacts) (ACM Publications Board 2020) due to potential model changes and imperfect research artifacts, we follow Angermeir et al. (2025)(Angermeir et al. 2025) and use the terms interchangeably. While previous guidelines regarding open science and empirical studies still apply, LLM-specific characteristics (e.g. inherent non-determinism (Song et al. 2025; Yuan et al. 2025), opaque and proprietary models) present additional replicability challenges, which, in turn, demand new guidance.
… Unchanged text
Each guideline below begins with a brief tl;dr summary, followed by its rationale, recommendations, examples, benefits, challenges, links to the relevant study types. The rationale articulates the underlying principle, i.e., why the guideline matters, while the recommendations provide concrete, actionable practices.
The guidelines further contain advice for reviewers. Broadly, reviewers should use our guidelines to help them interrogate the extent to which a manuscript’s authors have done what was practically possible to improve reproducibility. We must neither accept research absent reasonable efforts to improve reproducibility, nor reject research for failing to obtain an impossible goal. We borrow this principle of “reasonable efforts” from the SIGSOFT Empirical Standards (Ralph et al. 2021), where it applies to methodological rigor more generally. Of course, papers should acknowledge their limitations, but to determine whether these limitations are reasonable, reviewers should ask “have the authors done what they could to minimize limitations?” Our guidelines attempt to capture what “reasonable efforts” practically means for LLM-based studies.
To distinguish essential criteria from recommendations, our guidelines use two tiers. A MUST criterion is a requirement. Studies that intend to follow these guidelines are expected to meet all MUST criteria. A SHOULD criterion represents a desired practice that strengthens a study’s rigor or transparency. However, there may be valid reasons to deviate in particular circumstances (e.g., resource constraints, inapplicability to a specific study context or type). Nonetheless, studies that deviate from a SHOULD criterion should briefly justify the deviation and discuss its potential impact (e.g., on validity or reproducibility).
The following sections indicate which information we expect researchers to report, and whether it should be in the PAPER or SUPPLEMENTARY MATERIAL. Where a publication venue’s page limits hinder reporting all expected elements in the PAPER, it is better to report essential information in the SUPPLEMENTARY MATERIAL than not at all. The SUPPLEMENTARY MATERIAL should be published according to the ACM SIGSOFT Open Science Policies(Graziotin 2024).
tl;dr: Researchers MUST disclose any use of LLMs to support empirical studies in their PAPER, specifying which LLM was used, how it was used, and where in the research process it was employed. They SHOULD report the exact purpose, the tasks that were automated, and the expected benefits in the PAPER. When the LLM plays a central role, the declaration SHOULD be prominent and detailed in the methodology section; for tangential uses, a brief statement in the methodology or acknowledgments suffices.
… Unchanged text
Rationale
Transparency about LLM involvement is a prerequisite for informed assessment of a study’s scope, limitations, and potential biases. Without explicit disclosure, readers cannot evaluate how the LLM’s characteristics may have influenced the research process or its outcomes.
Recommendations
When conducting any kind of empirical study involving LLMs, researchers MUST clearly declare that an LLM was used (see Scope for what we consider relevant research support). This SHOULD be done in a suitable section of the PAPER, for example, in the introduction or research methods section. For authoring scientific articles, this transparency is, for example, required by the ACM Policy on Authorship: “The use of generative AI tools and technologies to create content is permitted but must be fully disclosed in the Work” (Association for Computing Machinery 2023).
Beyond generic declarations, researchers SHOULD report the exact purpose of using an LLM in a study, the tasks it was used to automate, and the expected benefits in the PAPER. A sufficient declaration specifies not only that an LLM was used, but also which LLM (name and version), how it was used (e.g., as an annotator, code generator, or judge), and where in the research process it was employed (e.g., data collection, analysis, or synthesis).
When the LLM plays a central role in the study (e.g., as the main tool being evaluated or as a core component of the research method), the declaration SHOULD be prominent and detailed, appearing in the methodology section with cross-references to the specific guidelines that apply (e.g., Sections Version and ConfigurationG2: Version and Configuration , ArchitectureG3: Architecture , and PromptsandLogsG4: PromptsandLogs ). When the LLM’s role is more tangential (e.g., used for a single preprocessing step), a brief but explicit statement in the methodology or acknowledgments section is sufficient. In either case, the disclosure MUST be specific enough for readers to assess how the LLM’s involvement may affect the study’s validity and reproducibility.
Example(s)
The ACM Policy on Authorship(Association for Computing Machinery 2023) suggests disclosing GenAI usage in the acknowledgments section of the PAPER, advising to “err on the side of caution, and include a disclosure in the acknowledgments section of the Work” when uncertain about the need (Association for Computing Machinery 2023). For double-blind review, researchers can add a temporary “AI Disclosure” section where the acknowledgments would appear. An example of an LLM disclosure beyond writing support can be found in a recent paper by Lubos et al. (2024)(Lubos et al. 2024), in which they write in the methodology section:
“We conducted an LLM-based evaluation of requirements utilizing the Llama 2 language model with 70 billion parameters, fine-tuned to complete chat responses...”
… Unchanged text
Benefits
Transparency in the use of LLMs helps other researchers understand the context and scope of the study, facilitating better interpretation and comparison of the results. Beyond this declaration, we recommend researchers to be explicit about the LLM version they used (see Version and ConfigurationG2: Version and Configuration ) and the LLM’s exact role.
Challenges
Declaring LLM usage requires only a brief statement and no additional experiments, making compliance straightforward. One challenge might be authors’ reluctance to disclose LLM usage for valid use cases, because they fear that AI-generated content makes reviewers think that the authors’ work is less original. In fact, there is evidence suggesting that AI disclosure can negatively affect trust in authors (Schilke and Reimann 2025). However, the ACM Policy on Authorship is very clear in that any use of GenAI tools to create content MUST be disclosed. Our guidelines focus on research support beyond proof-reading and writing support (see Scope ), but the threshold of what must be declared continues to evolve as organizations such as the ACM update their authorship policies.
The most common problem with disclosure is incompleteness or vagueness about how the LLM was used. If the paper says “we used LLM X to help with task Y” without specifying how, reviewers should request clarification. Such requests are typically minor revisions unless the missing details may reveal methodological problems.
Using an LLM as a copyeditor becomes problematic when authors do not or cannot take responsibility for the resulting text. If a reviewer finds clear evidence that LLM-generated text was not carefully reviewed (e.g., text like “As a Large Language Model, I...”), this may warrant rejection. If a reviewer suspects an author cannot take responsibility for AI-generated text, they should raise the concern with their editor or program chair. Due process requires that authors not be accused of misconduct without clear evidence.
If undisclosed LLM use is suspected, the reviewer should similarly consult their editor or program chair. When the evidence is conclusive, the key question is the degree to which undisclosed use affects the study’s contribution, ranging from negligible (e.g., word choice in a single sentence) to severe (e.g., generating ostensibly empirical data or statistical analyses).
tl;dr: Researchers MUST disclose any use of LLMs to support empirical studies in their PAPER, specifying which LLM was used, how it was used, and where in the research process it was employed. They SHOULD report the exact purpose, the tasks that were automated, and the expected benefits in the PAPER. When the LLM plays a central role, the declaration SHOULD be prominent and detailed in the methodology section; for tangential uses, a brief statement in the methodology or acknowledgments suffices.
… Unchanged text
Rationale
Transparency about LLM involvement is a prerequisite for informed assessment of a study’s scope, limitations, and potential biases. Without explicit disclosure, readers cannot evaluate how the LLM’s characteristics may have influenced the research process or its outcomes.
Recommendations
When conducting any kind of empirical study involving LLMs, researchers MUST clearly declare that an LLM was used (see Scope for what we consider relevant research support). This SHOULD be done in a suitable section of the PAPER, for example, in the introduction or research methods section. For authoring scientific articles, this transparency is, for example, required by the ACM Policy on Authorship: “The use of generative AI tools and technologies to create content is permitted but must be fully disclosed in the Work” (Association for Computing Machinery 2023).
Beyond generic declarations, researchers SHOULD report the exact purpose of using an LLM in a study, the tasks it was used to automate, and the expected benefits in the PAPER. A sufficient declaration specifies not only that an LLM was used, but also which LLM (name and version), how it was used (e.g., as an annotator, code generator, or judge), and where in the research process it was employed (e.g., data collection, analysis, or synthesis).
When the LLM plays a central role in the study (e.g., as the main tool being evaluated or as a core component of the research method), the declaration SHOULD be prominent and detailed, appearing in the methodology section with cross-references to the specific guidelines that apply (e.g., Sections Version and ConfigurationG2: Version and Configuration , System and Prompt DesignG3: System and Prompt Design , and Session TracesG4: Session Traces ). When the LLM’s role is more tangential (e.g., used for a single preprocessing step), a brief but explicit statement in the methodology or acknowledgments section is sufficient. In either case, the disclosure MUST be specific enough for readers to assess how the LLM’s involvement may affect the study’s validity and reproducibility.
Example(s)
The ACM Policy on Authorship(Association for Computing Machinery 2023) suggests disclosing GenAI usage in the acknowledgments section of the PAPER, advising to “err on the side of caution, and include a disclosure in the acknowledgments section of the Work” when uncertain about the need (Association for Computing Machinery 2023). For double-blind review, researchers can add a temporary “AI Disclosure” section where the acknowledgments would appear. An example of an LLM disclosure beyond writing support can be found in a recent paper by Lubos et al. (2024)(Lubos et al. 2024), in which they write in the methodology section:
“We conducted an LLM-based evaluation of requirements utilizing the Llama 2 language model with 70 billion parameters, fine-tuned to complete chat responses...”
A more contemporary declaration could similarly state:
“We used Claude Opus 4.7 via the Anthropic API to synthesize themes from interview transcripts, with all prompts and conversation logs published as SUPPLEMENTARY MATERIAL.”
… Unchanged text
Benefits
Transparency in the use of LLMs helps other researchers understand the context and scope of the study, facilitating better interpretation and comparison of the results. Beyond this declaration, we recommend researchers to be explicit about the LLM version they used (see Version and ConfigurationG2: Version and Configuration ) and the LLM’s exact role.
Challenges
Declaring LLM usage requires only a brief statement and no additional experiments, making compliance straightforward. One challenge might be authors’ reluctance to disclose LLM usage for valid use cases, because they fear that AI-generated content makes reviewers think that the authors’ work is less original. In fact, there is evidence suggesting that AI disclosure can negatively affect trust in authors (Schilke and Reimann 2025). However, the ACM Policy on Authorship is very clear in that any use of GenAI tools to create content MUST be disclosed. Our guidelines focus on research support beyond proof-reading and writing support (see Scope ), but the threshold of what must be declared continues to evolve as organizations such as the ACM update their authorship policies.
The most common problem with disclosure is incompleteness or vagueness about how the LLM was used. If the paper says “we used LLM X to help with task Y” without specifying how, reviewers should request clarification. Such requests are typically minor revisions unless the missing details may reveal methodological problems.
Using an LLM as a copyeditor becomes problematic when authors do not or cannot take responsibility for the resulting text. If a reviewer finds clear evidence that LLM-generated text was not carefully reviewed (e.g., text like “As a Large Language Model, I...”), this may warrant rejection. If a reviewer suspects an author cannot take responsibility for AI-generated text, they should raise the concern with their editor or program chair. Due process requires that authors not be accused of misconduct without clear evidence.
If undisclosed LLM use is suspected, the reviewer should similarly consult their editor or program chair. When the evidence is conclusive, the key question is the degree to which undisclosed use affects the study’s contribution, ranging from negligible (e.g., word choice in a single sentence) to severe (e.g., generating ostensibly empirical data or statistical analyses).
Report Model Version, Configuration, and Customizations (G2) Changed
2026.03
Report Model Version, Configuration, and Customizations (G2)
tl;dr: Researchers MUST report the exact LLM model or tool version, configuration, and experiment date in the PAPER. When using quantized models, researchers SHOULD report the quantization level and method. For fine-tuned models, they MUST describe the fine-tuning goal, dataset, and procedure in the PAPER. Researchers SHOULD include default parameters, explain model choices, compare base- and fine-tuned model using suitable metrics and benchmarks, and share fine-tuning data and weights as SUPPLEMENTARY MATERIAL (or alternatively justify in the PAPER why they cannot share them).
Rationale
LLMs and LLM-based tools are frequently updated, and configuration parameters such as temperature or seed values affect content generation. This guideline focuses on documenting the model-specific aspects of empirical studies involving LLMs, concentrating on the models themselves, their version, configuration parameters, and customizations (e.g., fine-tuning). While the ArchitectureG3: Architecture section addresses system-level integration, the information outlined here is always essential for reproducibility whenever an LLM is involved.
Recommendations
Researchers MUST document in the PAPER which model or tool version they used in their study, along with the date when the experiments were carried out and the configured parameters that affect output generation. Since default values might change over time, researchers SHOULDalways report all configuration values, even if they used the defaults. Checksums and fingerprints SHOULD be reported since they identify specific versions and configurations. Depending on the study context, other properties such as the context window size (number of tokens) SHOULD be reported. When using quantized models, researchers SHOULD report the quantization level (e.g., 4-bit, 8-bit) and method (e.g., GPTQ or AWQ), as different quantization approaches produce different outputs, affecting both output quality and reproducibility. Researchers SHOULD motivate in the PAPER why they selected certain models, versions, and configurations. Reasons may be monetary, technical, or methodological (e.g., planned comparison to previous work). Depending on the specific study context, additional information regarding the experiment or tool architecture SHOULD be reported.
A common customization approach for existing LLMs is fine-tuning. If a model was fine-tuned, researchers MUST describe the fine-tuning goal (e.g., improving the performance for a specific task), the fine-tuning procedure (e.g., full fine-tuning vs. Low-Rank Adaptation (LoRA), selected hyperparameters, loss function, learning rate, batch size, etc.), and the fine-tuning dataset (e.g., data sources, the preprocessing pipeline, dataset size) in the PAPER. Researchers SHOULD either share the fine-tuning dataset as part of the SUPPLEMENTARY MATERIAL or explain in the PAPER why the data cannot be shared (e.g., because it contains confidential or personal data that could not be anonymized). The same applies to the fine-tuned model weights. Suitable benchmarks and metrics SHOULD be used to compare the base model with the fine-tuned model.
In summary, our recommendation is to report:
Model/tool name and version (MUST in PAPER );
All relevant configured parameters that affect output generation (MUST in PAPER );
Default values of all available parameters (SHOULD );
Checksum/fingerprint of used model version and configuration (SHOULD );
Additional properties such as context window size (SHOULD );
Model quantization level and method, if applicable (SHOULD ).
For fine-tuned models, additional recommendations apply:
Fine-tuning goal (MUST in PAPER );
Fine-tuning dataset creation and characterization (MUST in PAPER );
Fine-tuning parameters and procedure (MUST in PAPER );
Fine-tuning dataset and fine-tuned model weights (SHOULD );
Validation metrics and benchmarks (SHOULD ).
Commercial models (e.g., GPT-5) or LLM-based tools (e.g., ChatGPT) might not give researchers access to all required information. Oursuggestionisto report what is available and openly acknowledge limitations that hinder reproducibility.
Example(s)
Based on the documentation that OpenAI and Azure provide (OpenAI 2025; Microsoft 2025), researchers might, for example, report:
“We integrated a gpt-4 model in version 0125-Preview via the Azure OpenAI Service, and configured it with a temperature of 0.7, top_p set to 0.8, a maximum token length of 512, and the seed value 23487. We ran our experiment on 10th January 2025. The system fingerprint was fp_6b68a8204b.
Kang, Yoon, and Yoo (2023) provide a similar statement in their paper on exploring LLM-based bug reproduction (Kang, Yoon, and Yoo 2023):
“We access OpenAI Codex via its closed beta API, using the code-davinci-002 model. For Codex, we set the temperature to 0.7, and the maximum number of tokens to 256.”
Our guidelines additionally suggesttoreport a checksum/fingerprint and exact dates,butotherwise this example is close to our recommendations.
Dhar, Vaidhyanathan, and Varma (2024)(Dhar, Vaidhyanathan, and Varma 2024) assessed whether LLMs can generate architectural design decisions, detailing the system architecture and the LLM’s role within it. They provide information on the fine-tuning approach and datasets, including the source of architectural decision records, preprocessing methods, and data selection criteria.
For self-hosted models, the SUPPLEMENTARY MATERIAL can become a true replication package. For example, for models provisioned using ollama, one can report the specific tag and checksum, e.g., “llama3.3, tag 70b-instruct-q8_0, checksum d5b5e1b84868.” Given suitable hardware, running the model is then as easy as executing the following command: ollama run llama3.3:70b-instruct-q8_0
… Unchanged text
Benefits
Reporting this information is a prerequisite for the verification, reproduction, and replication of LLM-based studies. While LLMs are inherently non-deterministic, this cannot excuse dismissing reproducibility. Although exact reproducibility is hard to achieve, reporting the information outlined in this guideline helps researchers come as close as possible to that standard.
Challenges
Different model providers and modes of operating the models allow for varying degrees of information. For example, OpenAI provides a model version and a system fingerprint describing the backend configuration, which can also influence the output. However,infact, the fingerprint is intended only to detect changes in the model or its configuration; one cannot go back to a certain fingerprint. As a beta feature, OpenAI lets users set a seed parameter to receive “(mostly) consistent output” (OpenAI 2023). However, the seed value does not allow for full reproducibility and the fingerprint changes frequently. Although, as motivated above, open models significantly simplify re-running experiments, they also come with challenges in terms of reproducibility, as generated outputs can be inconsistent despite setting the temperature to 0 and using a seed value (see GitHub issue for Llama3). Setting the temperature to 0 configures greedy decoding (always selecting the most probable next token), which minimizes output variability but can degrade quality by producing repetitive text and missing higher-quality responses (Holtzman et al. 2020).
Even with a temperature of 0, full determinism is rarely guaranteed: floating-point arithmetic on GPUs causes slight numerical differences that cascade into divergent token selections (Yuan et al. 2025), Sparse Mixture-of-Experts routing amplifies this effect (Chann 2023), silent backend changes in commercial APIs produce different outputs over time (L. Chen, Zaharia, and Zou 2024), and even self-hosted open models with identical settings do not always yield consistent outputs (Angermeir et al. 2025; Song et al. 2025). Researchers shouldthereforenot treat a temperature of 0 as a guarantee of reproducibility, but as one measure among several, including fixed seed values (OpenAI 2023), system fingerprints, and archiving of raw outputs. When a temperature of 0 is chosen primarily for reproducibility, this motivation SHOULD be stated explicitly, along with an acknowledgment of its potential impact on output quality.
Study Types
This guideline MUST be followed for all study types for which the researcher has access to (parts of) the model’s configuration. They MUST always report the configuration that is visible to them, acknowledging the reproducibility challenges of commercial tools and models that are offered as-a-service. Depending on the specific study type, researchers SHOULD provide additional information on the architectureofatooltheybuilt (see ArchitectureG3: Architecture ), promptsandinteractionslogs (see PromptsandLogsG4: PromptsandLogs ), and specific limitations and mitigations (see Limitations and MitigationsG8: Limitations and Mitigations ).
For example, when Studying LLM UsageS5: Studying LLM Usage by focusing on commercial tools such as ChatGPT or GitHub Copilot, researchers MUST be as specific as possible in describing their study setup. The configured model name, version, and the date when the experiment was conducted MUST always be reported. Inthosecases, reporting otheraspects,suchaspromptsandinteraction logs,isessential.
Missing version, configuration, or parameter information is typically a minor revision request. Before concluding that information is absent, reviewers should check appendices and supplementary materials, as details are sometimes reported there rather than in the main text. Rejection over missing details is rarely warranted unless the omissions obscure deeper methodological problems.
Report Model Version, Configuration, and Customizations (G2)
tl;dr: Researchers MUST report the exact LLM model or tool version, configuration, and experiment date in the PAPER. When using quantized models, researchers SHOULD report the quantization level and method. For fine-tuned models, they MUST describe the fine-tuning goal, dataset, and procedure in the PAPER. Researchers SHOULD include default parameters, explain model choices, compare base- and fine-tuned model using suitable metrics and benchmarks, and share fine-tuning data and weights as SUPPLEMENTARY MATERIAL (or alternatively justify in the PAPER why they cannot share them).
Rationale
LLMs and LLM-based tools are frequently updated, and configuration parameters such as temperature or seed values affect content generation. This guideline focuses on documenting the model-specific aspects of empirical studies involving LLMs, concentrating on the models themselves, their version, configuration parameters, and customizations (e.g., fine-tuning). While the System and Prompt DesignG3: System and Prompt Design section addresses system-level integration and the authored artifacts (including prompts) that shape each call, the information outlined here isessential for reproducibility whenever an LLM is involved.
Recommendations
Researchers MUST document in the PAPER which model or tool version they used in their study, along with the date when the experiments were carried out and the configured parameters that affect output generation. Since default values might change over time, researchers SHOULDreport all configuration values, even if they used the defaults. Checksums and fingerprints SHOULD be reported since they identify specific versions and configurations. Depending on the study context, other properties such as the context window size (number of tokens) SHOULD be reported. When using quantized models, researchers SHOULD report the quantization level (e.g., 4-bit, 8-bit) and method (e.g., GPTQ or AWQ), as different quantization approaches produce different outputs, affecting both output quality and reproducibility. Researchers SHOULD motivate in the PAPER why they selected certain models, versions, and configurations. Reasons may be monetary, technical, or methodological (e.g., planned comparison to previous work). Depending on the specific study context, additional information regarding the experiment or tool architecture SHOULD be reported.
A common customization approach for existing LLMs is fine-tuning. If a model was fine-tuned, researchers MUST describe the fine-tuning goal (e.g., improving the performance for a specific task), the fine-tuning procedure (e.g., full fine-tuning vs. Low-Rank Adaptation (LoRA), selected hyperparameters, loss function, learning rate, batch size, etc.), and the fine-tuning dataset (e.g., data sources, the preprocessing pipeline, dataset size) in the PAPER. Researchers SHOULD either share the fine-tuning dataset as part of the SUPPLEMENTARY MATERIAL or explain in the PAPER why the data cannot be shared (e.g., because it contains confidential or personal data that could not be anonymized). The same applies to the fine-tuned model weights. Suitable benchmarks and metrics SHOULD be used to compare the base model with the fine-tuned model.
In summary, our recommendation is to report:
Model/tool name and version (MUST in PAPER );
All relevant configured parameters that affect output generation (MUST in PAPER );
Default values of all available parameters (SHOULD );
Checksum/fingerprint of used model version and configuration (SHOULD );
Additional properties such as context window size (SHOULD );
Model quantization level and method, if applicable (SHOULD ).
For fine-tuned models, additional recommendations apply:
Fine-tuning goal (MUST in PAPER );
Fine-tuning dataset creation and characterization (MUST in PAPER );
Fine-tuning parameters and procedure (MUST in PAPER );
Fine-tuning dataset and fine-tuned model weights (SHOULD );
Validation metrics and benchmarks (SHOULD ).
Commercial models (e.g., GPT-5) or LLM-based tools (e.g., ChatGPT) might not give researchers access to all required information. For these tools, researchers SHOULD report what is available and openly acknowledge limitations that hinder reproducibility.
Example(s)
Based on the documentation that OpenAI and Azure provide (OpenAI 2025; Microsoft 2025), researchers might, for example, report:
“We integrated a gpt-4 model in version 0125-Preview via the Azure OpenAI Service, and configured it with a temperature of 0.7, top_p set to 0.8, a maximum token length of 512, and the seed value 23487. We ran our experiment on 10th January 2025. The system fingerprint was fp_6b68a8204b.
Kang, Yoon, and Yoo (2023) provide a similar statement in their paper on exploring LLM-based bug reproduction (Kang, Yoon, and Yoo 2023):
“We access OpenAI Codex via its closed beta API, using the code-davinci-002 model. For Codex, we set the temperature to 0.7, and the maximum number of tokens to 256.”
Our guidelines additionally recommend reporting a checksum/fingerprint and exact dates; otherwise,this example is close to our recommendations.
Dhar, Vaidhyanathan, and Varma (2024)(Dhar, Vaidhyanathan, and Varma 2024) assessed whether LLMs can generate architectural design decisions, detailing the system architecture and the LLM’s role within it. They provide information on the fine-tuning approach and datasets, including the source of architectural decision records, preprocessing methods, and data selection criteria.
For self-hosted models, the SUPPLEMENTARY MATERIAL can become a true replication package. For example, for models provisioned using ollama, one can report the specific tag and checksum, e.g., “llama3.3, tag 70b-instruct-q8_0, checksum d5b5e1b84868.” Given suitable hardware, running the model is then as easy as executing the following command: ollama run llama3.3:70b-instruct-q8_0
… Unchanged text
Benefits
Reporting this information is a prerequisite for the verification, reproduction, and replication of LLM-based studies. While LLMs are inherently non-deterministic, this cannot excuse dismissing reproducibility. Although exact reproducibility is hard to achieve, reporting the information outlined in this guideline helps researchers come as close as possible to that standard.
Challenges
Different model providers and modes of operating the models allow for varying degrees of information. For example, OpenAI provides a model version and a system fingerprint describing the backend configuration, which can also influence the output. However, the fingerprint is intended only to detect changes in the model or its configuration; one cannot go back to a certain fingerprint. As a beta feature, OpenAI lets users set a seed parameter to receive “(mostly) consistent output” (OpenAI 2023). However, the seed value does not allow for full reproducibility and the fingerprint changes frequently. Although, as motivated above, open models significantly simplify re-running experiments, they also come with challenges in terms of reproducibility, as generated outputs can be inconsistent despite setting the temperature to 0 and using a seed value (see GitHub issue for Llama3). Setting the temperature to 0 configures greedy decoding (always selecting the most probable next token), which minimizes output variability but can degrade quality by producing repetitive text and missing higher-quality responses (Holtzman et al. 2020).
Even with a temperature of 0, full determinism is rarely guaranteed: floating-point arithmetic on GPUs causes slight numerical differences that cascade into divergent token selections (Yuan et al. 2025), Sparse Mixture-of-Experts routing amplifies this effect (Chann 2023), silent backend changes in commercial APIs produce different outputs over time (L. Chen, Zaharia, and Zou 2024), and even self-hosted open models with identical settings do not always yield consistent outputs (Angermeir et al. 2025; Song et al. 2025). Researchers SHOULD NOT treat a temperature of 0 as a guarantee of reproducibility, but as one measure among several, including fixed seed values (OpenAI 2023), system fingerprints, and archiving of raw outputs. When a temperature of 0 is chosen primarily for reproducibility, this motivation SHOULD be stated explicitly, along with an acknowledgment of its potential impact on output quality.
Study Types
This guideline MUST be followed for all study types for which the researcher has access to (parts of) the model’s configuration. They MUST always report the configuration that is visible to them, acknowledging the reproducibility challenges of commercial tools and models that are offered as-a-service. Depending on the specific study type, researchers SHOULD provide additional information on the system and prompt design (see System and Prompt DesignG3: System and Prompt Design ), session traces (see Session TracesG4: Session Traces ), and specific limitations and mitigations (see Limitations and MitigationsG8: Limitations and Mitigations ).
Missing version, configuration, or parameter information is typically a minor revision request. Before concluding that information is absent, reviewers should check appendices and supplementary materials, as details are sometimes reported there rather than in the main text. Rejection over missing details is rarely warranted unless the omissions obscure deeper methodological problems.
tl;dr: Researchers MUSTdescribethe full architecture of LLM-based tools thattheydevelopintheirPAPER.Thisincludestheroleof the LLM, interactions with other components,andtheoverallsystembehavior. If autonomousagentsareused, researchers MUST specify agent roles, reasoning frameworks, and communication flows.Hosting,hardwaresetup, and latencyimplicationsMUSTbereported. For toolsusing retrieval oraugmentation methods,datasources, integrationmechanisms, and updateandversioningstrategiesMUSTbedescribed. Forensemblearchitectures, the coordinationlogicbetweenmodelsMUST be explainedinthePAPER.ResearchersSHOULDincludearchitecturaldiagramsandjustifydesigndecisions.Non-disclosed confidentialor proprietary components MUST be acknowledged as reproducibility limitations.
Rationale
LLM-based toolsoftenhavecomplex software layersaroundthemodel(s) that pre-process data, prepare prompts, filter user requests, or post-process responses. For example, ChatGPT and GitHub Copilot use the same underlying models, but their outputs differ significantly because Copilot automatically adds project context;researchers can also build tools using models directly via APIs. Theinfrastructure and businesslogicaround the baremodelcansignificantlycontributeto the performance of atoolforagiventask,forinstance,RAGpipelines,outputparsers,andretrylogicallshapethefinaloutputindependentlyofthemodelitself.
This section addresses thesystem-levelaspectsofLLM-basedtoolsthat researchers develop, complementing Version and ConfigurationG2: Version and Configuration , which focuses on model-specific details.Whilestandalone LLMs requirelimitedarchitecturaldocumentation, amoredetaileddescriptionisrequiredforstudysetupsandtoolarchitecturesthatintegrateLLMswithothercomponentstocreatemorecomplexsystems.Thissectionprovidesguidelinesfordocumentingthesebroaderarchitectures.
Recommendations
Researchers MUST clearly describe the tool architecture and what exactly the LLM (or ensemble of LLMs) contributes to the tool or method presented in a research paper. Researchers SHOULDprovideahigh-levelarchitecturaldiagramtoimprovetransparency.Toimproveclarity,researchersSHOULD explain design decisions,particularlyregardinghowthemodelswerehostedandaccessed(API-based,self-hosted,etc.) and which retrieval mechanisms were implemented (keyword search, semantic similarity matching, rule-based extraction, etc.). Researchers MUSTNOTomitcriticalarchitecturaldetailsthatcouldaffectreproducibility,suchasdependenciesonproprietarytoolsthatinfluencetoolbehavior.Especially for time-sensitive measurements, the descriptionof the hostingenvironmentiscentral, asitcansignificantlyimpacttheresults.ResearchersMUST clarify whether local infrastructure or cloud services were used, including detailed infrastructure specifications and latency considerations.
Agent-Based and ComplexSystems:
If an LLM is used as a standalone system, for example, by sending prompts directly to a GPT-4o model via the OpenAI API without pre-processing the prompts or post-processing the responses, a brief explanation ofthisapproach is usually sufficient. However,ifLLMsareintegratedintomorecomplex systems with pre-processing, retrieval mechanisms, or autonomous agents, researchersMUSTprovideadetaileddescriptionofthesystemarchitectureinthePAPER.Aspects to consider are how the LLM interacts with other components such as databases, external APIs, and frameworks. If the LLM is part of an agent-based system that autonomously plans or executes tasks, researchers MUST describe its exact architecture, including the agents’ roles (e.g., planner, executor, coordinator), whether it is a single-agent or multi-agent system, how it interacts with external tools and users, and the reasoning framework used (e.g., chain-of-thought, self-reflection, multi-turn dialogue). Anyagentconfigurationviacontextfiles(e.g.,AGENTS.md)SHOULDalsobedocumentedaspartofthesystemarchitecture(seePromptsandLogsG4:PromptsandLogsfordetailedreportingrequirements). For agent-based systems that use external tools (e.g., Claude Code and its subagents), researchers MUSTdiscussagent behavior traceability by clearly delineating threedistinctcomponents: (1) LLM Input/Output:Internal deliberation, planning, and interpretation performed by the model. (2) External tool calls:Specific invocations of APIs, databases, file systems, or other external services thattheagentexplicitlytriggers (e.g., via the Model Context Protocol (MCP)). (3) User or system interactions:Human-in-the-loop feedback, environment responses, or multi-agent communication. ResearchersSHOULDreportcompleteexecutiontracesthatshowthesequenceandcausalitybetweenthesecomponents,includinginputsandoutputsofallexternaltoolcalls. This traceabilityisessentialfordeterminingwhether task success isattributable to the LLM’soutputandtool-callingcapabilities, the external tools’functionality, or their interaction patterns. Whenfulltoolcallloggingisnotfeasibleduetoprivacyorproprietaryconstraints,researchersSHOULDproviderepresentativeexamplesoranonymizedtracesthatdemonstratetheagent’sdecision-makingprocess.
Retrieval,Augmentation, and Ensembles:
If retrievaloraugmentation methods were used (e.g.,retrieval-augmentedgeneration(RAG), rule-based retrieval, structured query generation, or hybrid approaches), researchers MUST describe how external data is retrieved, stored, and integrated into the LLM’s responses.Thisincludesspecifying the type of storage ordatabaseused (e.g., vector databases, relational databases, knowledge graphs) and how the retrieved information is selected andused. Stored data used for context augmentation MUST be reported, including detailson data preprocessing, versioning, and update frequency.Ifthisdatais not confidential, an anonymized snapshot ofthedatausedforcontextaugmentationSHOULD be made available as SUPPLEMENTARY MATERIAL. For ensemble models, in addition to following the Version and ConfigurationG2: Version and Configuration guideline for each model, the researchers MUST describe the architecture thatconnects the models.ThePAPERMUSTatleastcontainahigh-leveldescription,anddetailscanbereportedintheSUPPLEMENTARYMATERIAL.Aspectstoconsiderincludedocumentingthe logic that determines which model handles which input, theinteractionbetweenmodels, and the architectureforcombiningoutputs (e.g., majority voting, weighted averaging, sequential processing).
Example(s)
Schäfer et al. (2024)(Schäfer et al. 2024) evaluated LLMs for automated unit test generation, providing a comprehensive description of the system architecture including code parsing, prompt formulation, LLM interaction, and test suite integration. They also detail the datasets used, including sources, selection criteria, and preprocessing steps.
A second example is Yan et al. (2024)’s IVIE tool (Yan et al. 2024), which integrates LLMs into the VS Code interface. The authors document the tool architecture, detailing the IDE integration, context extraction from code editors, and the formatting pipeline for LLM-generated explanations. Thisdocumentationillustrateshowarchitecturalcomponentsbeyond the coreLLMaffect the overalltoolperformance and userexperience.
Benefits
ThesoftwarelayersaroundbareLLMssignificantlyimpacttoolperformanceandhenceneeddetaileddocumentation. Documenting the architecture and supplemental data of LLM-based systems enhances reproducibility and transparency (Lu et al. 2024), enabling experiment replication, result validation, and cross-study comparison.
Challenges
Researchersfacechallengesindocumenting LLM-based architectures,including proprietary APIs and dependencies that restrict disclosure, managing large-scale retrieval databases, and ensuring efficient query execution. They must also balance transparency with data privacy concerns, adapt to the evolving nature of LLM integrations, and,dependingonthecontext, handle the complexity of multi-agent interactions and decision-making logic, all of which can impact reproducibility and system clarity.
Study Types
This guideline MUST be followed for all studies that involve tools with system-level components beyond bare LLMs, from lightweight wrappers that pre-process user input or post-process model outputs, to systems employing retrieval-augmented methods or complex agent-based architectures.
ForLLMs for ToolsS6: LLMs for Tools , this guideline is of primary importance: researchers MUST describe the tool’s full architecture, including the role of each LLM,interactionswithothercomponents,and the overall system behavior. For Benchmarking LLMsS7: Benchmarking LLMs , researchers SHOULD describe the evaluation harness and infrastructure if it goes beyond bare model API calls (e.g., custom sandboxing, orchestration layers, or post-processing pipelines); see (Anthropic 2025) for a practitioner account of evaluation harness design for agent-based systems.ForStudying LLM UsageS5: Studying LLM Usage , researchers SHOULD describe the tool architecture of the studied tool to the extent that it is accessible, as architectural details may influence observed usage patterns. For LLMs as AnnotatorsS1: LLMs as Annotators , LLMs as JudgesS2: LLMs as Judges , LLMs for SynthesisS3: LLMs for Synthesis , and LLMs as SubjectsS4: LLMs as Subjects , if the research setup involves a custom pipeline (e.g., retrieval-augmented generation for annotation or chained prompts for synthesis), the architecture SHOULD also be reported.
Advice for Reviewers
As with other guidelines, missing architecturalinformation is typically a minor revision request unless so much is missing that methodological rigor cannot be assessed. Reviewers may ask authors to move key details from supplements into the paper body to ensure the main text is self-contained.
Regarding proprietary or confidential components, three principles apply: (1) empirically evaluating an opaque artifact is a valid scientific contribution; (2) the less available and inspectable the artifact, the weaker the contribution; (3) scientific instruments including tools, metrics, scales, and experimental materials, must be fully disclosed, even when the object under study cannot be.
2026.04
Report System and Prompt Design (G3)
tl;dr: Researchers MUST describe in the PAPER the full architecture of LLM-based tools, from standalone uses to agentic systems, including the LLM’s role and its interactions with other components. Hosting, infrastructure, and latency MUST be reported. Researchers MUST publish all prompts as SUPPLEMENTARY MATERIAL, including prompt templates with representative instances and the dynamic generation process where applicable; sensitive content MUST be anonymized. If full prompt disclosure is not feasible, summaries or examples SHOULD be provided. The prompting strategy and prompt reuse across models and configurations MUST be specified, and how the prompts were developed SHOULD be described. Researchers MUST describe context-file mechanisms used and summarize in the PAPER which tools and skills were exposed; full file contents and the complete tool catalog (schemas, definitions, Model Context Protocol (MCP) servers) SHOULD be published as SUPPLEMENTARY MATERIAL. For agentic systems, researchers MUST specify the agents’ roles, reasoning frameworks,communicationflows, and agent behavior traceability across LLM output, tool calls, and user interactions. Forretrieval-augmented generation (RAG) and similar methods, researchers MUST describe how external data was retrieved, stored, and integrated. Where legally possible, the implementation SHOULD be open-sourced; non-disclosedproprietary components MUST be acknowledged as reproducibility limitations.
Rationale
LLM-based studies rest on artifacts that researchers design, author, or configure before any model is invoked: software layersthat pre-process data, prepare prompts, filter user requests, or post-process responses, and the context those layers feed into each call (Galster et al. 2026). Context includes prompt templates, context files, tool and skill schemas, and retrieval mechanisms that bring external data into the call. For example, ChatGPT and GitHub Copilot use the same underlying models, but their outputs differ substantially because Copilot automatically adds project context. Researchers can also build tools using models directly via APIs.
Prompts are central to any LLM-based study (Sclar et al. 2024). A prompt is a concrete input to an LLM that guides its output (Schulhoff et al. 2024). Depending on the task, a prompt may include instructions (e.g., “classify the following bug report”), background information, input data, and output format specifications (e.g., “respond as JSON with fields ‘category’ and ‘justification’”), with outputs ranging from unstructured text to structured formats such as JSON (DAIR.AI 2024). A prompt template is a parameterized structure containing static elements (e.g., instructions, output format specifications) and placeholders for dynamic content (e.g., source code under analysis) that are filled in at runtime to construct concrete prompts (Schulhoff et al. 2024). In automated studies, researchers typically design prompt templates from which individual prompts are then instantiated. Prompts substantially influence a model’s output, so how they are formatted and integrated into an LLM-based study is essential for transparency, verifiability, and reproducibility. Sclar et al. (2024)(Sclar et al. 2024) question the methodological validity of comparing models with “an arbitrarily chosen, fixed prompt format”, because their research has shown that the performance of different prompt formats only weakly correlates between different models.
These guidelines do not apply when LLMs are used solely for language polishing, paraphrasing, translation, tone or style adaptation, or layout edits (see Scope ).
Recommendations
Researchers MUST clearly describe the tool architecture and what exactly the LLM (or ensemble of LLMs) contributes to the tool or method presented in a research paper, including any dependencies on proprietary tools that affect reproducibility. ResearchersSHOULD explain design decisionsand which retrieval mechanisms were implemented (keyword search, semantic similarity matching, rule-based extraction, etc.). Researchers SHOULD describe how the models were hosted and accessed; for time-sensitive measurements, the hosting choice (e.g., self-hosting on local hardware, an aggregator such as OpenRouter, or a vendor API such as the OpenAI API) can substantially impact results, so researchers MUST clarify whether local infrastructure or cloud services were used, including detailed infrastructure specifications and latency considerations. Where legally possible (e.g., not restricted by industry-partner agreements), researchers SHOULD release the source code of their implementation under an open-source license.
Standalone LLMs require limited architectural documentation, while study setups that integrate LLMs with other components, configure agents, or depend on particular context files or tool schemas require more detailed reporting. In the topic-specific paragraphs that follow, the PAPER MUST contain a high-level description of any reported component, with full details provided as SUPPLEMENTARY MATERIAL.
Prompt Reporting:
Researchers MUST report all prompts used in an empirical study, including instructions, background information, input data, and output indicators. The complete set MUST be made publicly available as SUPPLEMENTARY MATERIAL; when the paper cannot include all prompts in full, it SHOULD provide summaries and representative examples. The only exception to this disclosure requirement is when privacy, anonymity, or confidentiality concerns (e.g., when working with industry partners) prevent complete disclosure; in such cases, researchers SHOULD provide summaries and representative examples instead. For prompts containing sensitive information, researchers MUST anonymize personal identifiers, replace proprietary code with placeholders, and clearly highlight modified sections. When prompt templates are used, researchers MUST report them alongside representative instances, specifying which parts are static and which are dynamically filled. When prompts are generated dynamically (e.g., through preprocessing or retrieval-augmented generation (RAG)), the generation process MUST be thoroughly documented, including any automated algorithms or rules. Researchers SHOULD specify the exact formatting of prompts, including how code snippets were enclosed (e.g., triple backticks), whether whitespace was preserved, and how other artifacts such as error messages and stack traces were presented. For studies involving human participants who create or modify prompts, researchers SHOULD describe how these prompts were collected and analyzed.
Prompt Development:
Researchers SHOULD explain in the PAPER how they developed the prompts and why they decided to follow certain prompting strategies. If prompts from the early phases of a research project are unavailable, researchers SHOULD at least summarize the prompt evolution.
Prompt development is often iterative, involving collaboration between human researchers and AI tools. Researchers SHOULD report any instances in which LLMs were used to suggest prompt refinements and how these suggestions were incorporated. A prompt changelog can track prompt evolution, including key revisions and reasons for changes (e.g., v1.0: initial prompt; v2.0: incorporated examples of ideal responses). Because prompt effectiveness varies between models and model versions, researchers MUST make clear which prompts were used for which models in which versions and with which configuration.
Prompting Strategy and Input Handling:
Researchers MUST specify whether zero-shot, one-shot, or few-shot prompting was used. For few-shot prompts, the examples provided to the model SHOULD be clearly outlined, along with the rationale for selecting them. If multiple versions of a prompt were tested, researchers SHOULD describe how these variations were evaluated and how the final design was chosen.
When dealing with extensive or complex prompt context, researchers SHOULD describe the strategies they used to handle input length constraints (e.g., truncating, summarizing, or splitting prompts into multiple parts). Token optimization measures, such as simplifying code formatting or removing unnecessary comments, SHOULD also be documented if applied.
Context Files and Agent Configuration:
Agentic coding tools expose a range of configuration and extension mechanisms including context files, skills, subagents, hooks, settings, rules, and Model Context Protocol (MCP) servers (Galster et al. 2026). Context files such as AGENTS.md or CLAUDE.md are version-controlled Markdown files containing persistent, project-specific instructions automatically included in the context on each call (Mohsenimofidi et al. 2026). Researchers MUST describe which context-file mechanisms were used and SHOULD publish the full contents as SUPPLEMENTARY MATERIAL, because they shape agent behavior similarly to system prompts. Because context files are version-controlled, their evolution across a study is recoverable from the project’s git history and from the runtime traces reported under Session TracesG4: Session Traces .
Tool Catalog and Skill Definitions:
Agentic systems expose a set of callable tools, skills, and sub-agents to the model on each call, along with their schemas: parameter names, types, and natural-language descriptions as the model sees them. Tools vary in granularity, from one operation per tool to generic execution interfaces. Claude Code, for example, does not register individual command-line utilities but defines a single Bash tool (PowerShell on Windows) through which the agent runs arbitrary shell commands such as grep or ls(Anthropic 2026). The catalog, the wording of descriptions, and the order in which tools are presented all influence which tool the model selects, so the exact serialized form matters for reproducibility. Researchers MUST summarize in the PAPER which tools and skills were exposed to the model, and SHOULD include the complete list (names with one-line purposes), tool schemas, skill definitions, sub-agent definitions, and the identities of any connected Model Context Protocol (MCP) servers as SUPPLEMENTARY MATERIAL. Where sub-agents are used, researchers SHOULD also describe the delegation pattern: which sub-agent handles which kind of task, and how control returns to the orchestrator.
Agentic Systems and Pipelines:
If an LLM is used as a standalone system, for exampleby sending prompts directly to a GPT-5 model via the OpenAI API without pre-processing the prompts or post-processing the responses, a brief explanationis usually sufficient. For complex systems with pre-processing, retrieval mechanisms, or autonomous agents, aspects to consider include how the LLM interacts with other components such as databases, external APIs, and frameworks. If the LLM is part of an agent-based system that autonomously plans or executes tasks, researchers MUST describe its exact architecture, including the agents’ roles (e.g., planner, executor, coordinator), whether it is a single-agent or multi-agent system, how it interacts with external tools and users, and the reasoning framework used (e.g., chain-of-thought, self-reflection, multi-turn dialogue).For agent-based systems that use external tools (e.g., Claude Code and its subagents), researchers MUST describe agent behavior traceabilitybydelineating: (1) LLM Input/Output, the internal deliberation, planning, and interpretation performed by the model; (2) External tool calls, the agent’s explicit invocations of APIs, databases, file systems, or other external services(e.g., via the Model Context Protocol (MCP)); (3) User or system interactions, human-in-the-loop feedback, environment responses, or multi-agent communication.This separation lets readers attribute task successto the LLM, the external tools, or their interaction; runtime trace-reporting for these components is covered in Session TracesG4: Session Traces .
RAG and Ensembles:
If retrieval-augmented generation (RAG) or related methods were used (e.g., rule-based retrieval, structured query generation, or hybrid approaches), researchers MUST describe how external data was retrieved, stored, and integrated into the LLM’s responses, including the type of storage(e.g., vector databases, relational databases, knowledge graphs) and howretrieved information was selected. Stored data used for context augmentation SHOULD be reported, includingdata preprocessing, versioning, and update frequency; if not confidential, an anonymized snapshotSHOULD be made available as SUPPLEMENTARY MATERIAL. For ensemble models, in addition to following the Version and ConfigurationG2: Version and Configuration guideline for each model,researchersSHOULD describe the architecture connecting them: the routing logic that determines which model handles which input, model interactions, and the output combination strategy (e.g., majority voting, weighted averaging, sequential processing).
Example(s)
Schäfer et al. (2024)(Schäfer et al. 2024) evaluated LLMs for automated unit test generation, providing a detailed description of the system architecture including code parsing, prompt formulation, LLM interaction, and test suite integration. They also detail the datasets used, including sources, selection criteria, and preprocessing steps.
A second example is Yan et al. (2024)’s IVIE tool (Yan et al. 2024), which integrates LLMs into the VS Code interface. The authors document the tool architecture, detailing the IDE integration, context extraction from code editors, and the formatting pipeline for LLM-generated explanations.
Liang et al. (2024)’s paper (Liang et al. 2024) is a good example of comprehensive prompt reporting. The authors make the exact prompts available in their SUPPLEMENTARY MATERIAL on Figshare, including details such as code blocks enclosed in triple backticks. The PAPER explains the rationale behind the prompt design and the data output format, and it includes an overview figure and two concrete examples, keeping the main text concise while remaining reproducible.
Benefits
Documenting the architecture and supplemental data of LLM-based systems strengthens reproducibility and transparency (Lu et al. 2024), enabling experiment replication, result validation, and cross-study comparison.
Detailed prompt documentation strengthens verifiability, reproducibility, and comparability of LLM-based studies, letting other researchers replicate studies, refine prompts, and evaluate how different content types and formatting choices influence LLM behavior. Reporting the tool catalog and context files enables other researchers to reconstruct the exact context the model saw, which is often the dominant factor in agent behavior.
Challenges
Documenting LLM-based architectures involves challenges such as proprietary APIs and dependencies that restrict disclosure, managing large-scale retrieval databases, and ensuring efficient query execution. Researchers must also balance transparency with data privacy concerns, adapt to the evolving nature of LLM integrations, andhandle the complexity of multi-agent interactions and decision-making logic, all of which can impact reproducibility and system clarity.
Prompts themselves are challenging to document because they often combine multiple components such as code, error messages, and explanatory text, and privacy or confidentiality concerns can hinder sharing.
Not all systems allow reporting of complete (system) prompts, context files, and tool schemas. For commercial tools, researchers MUST report all available information and acknowledge unknown aspects as limitations. Disclosure practices vary across vendors: some publish their system prompts, others keep them proprietary. Where prompts are published, they also change between releases (Willison 2026), so researchers SHOULD record the tool version and date of use even when the prompt content itself is unavailable. Understanding suggestions of commercial tools such as GitHub Copilot might require recreating the exact state of the codebase at the time the suggestion was made, which is a challenging context to report. One solution is to use version control to capture the exact state of the codebase when a recommendation was made, keeping track of the files that were automatically added as context. We also recommend exploring open-source tools such as OpenCode(OpenCode Contributors 2025), which expose more of the configuration that shapes agent behavior.
Study Types
This guideline MUST be followed for all studies that involve tools with system-level components beyond bare LLMs, from lightweight wrappers that pre-process user input or post-process model outputs, to systems employing retrieval-augmented methods or complex agent-based architectures. It also MUST be followed by all studies that use concrete prompts or prompt templates.
For LLMs for ToolsS6: LLMs for Tools , this guideline is of primary importance: researchers MUST describe the tool’s full architecture, explain how prompts were generated and structured within the tool, report the context files, tool catalog, and skill definitions used, and document how the role of each LLM fits into the overall system behavior. For Benchmarking LLMsS7: Benchmarking LLMs , researchers SHOULD describe the evaluation harness and infrastructure if it goes beyond bare model API calls (e.g., custom sandboxing, orchestration layers, or post-processing pipelines); see (Anthropic 2025) for a practitioner account of evaluation harness design for agent-based systems. Researchers using pre-defined prompts (e.g., HumanEval, SWE-Bench) MUST specify the benchmark version and any modifications made to the prompts or evaluation setup. If prompt tuning, retrieval-augmented generation (RAG), or other methods were used to adapt prompts, researchers MUST disclose and justify those changes, and SHOULD make the relevant code publicly available. For Studying LLM UsageS5: Studying LLM Usage , researchers SHOULD describe the tool architecture of the studied tool to the extent that it is accessible, as architectural details may influence observed usage patterns. For controlled experiments under Studying LLM UsageS5: Studying LLM Usage , exact prompts MUST be reported for all conditions. For LLMs as AnnotatorsS1: LLMs as Annotators , researchers MUST document any predefined coding guides or instructions included in prompts, as these influence how the model labels artifacts. For LLMs as JudgesS2: LLMs as Judges , researchers MUST report the evaluation criteria, scales, and examples embedded in the prompt to ensure consistent interpretation. For LLMs for SynthesisS3: LLMs for Synthesis tasks (e.g., summarization, aggregation), researchers MUST document the sequence of prompts used to generate and refine outputs, including follow-ups and clarification queries. For LLMs as SubjectsS4: LLMs as Subjects (e.g., simulating human participants), researchers MUST report any role-playing instructions, constraints, or personas used to guide LLM behavior. For LLMs as AnnotatorsS1: LLMs as Annotators , LLMs as JudgesS2: LLMs as Judges , LLMs for SynthesisS3: LLMs for Synthesis , and LLMs as SubjectsS4: LLMs as Subjects , if the research setup involves a custom pipeline (e.g., retrieval-augmented generation for annotation or chained prompts for synthesis), the architecture SHOULD also be reported.
Advice for Reviewers
As with other guidelines, missing architectural or prompt information is typically a minor revision request unless so much is missing that methodological rigor cannot be assessed. Reviewers may ask authors to move key details from supplements into the paper body to ensure the main text is self-contained.
Regarding proprietary or confidential components, three principles apply: (1) empirically evaluating an opaque artifact is a valid scientific contribution; (2) the less available and inspectable the artifact, the weaker the contribution; (3) scientific instruments including tools, metrics, scales, and experimental materialsmust be fully disclosed, even when the object under study cannot be.
A challenging aspect of prompt reporting is the description of prompt development, which is inherently iterative and creative. Reviewers should not expect justification of each word choice or post-hoc rationalized accounts of prompt generation. Instead, reviewers should focus on whether (1) a new research team could use (not reproduce) exactly the same prompts in exactly the same way, and (2) potential biases or validity issues are transparent.
Section Session TracesG4: Session Traces : runtime reporting of interaction logs and runtime traces that correspond to the static artifacts reported here.
tl;dr: ResearchersMUSTpublishallpromptsor,whenusingtemplates,prompttemplateswithrepresentativeinstances,includingtheirstructure,content,formatting,anddynamiccomponents,asSUPPLEMENTARYMATERIAL.Iffullpromptdisclosureisnotfeasible,forexample,duetoprivacyorconfidentialityconcerns,summariesorexamplesSHOULDbeprovided.Promptdevelopmentstrategies(e.g.,zero-shot,few-shot),rationale,andselectionprocessMUSTbedescribedinthePAPER.Whenpromptsarelongorcomplex,inputhandlingandtokenoptimizationstrategiesMUSTbedocumented.Fordynamicallygeneratedoruser-authoredprompts,generationandcollectionprocessesMUSTbereported.PromptreuseacrossmodelsandconfigurationsMUSTbespecified.ResearchersSHOULDreportpromptrevisionsandpilottestinginsights.Foragenticsystems,developedplansSHOULDbereportedasSUPPLEMENTARYMATERIALandinteractionlogsSHOULDbegeneralizedtoincludehuman-agentinteractiontracesasSUPPLEMENTARYMATERIAL.AllcontextfilesusedtoconfigureAIagentbehaviorSHOULDbereportedasSUPPLEMENTARYMATERIAL. To address model non-determinism and ensure reproducibility, especially when targeting SaaS-based commercial tools,full interaction logs (prompts and responses) SHOULDbeincluded as SUPPLEMENTARY MATERIAL if privacy and confidentiality can be ensured.
Rationale
PromptsarecriticalforanystudyinvolvingLLMs(Sclaretal.2024).Apromptisaconcreteinput to anLLMthatguidesitsoutput(Schulhoffetal.2024).Dependingon the task,apromptmayincludeinstructions(e.g.,“classify the followingbugreport”),context(e.g.,ataxonomyofbugcategories,relatedsourcecode),inputdata(e.g.,thebugreporttext),andoutputformatspecifications(e.g.,“respondasJSONwithfields‘category’and‘justification’”),withoutputsrangingfromunstructuredtexttostructuredformatssuchasJSON.2Aprompttemplateisaparameterizedstructurecontainingstaticelements(e.g.,instructions,outputformatspecifications)andplaceholdersfordynamiccontent(e.g.,sourcecodeunderanalysis)thatarefilledinatruntimetoconstructconcreteprompts(Schulhoffetal.2024).Inautomatedstudies,researcherstypicallydesignprompttemplatesfromwhichindividualpromptsaretheninstantiated.Promptssignificantlyinfluencethequalityoftheoutputofamodel,andunderstandinghowexactlytheywereformattedandintegratedintoanLLM-basedstudyisessentialtoensuretransparency,verifiability,andreproducibility.Sclaretal.(2024)questionthemethodologicalvalidityofcomparingmodelswith“anarbitrarilychosen,fixedpromptformat”,becausetheirresearchhasshownthattheperformanceofdifferentpromptformatsonlyweaklycorrelatesbetweendifferentmodels(Sclaretal.2024).WeremindthereaderthattheseguidelinesdonotapplywhenLLMsareusedsolelyforlanguagepolishing,paraphrasing,translation,toneorstyleadaptation,orlayoutedits(seeScope).Thisimpliesthatourguidelinesdonotrecommendthatresearcherdisclosethepromptstheyhaveusedforsuchtasks.
Recommendations
PromptReporting:
ResearchersMUSTreportallpromptsthatwereusedforanempiricalstudy, including allinstructions,context,inputdata, and outputindicators.Theonlyexceptiontothisisiftranscriptsmightidentifyanonymousparticipantsorrevealpersonalorconfidentialinformation,forexample,whenworkingwithindustrypartners.Whenprompttemplatesareused,researchersMUSTreportthemalongsiderepresentativepromptinstances,specifyingwhichpartsarestaticandwhicharedynamicallyfilled.Moreover,researchersSHOULDspecifytheexactformattingofprompts,includinghowcodesnippetswereenclosed(e.g.,markdown-stylecodeblocks,triplebackticks),whetherwhitespacewaspreserved,andhowotherartifactssuchaserrormessagesandstacktraceswerepresented.
Promptdevelopmentisofteniterative,involvingcollaborationbetweenhumanresearchersandAItools. Researchers SHOULD report anyinstancesin which LLMs were used to suggestpromptrefinementsandhowthesesuggestionswereincorporated. Promptsmayneedrevisioninresponsetofailurecases, and pilottesting is vitaltoensurereliableresults. Ifsuchtestingwasconducted,researchersSHOULDsummarizekeyinsights,includinghowdifferentpromptvariationsaffectedoutputqualityandwhichcriteriawereusedtofinalizethepromptdesign.Apromptchangelogcantrackpromptevolution,includingkeyrevisionsandreasonsforchanges(e.g.,v1.0:initialprompt;v2.0:incorporatedexamplesofidealresponses).Sinceprompteffectivenessvariesbetweenmodelsandmodelversions,researchersMUSTmakeclearwhichpromptswereusedforwhichmodelsinwhichversionsandwithwhichconfiguration.
PromptingStrategyandInputHandling:
ResearchersMUSTspecifywhetherzero-shot, one-shot,orfew-shotpromptingwasused.Forfew-shot prompts,theexamplesprovided to the modelSHOULDbeclearlyoutlined, alongwith the rationaleforselectingthem. Ifmultipleversionsofapromptweretested,researchersSHOULDdescribehowthesevariationswereevaluatedandhowthefinaldesignwaschosen.
Whentryingtoverifyresults,even with the exact same prompts, decoding strategies, and parameters, LLMs can still behave non-deterministically. Non-determinism can arise frombatching, input preprocessing, and floatingpoint arithmetic on GPUs (Chann 2023). Thus,inordertoenable other researchers to verify the conclusions thatresearchershave drawn from LLM interactions, researchersSHOULDreport the fullinteractionlogs(promptsandresponses) as part of theirSUPPLEMENTARYMATERIAL. Reportingthisisespeciallyimportant for studies targeting commercial software-as-a-service (SaaS) solutions such as ChatGPT. The rationale forthis is similar to therationalefor reporting interview transcripts in qualitative research. Just as a human participant might give different answers to the same question asked two months apart, the responses from tools such as ChatGPT can also vary over time.
Agentic Systems:
For agentic systems that autonomously plan and execute tasks, the developed plans SHOULD be reported as SUPPLEMENTARY MATERIAL if available,asthey document the sequenceofactions the systemchosetopursue and the intermediategoalsitset. InteractionlogsSHOULD be generalizedtoincludefullinteractiontracesbetweenhumansandagenticsystems,capturinghuman-in-the-loopfeedback,approvalorrejectiondecisions,anditerativerefinements.ThesetracesSHOULDalsobereported as SUPPLEMENTARY MATERIAL sothatreaderscanreconstructwhichactionstheagenttook, verifyhumanoversightdecisions, andassessthequalityoftool-generatedoutputs.
An example of reporting full interaction logs is the study by Ronanki, Berger, and Horkoff (2023)(Ronanki, Berger, and Horkoff 2023), for which they reported the full answers of ChatGPT and uploaded them to Zenodo.
Unlike human participant conversations, which often cannot be reported dueto confidentiality, LLM interaction logs can be shared. This enables reproduction studies, tracking of response changes over time or across model versions, and secondary research on LLM consistency for specific SE tasks.
Not all systems allow the reporting of complete (system)promptsand interaction logs with ease.This hinders transparency and verifiability.Wealsorecommendexploringopen-sourcetoolssuchasOpenCode(OpenCodeContributors2025). For commercial tools, researchers MUST report all available information and acknowledge unknown aspects as limitations. Understandingsuggestions of commercial tools suchasGitHubCopilotmightrequirerecreatingtheexactstateofthecodebaseatthetimethesuggestionwasmade, achallengingcontext to report.Onesolutioncouldbetouseversioncontroltocapturetheexactstateofthecodebasewhenarecommendationwasmade, keepingtrackofthefilesthatwereautomaticallyaddedascontext.
ForLLMsforToolsS6:LLMsforTools,researchersMUSTexplainhowpromptsweregeneratedandstructuredwithin the tool. For Studying LLM UsageS5: Studying LLM Usage , especially forcontrolledexperiments,exactpromptsMUSTbereportedforallconditions.For observational studies,especiallytheones targeting commercial tools, researchers MUST report the full interaction logs except for when transcripts might identify anonymous participants or reveal personal or confidential information. Asmentionedbefore,ifthe complete interaction logs cannot be shared, e.g., because they contain confidential information, the prompts and responses MUST at least be summarized and described in the PAPER. For LLMsas AnnotatorsS1: LLMs as Annotators , researchersMUSTdocumentanypredefinedcodingguidesorinstructionsincludedinprompts,astheseinfluencehowthemodellabelsartifacts.ForLLMs as JudgesS2: LLMs as Judges , researchersMUSTreporttheevaluationcriteria,scales,andexamplesembeddedintheprompttoensureaconsistentinterpretation.ForLLMs for SynthesisS3: LLMs for Synthesistasks(e.g., summarization,aggregation),researchersMUSTdocumentthesequenceofpromptsusedtogenerate and refineoutputs,includingfollow-upsandclarificationqueries.ForLLMs as SubjectsS4: LLMs as Subjects(e.g., simulatinghumanparticipants), researchers MUST report anyrole-playinginstructions, constraints, orpersonasusedtoguideLLMbehavior.ForBenchmarkingLLMsS7:BenchmarkingLLMsstudiesusingpre-definedprompts(e.g.,HumanEval,SWE-Bench),researchersMUSTspecifythebenchmarkversionandanymodificationsmadetothepromptsorevaluationsetup.Ifprompttuning,retrieval-augmentedgeneration(RAG),orothermethodswereusedtoadaptprompts,researchersMUSTdiscloseandjustifythosechanges,andSHOULDmaketherelevantcodepubliclyavailable.
Advice for Reviewers
As with other guidelines, missing prompt information is typically a minor revision request unless so much is missing that methodological rigor cannot be assessed. Themostchallengingaspectisthedescriptionofpromptdevelopment,whichisinherentlyiterativeandcreative. Reviewers should notexpectjustificationofeachwordchoiceorpost-hocrationalizedaccountsofpromptgeneration. Instead, reviewers should focus on whether (1)anewresearchteamcoulduse(notreproduce)exactly the samepromptsinexactlythesameway; and (2)potentialbiasesorvalidityissuesaretransparent.
tl;dr:To address model non-determinism and ensure reproducibility, especially when targeting SaaS-based commercial tools, researchers SHOULD include full interaction logs (prompts and responses)as SUPPLEMENTARY MATERIAL if privacy and confidentiality can be ensured. For agentic systems, interaction logs SHOULD be extended to cover exchanges between humans and the agent and between external tools and the agent, including human-in-the-loop feedback and approval decisions. Researchers SHOULD report the complete runtime trace as SUPPLEMENTARY MATERIAL, covering both external tool invocations (tool name, arguments, result, ordering) and which configured artifacts (skills, context files, sub-agents reported under System and Prompt DesignG3: System and Prompt Design ) were activated, so that readers can attribute task outcomes to the model, the tools, or their interaction pattern. Developed plans SHOULD be reported as SUPPLEMENTARY MATERIAL if available. When full trace disclosure is not feasible, representative or anonymized examples SHOULD be provided, and unobservable aspects of commercial-tool runs MUST be acknowledged as reproducibility limitations.
Rationale
A session is any bounded stretch of activity during which an LLM is invoked. Sessions cover one-shot prompts, batch runs, multi-turn conversations, and agentic runs that span many tool calls. A session trace records everything that crosses the LLM boundary or is produced around the model during that stretch: prompts received, responses returned, tool calls the model made together with their arguments and results, plans the model developed, and which of the statically configured artifacts (skills, context files, sub-agents, tools) were actually picked up at runtime.
Two kinds of session trace are important to capture. An interaction log captures what humans or other software tools sent to the LLM and what the LLM returned. A runtime trace captures what the LLM did at runtime: calls to external tools (APIs, file systems, databases, MCP servers, sub-agents) and activations of configured artifacts (context files, skills). Reporting both kinds is needed for an agentic run, because neither tells the full story on its own.
Even with the exact same prompts, decoding strategies, and parameters, LLMs canbehave non-deterministically. Non-determinism can arise from probabilistic sampling and, even with greedy decoding (temperature = 0), from batching, input preprocessing, and floating-point arithmetic on GPUs (Chann 2023). For other researchers to verify the conclusionsdrawn from LLM interactions,the traces of what happened at runtime are often as important as the design of the system itself. This matters particularly for studies targeting commercial software-as-a-service (SaaS) solutions such as ChatGPT, and for agentic runs where behavior depends on how the model chose among many possible tool calls.
The rationaleis similar toreporting interview transcripts in qualitative research. Just as a human participant might give different answers to the same question asked two months apart, the responses from tools such as ChatGPT can also vary over time, and the trace of an agent’s decisions on any given run is often not reproducible at a later date.
Researchers SHOULD report the full interaction logs (prompts sent to the LLM and responses returned) as part of their SUPPLEMENTARY MATERIAL. For agentic systems, interaction logs generalize to cover exchanges between humans and the agent as well as between external tools and the agent, including human-in-the-loop feedback, approval or rejection decisions, and iterative refinements. These SHOULD also be reported as SUPPLEMENTARY MATERIAL so that readers can reconstruct the sequence of exchanges and assess human oversight decisions. For traces containing sensitive information, researchers MUST anonymize personal identifiers, replace proprietary code with placeholders, and clearly highlight modified sections.
Runtime Traces:
When an LLM calls out to external tools (subagents, MCP servers, file systems, APIs, databases) or activates configured artifacts (context files, skills, sub-agents reported under System and Prompt DesignG3: System and Prompt Design ), this runtime activity forms a runtime trace distinct from the interaction log. Researchers SHOULD report the complete runtime trace as SUPPLEMENTARY MATERIAL, including for each entry the tool or artifact name, arguments (if any), result, and ordering relative to surrounding interaction-log entries. This lets readers attribute task success to the model, the external tools, or their interaction pattern, and distinguish artifacts that were configured from those that actually influenced a given run.
Agentic Plans:
For agentic systems that autonomously plan and execute tasks, the developed plans SHOULD be reported as SUPPLEMENTARY MATERIAL if available. In Claude Code, for example, a plan is a short Markdown document the user can open and edit during a session, listing the proposed steps and the files or commands the agent intends to touch. Other frameworks such as LangGraph keep plans inside the agent’s internal execution state.
All reported traces MUST be made publicly available as SUPPLEMENTARY MATERIAL, subject to privacy and confidentiality constraints. When full trace logging is not feasible, researchers SHOULD provide representative examples or anonymized traces.
Example(s)
An example of reporting full interaction logs is the study by Ronanki, Berger, and Horkoff (2023)(Ronanki, Berger, and Horkoff 2023), for which the authors reported the full answers of ChatGPT and uploaded them to Zenodo.
Benefits
Unlike human participant conversations, which often cannot be reported because of confidentiality, LLM interaction logs can be shared. This enables reproduction studies, tracking of response changes over time or across model versions, and secondary research on LLM consistency for specific SE tasks.
For agentic systems, reporting runtime traces alongside interaction logs lets readers attribute task outcomes to the right component: the model, the tool, or the orchestration pattern. Usage traces complement the static configuration reported under System and Prompt DesignG3: System and Prompt Design by showing which of the configured artifacts actually influenced a given run.
Challenges
Not all systems allowreporting of completeinteraction logs with ease, and this hinders transparency and verifiability. For commercial tools, researchers MUST report all available information and acknowledge unknown aspects as limitations. Tool-call traces for commercial SaaS agents are often opaque: the user sees the final response but not the sequence of internal tool calls. When this is the case, authors should document what was and was not observable. For agents running locally or via open-source tools, these traces are usually more accessible and SHOULD be reported whenever available. Agent frameworks differ in whether and how they log agent-to-agent communication, so reporting practices vary across studies.
Study Types
Runtime trace reporting requirements depend heavily on the study type and on the accessibility of the underlying system.
For Studying LLM UsageS5: Studying LLM Usage , especiallyobservationalstudiestargeting commercial tools, researchers MUST report the full interaction logs exceptwhen transcripts might identify anonymous participants or reveal personal or confidential information. If complete interaction logs cannot be shared (e.g., because they contain confidential information), the prompts and responses MUST at least be summarized and described in the PAPER. For LLMs for ToolsS6: LLMs for Tools that use agentic execution, researchers SHOULD report runtime traces for the runs used to evaluate the tool. For Benchmarking LLMsS7: Benchmarking LLMs that use agent-based harnesses, researchers SHOULD report runtime traces for representative runs, letting readers understand how task success depends on the agent’s decision sequence rather than the model’s raw output alone. For LLMs as AnnotatorsS1: LLMs as Annotators,LLMs as JudgesS2: LLMs as Judges,LLMs for SynthesisS3: LLMs for Synthesis,andLLMs as SubjectsS4: LLMs as Subjects, when the research setup involves multi-turn interaction or agentic orchestration, researchers SHOULD report the corresponding interaction logs and, where applicable, runtime traces.
Advice for Reviewers
As with other guidelines, missing trace information is typically a minor revision request unless so much is missing that methodological rigor cannot be assessed.Reviewers should recognize that complete trace reporting is easier for local and open-source setups than for commercial SaaS agents. When reviewing commercial-tool studies, reviewers should focus on whether authors have reported everything the system exposed and have been explicit about what remains unobservable.
Use Suitable Baselines, Benchmarks, and Metrics (G5) Changed
2026.03
Use Suitable Baselines, Benchmarks, and Metrics (G7)
tl;dr: Researchers MUST justify all benchmark and metric choices in the PAPER and SHOULD summarize benchmark structure, task types, and limitations. Where possible, traditional (non-LLM) baselines SHOULD be used for comparison. Researchers MUST explain in the PAPER why the selected metrics are suitable for the specific study; prior adoption in related work alone does not constitute sufficient justification. They SHOULD report established metrics to make study results comparable, but can report additional metrics that they consider appropriate. Due to the inherent non-determinism of LLMs, experiments SHOULD be repeated; the result distribution SHOULD then be reported using descriptive statistics. If comparing models or tools, researchers MUST use appropriate inferential statistics rather than relying solely on summary statistics. Researchers SHOULD justify the number of experiment repetitions, for example through a power analysis or by monitoring convergence of descriptive statistics.
Rationale
Meaningful evaluation requires well-understood, valid measurement instruments. Without justified benchmarks and metrics, claims about LLM performance lack the rigor needed for scientific comparison and cumulative progress. Benchmarks,baselines,andmetricsplayanimportantroleinassessingtheeffectivenessofLLMsandLLM-basedtools. A benchmark is a “standard tool for the competitive evaluation and comparison of competing systems or components according to specific characteristics, such as performance, dependability, or security” (Kistowski et al. 2015). Meanwhile, “ametric is a method, algorithm, or procedure for assigning one or more numbers to a phenomenon” (Ralph et al. 2024). A baseline is a reference point, enabling comparison of LLMs against traditional algorithms with lower computational costs.
Recommendations
Whenselecting benchmarks and metrics,itisimportanttofullyunderstand the benchmark tasks, what exactly is being measured, and how it relates to the (often latent) variables researchers actually care about. If one or more metrics or benchmarks are used, researchers:
MUST briefly justify (in the PAPER ) why selected benchmarks and metrics are suitable for the given task or study;
MUST discuss the reliability and validity (especially construct validity) of selected metrics and benchmarks;
SHOULD summarize the structure and tasks of the selected benchmark(s), including the programming language(s) and descriptive statistics such as the number of contained tasks and test cases;
SHOULD discuss the limitations of the selected benchmark(s) (e.g. widely used benchmarks such as HumanEval(M. Chen et al. 2021) and MBPP(Austin et al. 2021) Python functions assess a very specific part of software development, which is not representative of the full breadth of SE work (Chandra 2025)).
SHOULD include an example of a task and the corresponding test case(s) to illustrate the structure of the benchmark.
If multiple benchmarks exist for the same task, researchers SHOULD compare performance between benchmarks. When selecting only a subset of all available benchmarks, researchers SHOULD use the most specific benchmarks given the context.
Furthermore, researchers SHOULD check whether a less resource-intensive approach (e.g., for static analysis tasks or program repair) can serve as a baseline. If so, the LLM or LLM-based tool should be compared with such baselines using suitable metrics. Even if LLM-based tools outperform baselines, researchers SHOULD discuss whether the resources consumed justify the (often marginal) improvements (Menzies 2025).
To compare traditional and LLM-based approaches or different LLM-based tools, researchers SHOULD report established metrics whenever possible, as this allows secondary research. They can,ofcourse, report additional metrics that they consider appropriate. We briefly discuss common metrics in the Example(s) subsection below. As mentioned, researchers MUST motivate why they chose a certain metric or variant thereof for their particular study. Prior adoption alone does not constitute sufficient justification; researchers MUST NOT justify metric choices solely by citing their use in prior work.
Due to LLM non-determinism, researchers SHOULD repeat experiments and report descriptive statistics of model or tool performance (e.g., arithmetic mean, median, confidence intervals, standard deviations) (Agarwal et al. 2021; Bjarnason, Silva, and Monperrus 2026). If comparing models or tools, researchers MUST use appropriate inferential statistics (e.g., hypothesis tests such as the Mann-Whitney U test, McNemar’s test for binary outcomes (Kübler et al. 2026), or bootstrap-based comparisons, along with effect sizes) rather than relying solely on differences in means or other summary statistics.
The number of required repetitions depends on factors such as the study type, the variability of the task, and the desired precision of estimates. As with sample sizes for human validation ( Human ValidationG5: Human Validation ), researchers SHOULD justify their chosen number of repetitions, for example, through a power analysis (Cohen 1992; Bjarnason, Silva, and Monperrus 2026) or by monitoring the convergence of descriptive statistics across incremental runs (Blackwell, Barry, and Cohn 2024). A pilot study can help estimate the expected variability and inform this decision.
From a measurement perspective, researchers SHOULD reflect on the theories, values, and measurement models on which the benchmarks and metrics they have selected for their study are based. For example, the phenomena labeled “bugs” in a large open dataset of software bugs are according to a certain theory of what constitutes a bug, and from the values and perspectives of the people who labeled the dataset. Reflecting on the context in which these labels were assigned and discussing whether and how the labels generalize to a new study context is crucial.
Example(s)
Common Metrics:
Two common metrics used for generation tasks are BLEU-N and pass@k. BLEU-N(Papineni et al. 2002) was originally developed for evaluating machine translation quality by measuring n-gram precision between a candidate and reference text, ranging from \(0\) (dissimilar) to \(1\) (similar). It has been widely adopted in SE for code generation tasks, though its validity in this context is debatable: n-gram overlap does not capture functional correctness, and syntactically different code can be semantically equivalent (see Challenges below). Code-specific variations such as CodeBLEU(Ren et al. 2020) and CrystalBLEU(Eghbali and Pradel 2022) attempt to address these limitations by introducing additional heuristics such as AST matching.
The metric pass@k reports the likelihood that a model correctly completes a code snippet at least once within \(k\) attempts. Originally introduced as success rate at B by Kulal et al. (2019)(Kulal et al. 2019) and popularized by M. Chen et al. (2021)(M. Chen et al. 2021), it ranges from \(0\) (no correct solution in \(k\) tries) to \(1\) (at least one correct solution), with correctness typically defined by passing test cases.
The pass@k metric is defined as: \[\text{pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}\,, \text{where:}\]
\(n\) is the total number of generated samples per prompt,
\(c\) is the number of correct samples among \(n\), and
\(k\) is the number of samples drawn.
The choice of \(k\) depends on the downstream task: pass@1 is critical for single-suggestion scenarios like code completion, while higher \(k\) values (e.g., \(2\), \(5\), \(10\)) assess multi-attempt capability (Rozière et al. 2023; Guo et al. 2024; Hui et al. 2024; R. Li et al. 2023). However, pass@k is not a universal metric suitable for all generation tasks. It requires a binary notion of correctness, making the metric appropriate for code synthesis evaluated via unit tests, but unsuitable for open-ended generation tasks such as comment generation, where multiple valid outputs exist.
A complementary perspective from industry practice is passk (also written pass^k), which measures the probability that all\(k\) trials succeed rather than at least one(Anthropic 2025). While pass@k increases with \(k\), passk decreases, making it useful for assessing reliability in deployment scenarios where consistent success matters (e.g., customer-facing agents).
If a study evaluates an LLM-based tool for supporting humans, a relevant metric is the acceptance rate, meaning the ratio of all accepted artifacts (e.g., test cases, code snippets) in relation to all artifacts that were generated and presented to the user. Another way of evaluating LLM-based tools is calculating inter-model agreement.Thisallowsresearcherstoassess how dependent a tool’s performance is on specific models and versions. Metrics used to measure inter-model agreements include general agreement (percentage), Cohen’s kappa, and Krippendorff’s \(\alpha\) (see Human ValidationG5: Human Validation for recommended thresholds and best practices for measuring agreement).
Common problem types for LLM-based studies are classification, recommendation, and generation, each requiring different metrics (Hou et al. 2024). Hu et al. (2025)(Hu et al. 2025) categorized 191 LLM benchmarks by SE task, providing a valuable reference. For an overview of code foundation models, agents, and their evaluation, see Y. Yang et al. (2025)(Y. Yang et al. 2025). From a practitioner perspective, Anthropic (2025)(Anthropic 2025) categorize evaluation approaches for LLM-based agents into code-based graders (e.g., test suite execution, static analysis), model-based graders (e.g., rubric-scored LLM judgments), and human graders—a taxonomy that may help researchers systematically design evaluation strategies for agent-based tools. Common metrics include BLEU, pass@k, Accuracy@k, and Exact Match for generation; Mean Reciprocal Rank for recommendation; and Precision, Recall, F1-score, and Accuracy for classification.
… Unchanged text
Benchmark Examples:
Benchmarks used for code generation include HumanEval (available on GitHub) (M. Chen et al. 2021), MBPP (available on Hugging Face) (Austin et al. 2021), ClassEval (available on GitHub) (Du et al. 2023), LiveCodeBench (available on GitHub) (Jain et al. 2024), and SWE-bench (available on GitHub) (Jimenez et al. 2024). An example of a code translation benchmark is TransCoder(Lachaux et al. 2020) (available on GitHub).
Most benchmarks focus on generation tasks, but benchmarks for classification and recommendation also exist. For classification, DiverseVul (available on GitHub) (Y. Chen et al. 2023) provides vulnerable and non-vulnerable functions evaluated using standard classification metrics. For recommendation, CodeSearchNet (available on GitHub) (Husain et al. 2019) contains code-documentation pairs evaluated using Mean Reciprocal Rank.
Benefits
Benchmarks and metrics improve reproducibility and comparability across studies, leading to faster improvements in the cumulative body of knowledge. It is simply easier to assess a new system against one or a few benchmarks than hundreds of competing systems, and when most systems are assessed against the same benchmarks and metrics, we can see which approaches work best (at least, on those benchmarks). This comparison enables progress tracking; for example, when researchers iteratively improve a new LLM-based tool and test it against benchmarks after significant changes. For practitioners, leaderboards (published benchmark results of models) support selecting models for downstream tasks. Meanwhile, baselines help us understand just how much LLMs improve performance over non-LLM-enabled alternatives.
Challenges
Computer scientists have a long historyofusing oversimplified, unvalidated metrics (Ralph and Tempero 2018) (e.g., lines of code, CPU time) as proxies for complex, multidimensional latent variables (e.g., system size, environmental sustainability). In fields such as psychology, where measurement theory has a longer tradition (Borsboom 2005), metricsarebackedbyfoundationaltheoriesorextensiveempiricalvalidationoftheirpsychometricproperties.Thefundamentalchallengewith AI metrics and benchmarks isthatmanyhaveneitherfirmlyunderstood theoretical underpinnings norextensive empirical validation of their construct, measurement, and ecological validity.
For example, pass@k is intended to measure a model’s functional correctness, that is, its ability to generate code that produces the expected output for a given specification. However, functional correctness is only one dimension of code quality. pass@k does not capture maintainability, readability, security, or efficiency of the generated code, all of which are critical for downstream use. Furthermore, “correctness” is defined entirely by test suites, whose coverage is itself unvalidated: a solution that passes all provided tests may still be incorrect for untested inputs. Whether a test suite adequately operationalizes correctness for a given task is a subjective judgment that is rarely examined.
Similarly, HumanEval is intended to measure a model’s ability to synthesize short Python functions from docstring specifications. This operationalizes a narrow slice of “code generation ability”: it covers neither multi-file tasks, nor debugging, nor the use of existing codebases—tasks that dominate real-world software engineering (Chandra 2025; Y. Yang et al. 2025). Notably, neither pass@k nor HumanEval has undergone rigorous empirical validation of its construct validity; their widespread adoption rests on face validity and convenience rather than on evidence that they reliably measure what researchers intend them to measure (Cao et al. 2025).
Benchmarks and metrics also have generalizability problems. Prominent LLM benchmarks such as HumanEval and MBPP use Python, so researchers can optimize for Python’s idiosyncrasies, creating illusory improvements that do not generalize to other languages. Similarly, the metric BLEU-N is a syntactic metric, so code can score highly without being executable. The metric Exact Match, meanwhile, does not account for functional equivalence of syntactically different code. Both BLEU-N and Exact Match are influenced by code formatting, which confounds their intended use.
Execution-based metrics such as pass@k directly evaluate correctness by running test cases, but they require a setup with an execution environment. When researchers observe unexpected values for certain metrics, the specific results should be investigated in more detail to uncover potential problems.
Finally, benchmark data contamination—where the benchmark itself is part of the training data—may lead to artificially high performance if the model remembers the solution from the training data rather than actually solving the task based on the input data (C. Xu et al. 2024). Therefore, for proprietary LLMs that do not release their training data, researchers SHOULD consider using human validation, curating new data, or refactoring existing data (C. Xu et al. 2024).
To mitigate contamination, researchers can create new benchmark datasets by collecting data after a specified cutoff date; researchers MUST disclose data sources and collection dates for each release. However, this temporal approach requires continuous updates as new models may include the benchmark in future training data. Alternatively, keeping the benchmark private prevents inclusion in training sets, but requires trust in the benchmark creator and a system to execute it without leaking data. Researchers can also evaluate how contaminated their benchmark is (Choi et al. 2025).
… Unchanged text
Study Types
This guideline MUST be followed for all study types that automatically evaluate the performance of LLMs or LLM-based tools. The design of a benchmark and the selection of appropriate metrics are highly dependent on the specific study type and research goal. Recommending specific metrics for specific study types is beyond the scope of these guidelines, but Hu et al. (2025) provide a good overview of existing metrics for evaluating LLMs (Hu et al. 2025).
For Benchmarking LLMsS7: Benchmarking LLMs , this guideline is of primary importance: researchers MUST use established benchmarks or rigorously justify the creation of new ones, and MUST report standard metrics to enable cross-study comparison. For LLMs as AnnotatorsS1: LLMs as Annotators , the research goal might be to assess which model comes close to a ground truth dataset created by human annotators. Especially for open annotation tasks, selecting suitable metrics to compare LLM-generated and human-generated labels is important. In general, annotation tasks can vary significantly. Are multiple labels allowed for the same sequence? Are the available labels predefined, or should the LLM generate a set of labels independently? Due to this task dependence, researchers MUST justify their metric choice, explaining what aspects of the task it captures together with known limitations. For LLMs for ToolsS6: LLMs for Tools , researchers SHOULD benchmark the tool against suitable baselines using appropriate metrics that capture the tool’s intended contribution. For LLMs as JudgesS2: LLMs as Judges , researchers SHOULD report inter-rater agreement metrics and validity measures to demonstrate the reliability and quality of LLM judgments. For LLMs for SynthesisS3: LLMs for Synthesis , researchers SHOULD specify metrics for comparing synthesized outputs (e.g., coverage, faithfulness) and justify their appropriateness for the synthesis task. For Studying LLM UsageS5: Studying LLM Usage , researchers SHOULD justify the measurement instruments and metrics used for studying LLM usage patterns, including any survey scales or behavioral measures. For LLMs as SubjectsS4: LLMs as Subjects , researchers SHOULD compare simulated and real human responses using appropriate metrics to assess simulation fidelity. If researchers assess a well-established task such as code generation, they SHOULD report standard metrics such as pass@k and compare the performance between models. If non-standard metrics are used, researchers MUST state their reasoning.
Advice for Reviewers
Reviewers should expect manuscripts to:
clearly identify the constructs or variables the study aims to measure (e.g., LLM performance, quality of generated code), including independent, dependent, and control variables;
present their measurement model, i.e., which metrics, benchmarks, or baselines are used and how they relate to the target constructs;
justify the selection of any metrics, benchmarks, and baselines used;
discuss in detail the assumptions, reliability, and validity—especially construct validity—of each benchmark and metric;
articulate any limitations regarding construct and measurement validity.
As with other guidelines, missing information about baselines or metrics is typically a revision request. However, vague descriptions that conflate broad concerns (e.g., effectiveness, quality) with specific counting methods should be questioned. Ubiquity of a benchmark or metric does not imply validity or appropriateness for a given context. Manuscripts should convey a solid understanding of construct and measurement validity by explaining and justifying their measurement models.
Use Suitable Baselines, Benchmarks, and Metrics (G5)
tl;dr: Researchers MUST justify all benchmark and metric choices in the PAPER and SHOULD summarize benchmark structure, task types, and limitations. Where possible, traditional (non-LLM) baselines SHOULD be used for comparison. Researchers MUST explain in the PAPER why the selected metrics are suitable for the specific study; prior adoption in related work alone does not constitute sufficient justification. They SHOULD report established metrics to make study results comparable, but can report additional metrics that they consider appropriate. Due to the inherent non-determinism of LLMs, experiments SHOULD be repeated; the result distribution SHOULD then be reported using descriptive statistics. If comparing models or tools, researchers MUST use appropriate inferential statistics rather than relying solely on summary statistics. Researchers SHOULD justify the number of experiment repetitions, for example through a power analysis or by monitoring convergence of descriptive statistics.
Rationale
Meaningful evaluation requires well-understood, valid measurement instruments. Without justified benchmarks and metrics, claims about LLM performance lack the rigor needed for scientific comparison and cumulative progress.Abenchmark is a “standard tool for the competitive evaluation and comparison of competing systems or components according to specific characteristics, such as performance, dependability, or security” (Kistowski et al. 2015).“A metric is a method, algorithm, or procedure for assigning one or more numbers to a phenomenon” (Ralph et al. 2024). A baseline is a reference point, enabling comparison of LLMs against traditional algorithms with lower computational costs.
Recommendations
Selecting benchmarks and metrics requires understanding the benchmark tasks, what exactly is being measured, and how it relates to the (often latent) variables researchers actually care about. If one or more metrics or benchmarks are used, researchers:
MUST briefly justify (in the PAPER ) why selected benchmarks and metrics are suitable for the given task or study;
MUST discuss the reliability and validity (especially construct validity) of selected metrics and benchmarks;
SHOULD summarize the structure and tasks of the selected benchmark(s), including the programming language(s) and descriptive statistics such as the number of contained tasks and test cases;
SHOULD discuss the limitations of the selected benchmark(s) (e.g. widely used benchmarks such as HumanEval(M. Chen et al. 2021) and MBPP(Austin et al. 2021) Python functions assess a very specific part of software development, which is not representative of the full breadth of SE work (Chandra 2025)).
SHOULD include an example of a task and the corresponding test case(s) to illustrate the structure of the benchmark.
If multiple benchmarks exist for the same task, researchers SHOULD compare performance between benchmarks. When selecting only a subset of all available benchmarks, researchers SHOULD use the most specific benchmarks given the context.
Furthermore, researchers SHOULD check whether a less resource-intensive approach (e.g., for static analysis tasks or program repair) can serve as a baseline. If so, the LLM or LLM-based tool SHOULD be compared with such baselines using suitable metrics. Even if LLM-based tools outperform baselines, researchers SHOULD discuss whether the resources consumed justify the (often marginal) improvements (Menzies 2025).
To compare traditional and LLM-based approaches or different LLM-based tools, researchers SHOULD report established metrics whenever possible, as this allows secondary research. They canreport additional metrics that they consider appropriate. We briefly discuss common metrics in the Example(s) subsection below. As mentioned, researchers MUST motivate why they chose a certain metric or variant thereof for their particular study. Prior adoption alone does not constitute sufficient justification; researchers MUST NOT justify metric choices solely by citing their use in prior work.
Due to LLM non-determinism, researchers SHOULD repeat experiments and report descriptive statistics of model or tool performance (e.g., arithmetic mean, median, confidence intervals, standard deviations) (Agarwal et al. 2021; Bjarnason, Silva, and Monperrus 2026). If comparing models or tools, researchers MUST use appropriate inferential statistics (e.g., hypothesis tests such as the Mann-Whitney U test, McNemar’s test for binary outcomes (Kübler et al. 2026), or bootstrap-based comparisons, along with effect sizes) rather than relying solely on differences in means or other summary statistics.
The number of required repetitions depends on factors such as the study type, the variability of the task, and the desired precision of estimates. As with sample sizes for human validation ( Human ValidationG7: Human Validation ), researchers SHOULD justify their chosen number of repetitions, for example, through a power analysis (Cohen 1992; Bjarnason, Silva, and Monperrus 2026) or by monitoring the convergence of descriptive statistics across incremental runs (Blackwell, Barry, and Cohn 2024). A pilot study can help estimate the expected variability and inform this decision.
From a measurement perspective, researchers SHOULD reflect on the theories, values, and measurement models on which the benchmarks and metrics they have selected for their study are based. For example, the phenomena labeled “bugs” in a large open dataset of software bugs are according to a certain theory of what constitutes a bug, and from the values and perspectives of the people who labeled the dataset. Reflecting on the context in which these labels were assigned and discussing whether and how the labels generalize to a new study context is crucial.
Example(s)
Common Metrics:
Two common metrics used for generation tasks are BLEU-N and pass@k. BLEU-N(Papineni et al. 2002) was originally developed for evaluating machine translation quality by measuring n-gram precision between a candidate and reference text, ranging from \(0\) (dissimilar) to \(1\) (similar). It has been widely adopted in SE for code generation tasks, though its validity in this context is debatable: n-gram overlap does not capture functional correctness, and syntactically different code can be semantically equivalent (see Challenges below). Code-specific variations such as CodeBLEU(Ren et al. 2020) and CrystalBLEU(Eghbali and Pradel 2022) attempt to address these limitations by introducing additional heuristics such as AST matching.
The metric pass@k reports the likelihood that a model correctly completes a code snippet at least once within \(k\) attempts. Originally introduced as success rate at B by Kulal et al. (2019)(Kulal et al. 2019) and popularized by M. Chen et al. (2021)(M. Chen et al. 2021), it ranges from \(0\) (no correct solution in \(k\) tries) to \(1\) (at least one correct solution), with correctness typically defined by passing test cases.
The pass@k metric is defined as: \[\text{pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}\,, \text{where:}\]
\(n\) is the total number of generated samples per prompt,
\(c\) is the number of correct samples among \(n\), and
\(k\) is the number of samples drawn.
The choice of \(k\) depends on the downstream task: pass@1 is critical for single-suggestion scenarios like code completion, while higher \(k\) values (e.g., \(2\), \(5\), \(10\)) assess multi-attempt capability (Rozière et al. 2023; Guo et al. 2024; Hui et al. 2024; R. Li et al. 2023). However, pass@k is not a universal metric suitable for all generation tasks. It requires a binary notion of correctness, making the metric appropriate for code synthesis evaluated via unit tests, but unsuitable for open-ended generation tasks such as comment generation, where multiple valid outputs exist.
A complementary perspective from industry practice is passk (also written pass^k), which measures the probability that all\(k\) trials succeed rather than at least one(Anthropic 2025). While pass@k increases with \(k\), passk decreases, making it useful for assessing reliability in deployment scenarios where consistent success matters (e.g., customer-facing agents).
If a study evaluates an LLM-based tool for supporting humans, a relevant metric is the acceptance rate, meaning the ratio of all accepted artifacts (e.g., test cases, code snippets) in relation to all artifacts that were generated and presented to the user. Another way of evaluating LLM-based tools is calculating inter-model agreement, which reveals how much a tool’s performance depends on specific models and versions. Metrics used to measure inter-model agreements include general agreement (percentage), Cohen’s kappa, and Krippendorff’s \(\alpha\) (see Human ValidationG7: Human Validation for recommended thresholds and best practices for measuring agreement).
Common problem types for LLM-based studies are classification, recommendation, and generation, each requiring different metrics (Hou et al. 2024). Hu et al. (2025)(Hu et al. 2025) categorized 191 LLM benchmarks by SE task, providing a valuable reference. For an overview of code foundation models, agents, and their evaluation, see Y. Yang et al. (2025)(Y. Yang et al. 2025). From a practitioner perspective, Anthropic (2025)(Anthropic 2025) categorize evaluation approaches for LLM-based agents into code-based graders (e.g., test suite execution, static analysis), model-based graders (e.g., rubric-scored LLM judgments), and human graders—a taxonomy that may help researchers systematically design evaluation strategies for agent-based tools. Common metrics include BLEU, pass@k, Accuracy@k, and Exact Match for generation; Mean Reciprocal Rank for recommendation; and Precision, Recall, F1-score, and Accuracy for classification.
… Unchanged text
Benchmark Examples:
Benchmarks used for code generation include HumanEval (available on GitHub) (M. Chen et al. 2021), MBPP (available on Hugging Face) (Austin et al. 2021), ClassEval (available on GitHub) (Du et al. 2023), LiveCodeBench (available on GitHub) (Jain et al. 2024), and SWE-bench (available on GitHub) (Jimenez et al. 2024). An example of a code translation benchmark is TransCoder(Lachaux et al. 2020) (available on GitHub).
Most benchmarks focus on generation tasks, but benchmarks for classification and recommendation also exist. For classification, DiverseVul (available on GitHub) (Y. Chen et al. 2023) provides vulnerable and non-vulnerable functions evaluated using standard classification metrics. For recommendation, CodeSearchNet (available on GitHub) (Husain et al. 2019) contains code-documentation pairs evaluated using Mean Reciprocal Rank.
Benefits
Benchmarks and metrics improve reproducibility and comparability across studies, leading to faster improvements in the cumulative body of knowledge. It is simply easier to assess a new system against one or a few benchmarks than hundreds of competing systems, and when most systems are assessed against the same benchmarks and metrics, we can see which approaches work best (at least, on those benchmarks). This comparison enables progress tracking; for example, when researchers iteratively improve a new LLM-based tool and test it against benchmarks after significant changes. For practitioners, leaderboards (published benchmark results of models) support selecting models for downstream tasks. Meanwhile, baselines help us understand just how much LLMs improve performance over non-LLM-enabled alternatives.
Challenges
Computer scientists havelong used oversimplified, unvalidated metrics (Ralph and Tempero 2018) (e.g., lines of code, CPU time) as proxies for complex, multidimensional latent variables (e.g., system size, environmental sustainability). Unlike fields such as psychologywhere measurement theory has a longer tradition (Borsboom 2005), many AI metrics and benchmarks lack both theoretical underpinnings and empirical validation of their construct, measurement, and ecological validity.
For example, pass@k is intended to measure a model’s functional correctness, that is, its ability to generate code that produces the expected output for a given specification. However, functional correctness is only one dimension of code quality. pass@k does not capture maintainability, readability, security, or efficiency of the generated code, all of which are critical for downstream use. Furthermore, “correctness” is defined entirely by test suites, whose coverage is itself unvalidated: a solution that passes all provided tests may still be incorrect for untested inputs. Whether a test suite adequately operationalizes correctness for a given task is a subjective judgment that is rarely examined.
Similarly, HumanEval is intended to measure a model’s ability to synthesize short Python functions from docstring specifications. This operationalizes a narrow slice of “code generation ability”: it covers neither multi-file tasks, nor debugging, nor the use of existing codebases—tasks that dominate real-world software engineering (Chandra 2025; Y. Yang et al. 2025). Notably, neither pass@k nor HumanEval has undergone rigorous empirical validation of its construct validity; their widespread adoption rests on face validity and convenience rather than on evidence that they reliably measure what researchers intend them to measure (Cao et al. 2025).
Benchmarks and metrics also have generalizability problems. Prominent LLM benchmarks such as HumanEval and MBPP use Python, so researchers can optimize for Python’s idiosyncrasies, creating illusory improvements that do not generalize to other languages. Similarly, the metric BLEU-N is a syntactic metric, so code can score highly without being executable. The metric Exact Match, meanwhile, does not account for functional equivalence of syntactically different code. Both BLEU-N and Exact Match are influenced by code formatting, which confounds their intended use.
Execution-based metrics such as pass@k directly evaluate correctness by running test cases, but they require a setup with an execution environment. When researchers observe unexpected values for certain metrics, the specific results should be investigated in more detail to uncover potential problems.
Finally, benchmark data contamination—where the benchmark itself is part of the training data—may lead to artificially high performance if the model remembers the solution from the training data rather than actually solving the task based on the input data (C. Xu et al. 2024). Therefore, for proprietary LLMs that do not release their training data, researchers SHOULD consider using human validation, curating new data, or refactoring existing data (C. Xu et al. 2024).
To mitigate contamination, researchers can create new benchmark datasets by collecting data after a specified cutoff date; researchers MUST disclose data sources and collection dates for each release. However, this temporal approach requires continuous updates as new models may include the benchmark in future training data. Alternatively, keeping the benchmark private prevents inclusion in training sets, but requires trust in the benchmark creator and a system to execute it without leaking data. Researchers can also evaluate how contaminated their benchmark is (Choi et al. 2025).
… Unchanged text
Study Types
This guideline MUST be followed for all study types that automatically evaluate the performance of LLMs or LLM-based tools. The design of a benchmark and the selection of appropriate metrics are highly dependent on the specific study type and research goal. Recommending specific metrics for specific study types is beyond the scope of these guidelines, but Hu et al. (2025) provide a good overview of existing metrics for evaluating LLMs (Hu et al. 2025).
For Benchmarking LLMsS7: Benchmarking LLMs , this guideline is of primary importance: researchers MUST use established benchmarks or rigorously justify the creation of new ones, and MUST report standard metrics to enable cross-study comparison. For LLMs as AnnotatorsS1: LLMs as Annotators , the research goal might be to assess which model comes close to a ground truth dataset created by human annotators. Especially for open annotation tasks, selecting suitable metrics to compare LLM-generated and human-generated labels is important. In general, annotation tasks can vary significantly. Are multiple labels allowed for the same sequence? Are the available labels predefined, or should the LLM generate a set of labels independently? Due to this task dependence, researchers MUST justify their metric choice, explaining what aspects of the task it captures together with known limitations. For LLMs for ToolsS6: LLMs for Tools , researchers SHOULD benchmark the tool against suitable baselines using appropriate metrics that capture the tool’s intended contribution. For LLMs as JudgesS2: LLMs as Judges , researchers SHOULD report inter-rater agreement metrics and validity measures to demonstrate the reliability and quality of LLM judgments. For LLMs for SynthesisS3: LLMs for Synthesis , researchers SHOULD specify metrics for comparing synthesized outputs (e.g., coverage, faithfulness) and justify their appropriateness for the synthesis task. For Studying LLM UsageS5: Studying LLM Usage , researchers SHOULD justify the measurement instruments and metrics used for studying LLM usage patterns, including any survey scales or behavioral measures. For LLMs as SubjectsS4: LLMs as Subjects , researchers SHOULD compare simulated and real human responses using appropriate metrics to assess simulation fidelity. If researchers assess a well-established task such as code generation, they SHOULD report standard metrics such as pass@k and compare the performance between models. If non-standard metrics are used, researchers MUST state their reasoning.
Advice for Reviewers
Reviewers should expect manuscripts to:
clearly identify the constructs or variables the study aims to measure (e.g., LLM performance, quality of generated code), including independent, dependent, and control variables;
present their measurement model, i.e., which metrics, benchmarks, or baselines are used and how they relate to the target constructs;
justify the selection of any metrics, benchmarks, and baselines used;
discuss in detail the assumptions, reliability, and validity—especially construct validity—of each benchmark and metric;
articulate any limitations regarding construct and measurement validity.
As with other guidelines, missing information about baselines or metrics is typically a revision request. However, vague descriptions that conflate broad concerns (e.g., effectiveness, quality) with specific counting methods should be questioned. Ubiquity of a benchmark or metric does not imply validity or appropriateness for a given context. Manuscripts should convey a solid understanding of construct and measurement validity by explaining and justifying their measurement models.
tl;dr: Researchers SHOULD include an open LLM as a baseline when using commercial models and report inter-model agreement. By open LLM, we mean a model that everyone with the required hardware can deploy and operate. Such models are usually “open weight.” A full replication package with step-by-step instructions SHOULD be provided as part of the SUPPLEMENTARY MATERIAL.
… Unchanged text
Rationale
Reproducibility depends on access to the model under study. When research relies exclusively on proprietary models, other researchers cannot independently verify or build upon the findings. Including an open LLM as a baseline ensures that at least part of the study can be fully replicated.
Recommendations
Empirical studies using LLMs in SE, especially those that target commercial tools or models, SHOULD incorporate an open LLM as a baseline and report established metrics for inter-model agreement. We acknowledge that including an open LLM baseline might not always be possible, for example, if the study involves human participants, and letting them work on the tasks using two different models might not be feasible. Using an open model as a baseline is also not necessary if the use of the LLM is tangential to the study goal.
Open models allow other researchers to verify research results and build upon them, even without access to commercial models. A comparison of commercial and open models also allows researchers to contextualize model performance. Researchers SHOULD provide a complete replication package as part of their SUPPLEMENTARY MATERIAL, including clear step-by-step instructions on how to verify and reproduce the results reported in the paper.
Open LLMs are available on platforms such as Hugging Face and can be hosted locally using frameworks such as Ollama or LM Studio, or on cloud services such as Together AI, AWS, Azure, and Google Cloud.
The term “open” can have different meanings in the context of LLMs. Widder, Whittaker, and West (2024) discuss three types of openness (transparency, reusability, and extensibility) and what openness can and cannot provide (Widder, Whittaker, and West 2024). The Open Source Initiative (OSI) (Open Source Initiative (OSI) 2025) defines open-source AI as having access to everything needed to understand, modify, share, retrain, and recreate the model.
Researchers can also explore open-source tools such as Continue, Cline, and opencode as alternatives to commercial tools like GitHub Copilot and Claude Code, enabling instrumentation, architectural transparency, and detailed telemetry collection.
Example(s)
An increasing number of studies have adopted open LLMs as baseline models. For example, C. Wang et al. (2024) evaluated seven advanced LLMs, six of which were open-source, testing 145 API mappings drawn from eight popular Python libraries across 28,125 completion prompts aimed at detecting deprecated API usage in code completion (C. Wang et al. 2024). Moumoula et al. (2024) compared four LLMs on a cross-language code clone detection task (Moumoula et al. 2024). Three evaluated models were open-source. Gonçalves et al. (2025) fine-tuned the open LLM LLaMA 3.2 on a refined version of the DiverseVul dataset to benchmark vulnerability detection performance (Gonçalves et al. 2025). CodeBERT, a bimodal transformer pre-trained by Microsoft Research, has been widely used as an open baseline (G. Yang et al. 2023; Xia, Shao, and Deng 2024; Sonnekalb et al. 2022; Cai et al. 2024), with model weights, source code, and data-processing scripts published on GitHub (Microsoft 2023).
Benefits
Usinga true open LLM asa baseline improves the reproducibility ofscientificresearch by providingaccessto model architectures and parameter settings, and ideally training data, therebyallowing independent reconstructionand verification of experimental results. Moreover, by adopting an open-source baseline, researchers can directly compare novel methods against established performance metrics without the variability introduced by proprietary systems. Thetransparentnatureofthese models allowsfor detailed inspection of data processing pipelines and decision-making routines, whichisessentialforidentifyingpotentialsourcesofbias and delineating model limitations. Furthermore, unlike closed-source alternatives, which can be withdrawn or altered without notice, open LLMs ensure long-term accessibility and stability, preserving critical resources for future studies. Finally, the licensing requirements associated with open-source implementations lower financial barriers, making advanced language models attainable for research groups operating under constrained budgets.
… Unchanged text
Challenges
Open-source LLMs face several notable challenges. First, they often lag behind the most advanced proprietary “frontier” models in common benchmarks, making it difficult for researchers to demonstrate clear improvements when evaluating new methods using open LLMs alone. Additionally, deploying and experimenting with these models typically requires substantial hardware resources, in particular high-performance GPUs, which may be beyond reach for many academic groups. The notion of “openness” remains in flux: many models release only trained weights without training data or methodological details (“open weight” openness) (Gibney 2024), which is why we reference the OSI definition in our recommendations (Open Source Initiative (OSI) 2025). Finally, unlike APIs provided by proprietary vendors (e.g., the OpenAI API), installing, configuring, and fine-tuning open-source models can be technically demanding. Documentation is often sparse or fragmented, placing a high barrier to entry for researchers without specialized engineering support.
Study Types
This guideline applies primarily to study types in which the researcher controls which LLM is used. In formal benchmarking studies and controlled experiments (see Benchmarking LLMsS7: Benchmarking LLMs ), an open LLM MUST be one of the models under evaluation. When evaluating LLMs for ToolsS6: LLMs for Tools , researchers SHOULD use an open LLM as a baseline whenever it is technically feasible; if integration proves too complex, they SHOULD report the initial benchmarking results of open models. For LLMs as AnnotatorsS1: LLMs as Annotators and LLMs as JudgesS2: LLMs as Judges , researchers SHOULD compare annotation or judgment quality from open vs. commercial models to assess the extent to which results depend on a specific proprietary model. For LLMs for SynthesisS3: LLMs for Synthesis , researchers SHOULD compare synthesis results from open and commercial models to evaluate robustness of the findings. For Studying LLM UsageS5: Studying LLM Usage , using an open LLM as a baseline is often not feasible when the study observes participants using specific commercial tools; in such cases, investigators SHOULD explicitly acknowledge its absence and discuss how this limitation might affect their conclusions. For LLMs as SubjectsS4: LLMs as Subjects , using an open LLM as a baseline may similarly be impractical if the study design requires the capabilities of a specific model; researchers SHOULD acknowledge this limitation when applicable.
Advice for Reviewers
Reviewers should distinguish between LLM use that is central to the research (e.g., building LLM-driven SE tools, using LLMs for synthesis) and use that is tangential (e.g., generating recruiting materials). An open LLM baseline is expected only when LLM use is central; otherwise, its absence need not be justified. When an open baseline is expected, reviewers should look for either the use of an open LLM or a convincing argument for why it is impractical. If authors claim openness, some justification for that characterization is appropriate. Where practical, reviewers should examine whether the replication package contains sufficiently detailed instructions. Reviewers should not penalize studies for performance differences between open and proprietary models, as this is beyond the authors’ control.
tl;dr: Researchers SHOULD include an open LLM as a baseline when using commercial models and report inter-model agreement. By open LLM, we mean a model that everyone with the required hardware can deploy and operate. Such models are usually “open weight.” A full replication package with step-by-step instructions SHOULD be provided as part of the SUPPLEMENTARY MATERIAL.
… Unchanged text
Rationale
Reproducibility depends on access to the model under study. When research relies exclusively on proprietary models, other researchers cannot independently verify or build upon the findings. Including an open LLM as a baseline ensures that at least part of the study can be fully replicated.
Recommendations
Empirical studies using LLMs in SE, especially those that target commercial tools or models, SHOULD incorporate an open LLM as a baseline and report established metrics for inter-model agreement. We acknowledge that including an open LLM baseline might not always be possible, for example, if the study involves human participants, and letting them work on the tasks using two different models might not be feasible. Using an open model as a baseline is also not necessary if the use of the LLM is tangential to the study goal.
Open models allow other researchers to verify research results and build upon them, even without access to commercial models. A comparison of commercial and open models also allows researchers to contextualize model performance. Researchers SHOULD provide a complete replication package as part of their SUPPLEMENTARY MATERIAL, including clear step-by-step instructions on how to verify and reproduce the results reported in the paper.
Open LLMs are available on platforms such as Hugging Face and can be hosted locally using frameworks such as Ollama or LM Studio, or on cloud services such as Together AI, AWS, Azure, and Google Cloud.
The term “open” can have different meanings in the context of LLMs. Widder, Whittaker, and West (2024) discuss three types of openness (transparency, reusability, and extensibility) and what openness can and cannot provide (Widder, Whittaker, and West 2024). The Open Source Initiative (OSI) (Open Source Initiative (OSI) 2025) defines open-source AI as having access to everything needed to understand, modify, share, retrain, and recreate the model.
Researchers can also explore open-source tools such as Continue, Cline, and opencode as alternatives to commercial tools like GitHub Copilot and Claude Code, enabling instrumentation, architectural transparency, and detailed telemetry collection.
Example(s)
An increasing number of studies have adopted open LLMs as baseline models. For example, C. Wang et al. (2024) evaluated seven advanced LLMs, six of which were open-source, testing 145 API mappings drawn from eight popular Python libraries across 28,125 completion prompts aimed at detecting deprecated API usage in code completion (C. Wang et al. 2024). Moumoula et al. (2024) compared four LLMs on a cross-language code clone detection task (Moumoula et al. 2024). Three evaluated models were open-source. Gonçalves et al. (2025) fine-tuned the open LLM LLaMA 3.2 on a refined version of the DiverseVul dataset to benchmark vulnerability detection performance (Gonçalves et al. 2025). CodeBERT, a bimodal transformer pre-trained by Microsoft Research, has been widely used as an open baseline (G. Yang et al. 2023; Xia, Shao, and Deng 2024; Sonnekalb et al. 2022; Cai et al. 2024), with model weights, source code, and data-processing scripts published on GitHub (Microsoft 2023).
Benefits
A true open LLMbaselineimprovesreproducibilityby exposing model architectures, parameter settings, and ideally training data, enabling independentverificationofresults. Moreover, by adopting an open-source baseline, researchers can directly compare novel methods against established performance metrics without the variability introduced by proprietary systems. These models allow detailed inspection of data processing pipelines and decision-making routines, helping identify biases andmodel limitations. Furthermore, unlike closed-source alternatives, which can be withdrawn or altered without notice, open LLMs ensure long-term accessibility and stability, preserving critical resources for future studies. Finally, the licensing requirements associated with open-source implementations lower financial barriers, making advanced language models attainable for research groups operating under constrained budgets.
… Unchanged text
Challenges
Open-source LLMs face several notable challenges. First, they often lag behind the most advanced proprietary “frontier” models in common benchmarks, making it difficult for researchers to demonstrate clear improvements when evaluating new methods using open LLMs alone. Additionally, deploying and experimenting with these models typically requires substantial hardware resources, in particular high-performance GPUs, which may be beyond reach for many academic groups. The notion of “openness” remains in flux: many models release only trained weights without training data or methodological details (“open weight” openness) (Gibney 2024), which is why we reference the OSI definition in our recommendations (Open Source Initiative (OSI) 2025). Finally, unlike APIs provided by proprietary vendors (e.g., the OpenAI API), installing, configuring, and fine-tuning open-source models can be technically demanding. Documentation is often sparse or fragmented, placing a high barrier to entry for researchers without specialized engineering support.
Study Types
This guideline applies primarily to study types in which the researcher controls which LLM is used. In formal benchmarking studies and controlled experiments (see Benchmarking LLMsS7: Benchmarking LLMs ), an open LLM MUST be one of the models under evaluation. When evaluating LLMs for ToolsS6: LLMs for Tools , researchers SHOULD use an open LLM as a baseline whenever it is technically feasible; if integration proves too complex, they SHOULD report the initial benchmarking results of open models. For LLMs as AnnotatorsS1: LLMs as Annotators and LLMs as JudgesS2: LLMs as Judges , researchers SHOULD compare annotation or judgment quality from open vs. commercial models to assess the extent to which results depend on a specific proprietary model. For LLMs for SynthesisS3: LLMs for Synthesis , researchers SHOULD compare synthesis results from open and commercial models to evaluate robustness of the findings. For Studying LLM UsageS5: Studying LLM Usage , using an open LLM as a baseline is often not feasible when the study observes participants using specific commercial tools; in such cases, investigators SHOULD explicitly acknowledge its absence and discuss how this limitation might affect their conclusions. For LLMs as SubjectsS4: LLMs as Subjects , using an open LLM as a baseline may similarly be impractical if the study design requires the capabilities of a specific model; researchers SHOULD acknowledge this limitation when applicable.
Advice for Reviewers
Reviewers should distinguish between LLM use that is central to the research (e.g., building LLM-driven SE tools, using LLMs for synthesis) and use that is tangential (e.g., generating recruiting materials). An open LLM baseline is expected only when LLM use is central; otherwise, its absence need not be justified. When an open baseline is expected, reviewers should look for either the use of an open LLM or a convincing argument for why it is impractical. If authors claim openness, some justification for that characterization is appropriate. Where practical, reviewers should examine whether the replication package contains sufficiently detailed instructions. Reviewers should not penalize studies for performance differences between open and proprietary models, as this is beyond the authors’ control.
tl;dr: If assessing the quality of generated artifacts is important and no reference datasets or suitable comparison metrics exist, researchers SHOULD use human validation for LLM outputs. If they do, they MUST define the measured construct (e.g., usability, maintainability) and describe the measurement instrument in the PAPER. Researchers SHOULD consider human validation early in the study design (not as an afterthought) and build on established reference models for human-LLM comparison. When developing or adapting measurement instruments, researchers MUST share them. When aggregating LLM judgments, methods and rationale SHOULD be reported and inter-rater agreement SHOULD be assessed. Confounding factors SHOULD be controlled for, and power analysis SHOULD be conducted to ensure statistical robustness.
Rationale
Automated metrics alonecannotfullycapture the validityofsubjectiveorcomplexconstructs.WhenLLMsareusedtoautomatetasks that involve human judgment,theiroutputsmustbevalidatedagainsthumanassessmentstoensurethattheresultsarenotonlyreliablebutalsovalid.
Recommendations
When LLMs areusedto automate research or software development tasks previously performed by humans, itisessentialtoassess the LLM’s performance.Whenexistingreferencedatasetsorcomparisonmetricscannotfullycapture the target qualitiesrelevantinthestudycontext, researchers SHOULDrelyon human judgment tovalidatetheLLMoutputs.
Study Design Considerations:
Integrating human participants in the study design will require additional considerations, including a recruitment strategy, annotation guidelines, training sessions, or ethical approvals. Therefore, researchers SHOULD consider human validation early in the study design, not as an afterthought. Authors MUST clearly define in the PAPER the constructs that the human and LLM annotators evaluate (Ralph and Tempero 2018). When designing custom instruments to assess LLM output (e.g., questionnaires, scales), researchers MUST share their instruments.
Subjective Judgment and Agreement:
When the judgment is subjective (i.e., depends on the judge’s values or theories), the LLM output SHOULD be evaluated against judgments aggregated from multiple human judges, and researchers SHOULD clearly describe their aggregation method and reasoning. Thepreferredmethod is to randomly order the objects requiring judgment and put them in groups small enough to judge in one to three hours. Two or three human experts review the first group together, simultaneously judging, discussing, and creating a set of decision rules documenting their reasoning. Then, the experts iterate among rounds of:
independently rating one group of objects,
calculating inter-rater agreement (IRA) or reliability (IRR),
meeting to discuss disagreements and reach consensus,
updating the decision rules so the disagreement will not recur.
The goal is to reach a target IRA or IRR (e.g., Krippendorff’s \(\alpha>0.8\)), indicating sufficient decision rules. Once this target is reached and sustained for two to three groups, it is permissible to continue with a single rater. Researchers SHOULD report measures of IRA or IRR, preferably broken down by round.
Confounding factors SHOULD be discussed and, where feasible, controlled for (e.g., by categorizing participants according to their level of experience or expertise). Where applicable, researchers may perform a power analysis (Cohen 1992; Dybå, Kampenes, and Sjøberg 2006) to estimate the required sample size, ensuring sufficient statistical power in their experimental design. However, we also note that to our knowledge, there is limited established guidance specific to determining sample sizes for LLM-human comparisons. In other fields, 100 comparisons are often made without further explanation (Tam et al. 2024).
Researchers SHOULD use established reference models to compare humans with LLMs. For example, Schneider, Fotrousi, and Wohlrab (2025)(Schneider, Fotrousi, and Wohlrab 2025) outline design considerations for studies comparing LLMs with humans. If studies aim to automate annotating software artifacts using an LLM, researchers SHOULD follow systematic approaches to decide whether and how human annotators can be replaced (e.g., showing that a jury of three LLMs exhibit model-to-model agreement comparable to trained human annotators, such as Krippendorff’s \(\alpha>0.8\)). Model-to-model agreement shouldnotbeheldtoalowerstandardthan human-to-human agreement, as low thresholds may simply reflect shared model biases rather than annotation quality.
Agentic Tools
Besides generating and modifying content, agentic software development tools such as Claude Code can autonomously call command-line tools or pull in additional information from MCP servers. For such tools, it makes sense to separate textual output (e.g., file changes) from output related to tool execution. When evaluating agentic tools, researchers SHOULD assess the feedback that users provided for proposed changes, report statistics on how frequently they accepted the content, and how they modified it. Thisagentic human-in-the-loop interaction approach is essentially a built-in human validation, eventhoughthe degrees of freedom arelarger than inmore traditional experiments that use LLMs directly.
… Unchanged text
Example(s)
T. Ahmed et al. (2025)(T. Ahmed et al. 2025) proposed a systematic method for deciding whether LLMs can replace human annotators on a given task, using model-to-model agreement as an initial screening criterion (with a threshold of \(\alpha>0.5\)) and model confidence for sample-level decisions. However, their threshold is well below the levels generally considered acceptable for inter-rater agreement. Krippendorff (2018) recommends discarding data with \(\alpha<0.667\), considers \(0.667 \leq \alpha < 0.8\) sufficient only for tentative conclusions, and requires \(\alpha \geq 0.8\) for reliable data (Krippendorff 2018). Researchers adopting similar screening approaches should apply thresholds consistent with these established standards. Notably, while three LLMs exhibited higher inter-model agreement than human annotators on some tasks, human-model agreement remained low on others. This illustrates that high model-model reliability does not guarantee alignment with human judgment, reinforcing the need for human validation before assuming that LLM annotations are valid. Moreover, a high model-model agreement could merely indicate that the models share systematic biases and hence reliably agree on the wrong answer.
Hymel and Johnson (2025)(Hymel and Johnson 2025) evaluated ChatGPT’s capability to generate requirements documents by comparing an LLM-generated and a human-generated document based on the same business use case. Domain experts reviewed both documents and attempted to distinguish their origin.
Benefits
Validating LLMs against human judgments builds confidence in the LLM’s accuracy and validity, increasing the trustworthiness of findings, especially when IRA or IRR metrics are reported (Khraisha et al. 2024). Furthermore, incorporating feedback from human judges may help to improve the LLM or LLM-based tool based on the reported experiences.
Challenges
Assessing a construct like LLM performance is challenging because ensuring that a construct is defined well and operationalized using an appropriate measurement model requires a deep understanding of (1) the construct, (2) construct validity in general, and (3) instrumentation (Sjøberg and Bergersen 2023; Ralph et al. 2024). Comparing an LLM to human judges is typically slower and more expensive than machine-generated measures. More fundamentally, neither human judgment nor machine-generated measures provides an objective ground truth against which LLM accuracy can be firmly determined.
Human judgments exhibit variability due to differences in experience, expertise, interpretations, and personal biases (McDonald, Schoenebeck, and Forte 2019). When diverse humans rate items reliably given clear decision rules, we assume that reliability implies validity, but it does not. Measuring reliability is much easier than measuring validity, and often the best researchers can do is argue conceptually for why their judges, decision rules, and constructs should produce valid ratings.
Researchers must weigh the cost and time need for human validation against the expected corresponding improvement in validity over machine-generated measures.
Study Types
This guideline applies to all study types, although the need for human validation varies. For LLMs as AnnotatorsS1: LLMs as Annotators , researchers SHOULD validate LLM-generated annotations against human annotators to assess labeling quality and identify systematic biases. When using LLMs as JudgesS2: LLMs as Judges , researchers SHOULD co-create initial rating criteria with humans and validate a sample of LLM judgments against human expert assessments. For LLMs for SynthesisS3: LLMs for Synthesis , researchers SHOULD employ human oversight to verify that qualitative interpretations and synthesized outputs faithfully represent the underlying data. For LLMs as SubjectsS4: LLMs as Subjects , researchers SHOULD validate simulated responses against real human data to assess the fidelity of the simulation. For Studying LLM UsageS5: Studying LLM Usage , researchers SHOULD carefully reflect on the validity of their evaluation criteria and validate subjective assessments with human experts. IndevelopingLLMs for ToolsS6: LLMs for Tools,humanvalidationoftheLLMoutputiscriticalwhenthe output is supposed to match human expectations. For Benchmarking LLMsS7: Benchmarking LLMs , there is less need for human validation when using extensively validated and widely-used benchmarks, but researchers SHOULD employ human validation when creating or adapting new benchmarks.
… Unchanged text
Advice for Reviewers
Human validation may be the most challenging of our guidelines to assess because it often requires evaluating conceptual arguments. If LLM output is validated only by comparison with other LLMs, reviewers should look for quantitative empirical evidence that such comparison is reliable and valid. High inter-model agreement alone is insufficient, as reliability does not imply validity. Similarly, reviewers should expect evidence that any employed benchmarks are reliable and valid. Absent such evidence, human validation is warranted. A single human judge is appropriate only when judgments depend on widely accepted theories and involve limited value conflict (e.g., tagging method names containing abbreviations). For multiple judges, reviewers should expect IRA/IRR improvement techniques as described in the recommendations above (experienced raters, organized rounds, consensus meetings, updated decision rules). Low IRA or IRR (e.g., Krippendorff’s \(\alpha<0.8\)) without these techniques is a concern. Conversely, if authors have followed best practices and still obtained mediocre results (e.g., \(0.66<\alpha<0.8\)), this should be noted as a limitation. Beyond reliability, reviewers should expect authors to explain conceptually why their human judgments should be valid, considering construct definitions, decision rules, and judge expertise. As with other guidelines, missing information is typically a revision request. Absent judge instructions, instruments, decision rules, or construct definitions may prevent assessment of rigor and validity, while missing recruitment details are less critical. Clarification requests about construct definitions are routine and should not alone warrant rejection.
2026.04
Use Human Validation for LLM Outputs (G7)
tl;dr: If assessing the quality of generated artifacts is important and no reference datasets or suitable comparison metrics exist, researchers SHOULD use human validation for LLM outputs. If they do, they MUST define the measured construct (e.g., usability, maintainability) and describe the measurement instrument in the PAPER. Researchers SHOULD consider human validation early in the study design (not as an afterthought) and build on established reference models for human-LLM comparison. When developing or adapting measurement instruments, researchers MUST share them. When aggregating LLM judgments, methods and rationale SHOULD be reported and inter-rater agreement SHOULD be assessed. Confounding factors SHOULD be controlled for, and power analysis SHOULD be conducted to ensure statistical robustness.
Rationale
Even with well-justified benchmarks and metrics ( Benchmarks and MetricsG5: Benchmarks and Metrics ), automated measurement captures only theconstructs the benchmark was designed to measure. Subjective constructs and constructs that no current benchmark covers require human assessment.
Recommendations
When LLMsautomate research or software development tasks previously performed by humans,the LLM’s performance needs to be assessed. Where no benchmark adequately operationalizes the target construct, researchers SHOULD validate LLM outputs against human judgment.
Study Design Considerations:
Integrating human participants in the study design will require additional considerations, including a recruitment strategy, annotation guidelines, training sessions, or ethical approvals. Therefore, researchers SHOULD consider human validation early in the study design, not as an afterthought. Authors MUST clearly define in the PAPER the constructs that the human and LLM annotators evaluate (Ralph and Tempero 2018). When designing custom instruments to assess LLM output (e.g., questionnaires, scales), researchers MUST share their instruments.
Subjective Judgment and Agreement:
When the judgment is subjective (i.e., depends on the judge’s values or theories), the LLM output SHOULD be evaluated against judgments aggregated from multiple human judges, and researchers SHOULD clearly describe their aggregation method and reasoning. One systematic approach is to randomly order the objects requiring judgment and put them in groups small enough to judge in one to three hours. Two or three human experts review the first group together, simultaneously judging, discussing, and creating a set of decision rules documenting their reasoning. Then, the experts iterate among rounds of:
independently rating one group of objects,
calculating inter-rater agreement (IRA) or reliability (IRR),
meeting to discuss disagreements and reach consensus,
updating the decision rules so the disagreement will not recur.
The goal is to reach a target IRA or IRR (e.g., Krippendorff’s \(\alpha>0.8\)), indicating sufficient decision rules. Once this target is reached and sustained for two to three groups, it is permissible to continue with a single rater. Researchers SHOULD report measures of IRA or IRR, preferably broken down by round.
Confounding factors SHOULD be discussed and, where feasible, controlled for (e.g., by categorizing participants according to their level of experience or expertise). Where applicable, researchers may perform a power analysis (Cohen 1992; Dybå, Kampenes, and Sjøberg 2006) to estimate the required sample size, ensuring sufficient statistical power in their experimental design. However, we also note that to our knowledge, there is limited established guidance specific to determining sample sizes for LLM-human comparisons. In other fields, 100 comparisons are often made without further explanation (Tam et al. 2024).
Researchers SHOULD use established reference models to compare humans with LLMs. For example, Schneider, Fotrousi, and Wohlrab (2025)(Schneider, Fotrousi, and Wohlrab 2025) outline design considerations for studies comparing LLMs with humans. If studies aim to automate annotating software artifacts using an LLM, researchers SHOULD follow systematic approaches to decide whether and how human annotators can be replaced (e.g., showing that a jury of three LLMs exhibit model-to-model agreement comparable to trained human annotators, such as Krippendorff’s \(\alpha>0.8\)). Model-to-model agreement requires the same threshold as human-to-human agreement, since low thresholds may simply reflect shared model biases rather than annotation quality.
Agentic Tools:
Besides generating and modifying content, agentic software development tools such as Claude Code can autonomously call command-line tools or pull in additional information from MCP servers. For such tools, it makes sense to separate textual output (e.g., file changes) from output related to tool execution. When evaluating agentic tools, researchers SHOULD assess the feedback that users provided for proposed changes, report statistics on how frequently they accepted the content, and how they modified it. Agentic human-in-the-loop interactionisa built-in human validation, with larger degrees of freedomthantraditional experiments that use LLMs directly.
… Unchanged text
Example(s)
T. Ahmed et al. (2025)(T. Ahmed et al. 2025) proposed a systematic method for deciding whether LLMs can replace human annotators on a given task, using model-to-model agreement as an initial screening criterion (with a threshold of \(\alpha>0.5\)) and model confidence for sample-level decisions. However, their threshold is well below the levels generally considered acceptable for inter-rater agreement. Krippendorff (2018) recommends discarding data with \(\alpha<0.667\), considers \(0.667 \leq \alpha < 0.8\) sufficient only for tentative conclusions, and requires \(\alpha \geq 0.8\) for reliable data (Krippendorff 2018). Researchers adopting similar screening approaches should apply thresholds consistent with these established standards. Notably, while three LLMs exhibited higher inter-model agreement than human annotators on some tasks, human-model agreement remained low on others. This illustrates that high model-model reliability does not guarantee alignment with human judgment, reinforcing the need for human validation before assuming that LLM annotations are valid. Moreover, a high model-model agreement could merely indicate that the models share systematic biases and hence reliably agree on the wrong answer.
Hymel and Johnson (2025)(Hymel and Johnson 2025) evaluated ChatGPT’s capability to generate requirements documents by comparing an LLM-generated and a human-generated document based on the same business use case. Domain experts reviewed both documents and attempted to distinguish their origin.
Benefits
Validating LLMs against human judgments builds confidence in the LLM’s accuracy and validity, increasing the trustworthiness of findings, especially when IRA or IRR metrics are reported (Khraisha et al. 2024). Furthermore, incorporating feedback from human judges may help to improve the LLM or LLM-based tool based on the reported experiences.
Challenges
Assessing a construct like LLM performance is challenging because ensuring that a construct is defined well and operationalized using an appropriate measurement model requires a deep understanding of (1) the construct, (2) construct validity in general, and (3) instrumentation (Sjøberg and Bergersen 2023; Ralph et al. 2024). Comparing an LLM to human judges is typically slower and more expensive than machine-generated measures. More fundamentally, neither human judgment nor machine-generated measures provides an objective ground truth against which LLM accuracy can be firmly determined.
Human judgments exhibit variability due to differences in experience, expertise, interpretations, and personal biases (McDonald, Schoenebeck, and Forte 2019). When diverse humans rate items reliably given clear decision rules, we assume that reliability implies validity, but it does not. Measuring reliability is much easier than measuring validity, and often the best researchers can do is argue conceptually for why their judges, decision rules, and constructs should produce valid ratings.
Researchers must weigh the cost and time need for human validation against the expected corresponding improvement in validity over machine-generated measures.
Study Types
This guideline applies to all study types, although the need for human validation varies. For LLMs as AnnotatorsS1: LLMs as Annotators , researchers SHOULD validate LLM-generated annotations against human annotators to assess labeling quality and identify systematic biases. When using LLMs as JudgesS2: LLMs as Judges , researchers SHOULD co-create initial rating criteria with humans and validate a sample of LLM judgments against human expert assessments. For LLMs for SynthesisS3: LLMs for Synthesis , researchers SHOULD employ human oversight to verify that qualitative interpretations and synthesized outputs faithfully represent the underlying data. For LLMs as SubjectsS4: LLMs as Subjects , researchers SHOULD validate simulated responses against real human data to assess the fidelity of the simulation. For Studying LLM UsageS5: Studying LLM Usage , researchers SHOULD carefully reflect on the validity of their evaluation criteria and validate subjective assessments with human experts. For LLMs for ToolsS6: LLMs for Tools whose output is supposed to match human expectations, researchers SHOULD validate the LLM output against human judgment. For Benchmarking LLMsS7: Benchmarking LLMs , there is less need for human validation when using extensively validated and widely-used benchmarks, but researchers SHOULD employ human validation when creating or adapting new benchmarks.
… Unchanged text
Advice for Reviewers
Human validation may be the most challenging of our guidelines to assess because it often requires evaluating conceptual arguments. If LLM output is validated only by comparison with other LLMs, reviewers should look for quantitative empirical evidence that such comparison is reliable and valid. High inter-model agreement alone is insufficient, as reliability does not imply validity. Similarly, reviewers should expect evidence that any employed benchmarks are reliable and valid. Absent such evidence, human validation is warranted. A single human judge is appropriate only when judgments depend on widely accepted theories and involve limited value conflict (e.g., tagging method names containing abbreviations). For multiple judges, reviewers should expect IRA/IRR improvement techniques as described in the recommendations above (experienced raters, organized rounds, consensus meetings, updated decision rules). Low IRA or IRR (e.g., Krippendorff’s \(\alpha<0.8\)) without these techniques is a concern. Conversely, if authors have followed best practices and still obtained mediocre results (e.g., \(0.66<\alpha<0.8\)), this should be noted as a limitation. Beyond reliability, reviewers should expect authors to explain conceptually why their human judgments should be valid, considering construct definitions, decision rules, and judge expertise. As with other guidelines, missing information is typically a revision request. Absent judge instructions, instruments, decision rules, or construct definitions may prevent assessment of rigor and validity, while missing recruitment details are less critical. Clarification requests about construct definitions are routine and should not alone warrant rejection.
Report Limitations and Mitigations (G8) Changed
2026.03
Report Limitations and Mitigations (G8)
tl;dr: Researchers MUST transparently report study limitations, including the impact of non-determinism and generalizability constraints. The PAPER MUST specify whether generalization across LLMs or across time was assessed, and discuss model and version differences. Authors MUST describe measurement constructs and methods, disclose any data leakage risks, and avoid leaking evaluation data into LLM improvement pipelines. TheyMUSTprovidemodeloutputsanddiscuss sensitive data handling. They SHOULD justify LLM usage in light of its resource demands. Mitigation strategies such as replication packages, human validation, longitudinal re-runs, triangulation, and sensitivity analysis SHOULD be employed and reported where applicable. Where full data sharing is not possible, a subset of the validation data SHOULD be included to enable partial replication.
Rationale
When using LLMs for empirical studies in SE, researchers face unique challenges and potential limitations that can influence the validity, reliability, and reproducibility of their findings (Sallou, Durieux, and Panichella 2024). Itisimportantto openly discuss these limitations and explain how their impact was mitigated. These limitations are relative to current LLM capabilities and tool architectures; speculating about future improvements is beyond the scope of a paper’s limitation section. Nevertheless, risk management and threat mitigation should be planned during study design, not as an afterthought.
Recommendations
Researchers MUSTmakeeveryeffortto present the limitations of their work clearly without defensiveness or obfuscation. These limitations may concern diverse topics including generalizability, internal validity (e.g. data leakage), reliability (i.e. non-determinism), and reproducibility (e.g. resource requirements). When deterministic reproducibility is structurally unattainable (e.g., SaaS-based models with opaque versioning), researchers SHOULD adopt trustworthiness criteria from qualitative research to substantiate dependability and confirmability of findings. LLM-based studies may involve many kinds of generalization including the following:
From tested LLMs to others; that is, do the performance characteristics of the LLM(s) studied generalize to LLM(s) not included in the study.
From tested configurations to others; that is, how sensitive are the results to the specific configuration(s) of the LLM under test.
From research to practice; for instance, just because researchers can get a certain level of code generation performance under ideal conditions does not mean software developers, given the same tools, will get similar results without having the researchers present to show them exactly what to do.
From a sample of people (e.g. human validators; human participants) to a larger population of people.
From one time period to another; i.e., does the same LLM, or other versions of the same LLM, produce similar results at a different point in time?
The following concerns present an overview that has to be tailored to the individual study context. This section does not repeat requirements from other recommendations.
Internal Validity:
The primary threats to internal validity are:
Data leakage and contamination, including inter-dataset duplication, potentially resulting in a training–evaluation overlaps, leading to overly optimistic evaluation results.
Unintended inclusion of evaluation data in model-improvement pipelines, contributing to data leakage. This is especially relevant for longitudinal studies including LLMs.
Incomplete architecture, prompt, or pipeline reporting, posing hidden confounding factors.
Inter-dataset duplication is prevalent in SE, particularly for code-related benchmarks. As transparency on training data is limited for LLMs, researchers MUST discuss potential data leakage effects and their impact on results.
Construct Validity:
The primary threats to construct validity are:
Metric–construct mismatch (e.g., BLEU/ROUGE vs. functional correctness). More traditional metrics might not capture all relevant SE-specific aspects.
Over-reliance on benchmark-specific metrics. Optimizing for a single benchmark might result in dataset-specific heuristics rather than the intended construct, overstating real-world utility.
Benchmark scope limitations. Benchmarks commonly ignore runtime behaviors, security implications, readability, testability, and maintainability, yielding results that may not transfer to realistic development settings.
If constructs are based on subjective interpretations, purely automated metrics are insufficient. Researchers MUST discuss how they ensured quality of subjective results, similarly to qualitative research.
External Validity:
The primary threats to external validity are:
Cross-model transfer limitations. Results obtained with one LLM (or family of LLMs) may not generalize to others due to differences in training data, architecture, and capabilities.
Tool-architecture specificity. Tools built around a specific LLM’s API or capabilities may not transfer to other models without significant re-engineering.
Limited domain coverage. Studies often focus on a narrow set of programming languages, task types, or application domains, limiting generalizability to other SE contexts.
Limited participant diversity. Study participants (e.g., human validators, developers) may not represent the broader population in terms of expertise, geographic location, or cultural background.
Cross-time instability (model evolution). Performance of proprietary models can change over time, leading to non-generalizable study outcomes (L. Chen, Zaharia, and Zou 2024; D. Li et al. 2024).
Generalizability is particularly critical for proprietary and non-deterministic systems whose behavior is subject to drift (i.e., silent changes in model output over time) (L. Chen, Zaharia, and Zou 2024). The limitations and mitigations usedfor external validity MUSTbediscussedbytheresearchers.
Reliability & Reproducibility:
The primary threats to reliability and reproducibility are:
Non-deterministic outputs. Identical prompts and configurations can yield different outputs across runs due to factors such as floating-point arithmetic, batching, and stochastic decoding strategies.
Infrastructure dependence. Results may vary depending on the hardware, software stack, and hosting environment used, making exact replication challenging across different infrastructure setups.
Resource inequality preventing replication. LLM research is resource-intensive and hence remains predominantly in the domain of private companies or well-funded research institutions (Schwartz et al. 2020; N. Ahmed, Wahed, and Thompson 2023), excluding researchers from under-resourced institutions.
Researchers MUST discuss the measures taken to increase reliability and reproducibility. However, non-deterministic reproducibility is not inherently disqualifying. Qualitative traditions such as ethnography, grounded theory, and action research ensure trustworthiness through credibility, transferability, dependability, and confirmability(Guba 1981).SaaS-based LLM research facesanalogousconditions, as providers frequently deprecate model versions without guaranteeing stable behavior.
Ethical & Regulatory Boundaries:
The primary concerns for ethical and regulatory matters are:
Use of sensitive or proprietary data. Studies involving proprietary code, confidential business data, or personally identifiable information may face restrictions on data sharing that limit reproducibility.
Jurisdictional obligations. Data protection regulations (e.g., GDPR, CCPA) and institutional policies may impose constraints on data collection, processing, and sharing in LLM-based studies.
Implicit model bias. Especially for qualitative research LLM might “reinforce dominant paradigms and biases” and “identify, replicate and reinforce dominant language and patterns” (Jowsey et al. 2025).
Studies involving sensitive data MUST discuss data governance mechanisms tailored towards LLM environments, compliant with applicable juristical obligations. If applicable, researchers SHOULD discuss how model biases potentially impact the study outcomes and how those biases were evaluated.
Environmental & Sustainability Constraints:
Energy consumption. With growing model size, the environmental impact of experiments with LLMs increases. Considering the environmental impact in the study design is important given the substantial energy costs of LLM experiments (Strubell, Ganesh, and McCallum 2019).
Trade-off between repetition and sustainability. Increased reliability throughrepetitionandtheaimofcautious energy consumption poseaninherenttensionfieldtonavigate during the study design.
ThestudyfindingsSHOULDbediscussedinthecontextoftheresource-intensivenessofLLMsto justify the increased resource consumptionover traditional approaches.
Mitigation Strategies
The following mitigation strategies can help address the threats described above, depending on the study scope and feasibility.
Replication Packages covering prompt and architecture specifications, model outputs, and representative examples for partial replicability, accompanied by an implementation using an open model for long-term stability.
Human Validation of subjective constructs, following quality criteria known from qualitative research.
Longitudinal Re-Runs to repeat experiments with LLMs over time, complemented by statistical analyses.
Methodological Trustworthiness Measures. Researchers SHOULD consider triangulation, reflexivity, audit trails, and peer debriefing as complementary measures when deterministic reproduction is structurally impossible.
Triangulation via multiple models (e.g., proprietary and open models), multiple independent datasets, and multiple complementary metrics.
Cost Accounting through reporting of input/output tokens, token and service costs, or hardware specifications.
Energy Preservation by selecting smaller or newer less resource-intensive models and employing techniques such as input/output token reduction, model pruning, quantization, or knowledge distillation (Mitu and Mitu 2024) where feasible. Carbon footprint estimation is desirable, but still difficult.
Ethical and Regulatory Considerations through data governance mechanisms, ethical reviews, and bias assessment procedures.
Sensitivity Analysis through variation of LLM configurations, prompts, architecture decisions, datasets, and if applicable human participant backgrounds.
… Unchanged text
Study Types
Researchers MUST follow this guideline for all study types. Transparently reporting limitations and mitigations is a universal requirement, but specific concerns vary by study type. For LLMs as AnnotatorsS1: LLMs as Annotators , researchers MUST discuss potential biases in label assignment, label reliability limitations, and sensitivity of annotations to prompt wording and model choice. For LLMs as JudgesS2: LLMs as Judges , researchers MUST address measurement validity concerns, known biases such as position bias or verbosity bias, and the extent to which LLM judgments align with human expert assessments. For LLMs for SynthesisS3: LLMs for Synthesis , researchers MUST discuss the risk of contextual misinterpretation, potential loss of nuance in summarized or aggregated outputs, and reflexivity limitations inherent in using an LLM for qualitative interpretation. For LLMs as SubjectsS4: LLMs as Subjects , researchers MUST discuss the fundamental inability of LLMs to truly simulate human behavior, the risk of stereotype amplification, and the limited ecological validity of simulated responses. For Studying LLM UsageS5: Studying LLM Usage , researchers MUST discuss generalizability constraints across different tools and user populations, and acknowledge how observed usage patterns may not transfer to other contexts. For LLMs for ToolsS6: LLMs for Tools , researchers MUST discuss replicability constraints arising from dependencies on commercial models, the impact of model updates on tool behavior, and limitations of the evaluation setup. For Benchmarking LLMsS7: Benchmarking LLMs , researchers MUST discuss potential data contamination, benchmark scope limitations, and the extent to which benchmark performance generalizes to real-world tasks.
Advice for Reviewers
Reviewers should verify that the limitation section is comprehensive and appropriate for the specific study type, checking that: (1) limitations address the specific threats relevant to the study type (e.g., label reliability for annotation studies, simulation fidelity for studies using LLMs as subjects); (2) mitigations are concrete and correspond to identified limitations rather than being generic statements; (3) the impact of LLM non-determinism on findings is discussed; (4) generalizability constraints—across models, configurations, time periods, and populations—are acknowledged. When important limitations are missing, reviewers should request they be added. The absence of a limitation section, or one that is formulaic or insufficiently specific, is a more serious concern than any individual missing limitation and may warrant a major revision.
● Justify metric and benchmark choices; discuss their validity. ○ Repeat experiments and report descriptive statistics.
G8
Honest acknowledgment of threats strengthens a study.
● Report threats by validity type. ○ Employ mitigations where possible.
2026.04
Report Limitations and Mitigations (G8)
tl;dr: Researchers MUST transparently report study limitations, including the impact of non-determinism and generalizability constraints. The PAPER MUST specify whether generalization across LLMs or across time was assessed, and discuss model and version differences. Authors MUST describe measurement constructs and methods, disclose any data leakage risks, and avoid leaking evaluation data into LLM improvement pipelines. For studies involving sensitive data, they MUST discuss data governance mechanisms. They SHOULD justify LLM usage in light of its resource demands. Mitigation strategies such as replication packages, human validation, longitudinal re-runs, triangulation, and sensitivity analysis SHOULD be employed and reported where applicable. Where full data sharing is not possible, a subset of the validation data SHOULD be included to enable partial replication.
Rationale
When using LLMs for empirical studies in SE, researchers face unique challenges and potential limitations that can influence the validity, reliability, and reproducibility of their findings (Sallou, Durieux, and Panichella 2024). Researchers must openly discuss these limitations and explain how their impact was mitigated. These limitations are relative to current LLM capabilities and tool architectures; speculating about future improvements is beyond the scope of a paper’s limitation section. Nevertheless, risk management and threat mitigation SHOULD be planned during study design, not as an afterthought.
Recommendations
Researchers MUST clearly present the limitations of their workwithout defensiveness or obfuscation. These limitations may concern diverse topics including generalizability, internal validity (e.g. data leakage), reliability (i.e. non-determinism), and reproducibility (e.g. resource requirements). When deterministic reproducibility is structurally unattainable (e.g., SaaS-based models with opaque versioning), researchers SHOULD adopt trustworthiness criteria from qualitative research to substantiate dependability and confirmability of findings. LLM-based studies may involve many kinds of generalization including the following:
From tested LLMs to others; that is, do the performance characteristics of the LLM(s) studied generalize to LLM(s) not included in the study.
From tested configurations to others; that is, how sensitive are the results to the specific configuration(s) of the LLM under test.
From research to practice; for instance, just because researchers can get a certain level of code generation performance under ideal conditions does not mean software developers, given the same tools, will get similar results without having the researchers present to show them exactly what to do.
From a sample of people (e.g. human validators; human participants) to a larger population of people.
From one time period to another; i.e., does the same LLM, or other versions of the same LLM, produce similar results at a different point in time?
The following concerns present an overview that has to be tailored to the individual study context. This section does not repeat requirements from other recommendations.
Internal Validity:
The primary threats to internal validity are:
Data leakage and contamination, including inter-dataset duplication, potentially resulting in a training–evaluation overlaps, leading to overly optimistic evaluation results.
Unintended inclusion of evaluation data in model-improvement pipelines, contributing to data leakage. This is especially relevant for longitudinal studies including LLMs.
Incomplete architecture, prompt, or pipeline reporting, posing hidden confounding factors.
Inter-dataset duplication is prevalent in SE, particularly for code-related benchmarks. As transparency on training data is limited for LLMs, researchers MUST discuss potential data leakage effects and their impact on results.
Construct Validity:
The primary threats to construct validity are:
Metric–construct mismatch (e.g., BLEU/ROUGE vs. functional correctness). More traditional metrics might not capture all relevant SE-specific aspects.
Over-reliance on benchmark-specific metrics. Optimizing for a single benchmark might result in dataset-specific heuristics rather than the intended construct, overstating real-world utility.
Benchmark scope limitations. Benchmarks commonly ignore runtime behaviors, security implications, readability, testability, and maintainability, yielding results that may not transfer to realistic development settings.
If constructs are based on subjective interpretations, purely automated metrics are insufficient. Researchers MUST discuss how they ensured quality of subjective results, similarly to qualitative research.
External Validity:
The primary threats to external validity are:
Cross-model transfer limitations. Results obtained with one LLM (or family of LLMs) may not generalize to others due to differences in training data, architecture, and capabilities.
Tool-architecture specificity. Tools built around a specific LLM’s API or capabilities may not transfer to other models without significant re-engineering.
Limited domain coverage. Studies often focus on a narrow set of programming languages, task types, or application domains, limiting generalizability to other SE contexts.
Limited participant diversity. Study participants (e.g., human validators, developers) may not represent the broader population in terms of expertise, geographic location, or cultural background.
Cross-time instability (model evolution). Performance of proprietary models can change over time, leading to non-generalizable study outcomes (L. Chen, Zaharia, and Zou 2024; D. Li et al. 2024).
Generalizability is particularly critical for proprietary and non-deterministic systems whose behavior is subject to drift (i.e., silent changes in model output over time) (L. Chen, Zaharia, and Zou 2024). Researchers MUST discuss the limitations and mitigations of external validity.
Reliability & Reproducibility:
The primary threats to reliability and reproducibility are:
Non-deterministic outputs. Identical prompts and configurations can yield different outputs across runs due to factors such as floating-point arithmetic, batching, and stochastic decoding strategies.
Infrastructure dependence. Results may vary depending on the hardware, software stack, and hosting environment used, making exact replication challenging across different infrastructure setups.
Resource inequality preventing replication. LLM research is resource-intensive and hence remains predominantly in the domain of private companies or well-funded research institutions (Schwartz et al. 2020; N. Ahmed, Wahed, and Thompson 2023), excluding researchers from under-resourced institutions.
Researchers MUST discuss the measures taken to increase reliability and reproducibility. However, non-deterministic reproducibility is not inherently disqualifying. Qualitative traditions such as ethnography, grounded theory, and action research ensure trustworthiness through credibility, transferability, dependability, and confirmability(Guba 1981). The same applies to SaaS-based LLM research, where providers frequently deprecate model versions without guaranteeing stable behavior.
Ethical & Regulatory Boundaries:
The primary concerns for ethical and regulatory matters are:
Use of sensitive or proprietary data. Studies involving proprietary code, confidential business data, or personally identifiable information may face restrictions on data sharing that limit reproducibility.
Jurisdictional obligations. Data protection regulations (e.g., GDPR, CCPA) and institutional policies may impose constraints on data collection, processing, and sharing in LLM-based studies.
Implicit model bias. Especially for qualitative research LLM might “reinforce dominant paradigms and biases” and “identify, replicate and reinforce dominant language and patterns” (Jowsey et al. 2025).
Studies involving sensitive data MUST discuss data governance mechanisms tailored towards LLM environments, compliant with applicable juristical obligations. If applicable, researchers SHOULD discuss how model biases potentially impact the study outcomes and how those biases were evaluated.
Environmental & Sustainability Constraints:
Energy consumption. With growing model size, the environmental impact of experiments with LLMs increases. Considering the environmental impact in the study design is important given the substantial energy costs of LLM experiments (Strubell, Ganesh, and McCallum 2019).
Trade-off between repetition and sustainability. Repeating experiments increases reliability but also energy consumption, requiring trade-offs duringstudy design.
Researchers SHOULDjustify the LLM’s resource consumption against the benefits over traditional approaches.
Mitigation Strategies:
The following mitigation strategies can help address the threats described above, depending on the study scope and feasibility.
Replication Packages covering prompt and architecture specifications, model outputs, and representative examples for partial replicability, accompanied by an implementation using an open model for long-term stability.
Human Validation of subjective constructs, following quality criteria known from qualitative research.
Longitudinal Re-Runs to repeat experiments with LLMs over time, complemented by statistical analyses.
Methodological Trustworthiness Measures. Researchers SHOULD consider triangulation, reflexivity, audit trails, and peer debriefing as complementary measures when deterministic reproduction is structurally impossible.
Triangulation via multiple models (e.g., proprietary and open models), multiple independent datasets, and multiple complementary metrics.
Cost Accounting through reporting of input/output tokens, token and service costs, or hardware specifications.
Energy Preservation by selecting smaller or newer less resource-intensive models and employing techniques such as input/output token reduction, model pruning, quantization, or knowledge distillation (Mitu and Mitu 2024) where feasible. Carbon footprint estimation is desirable, but still difficult.
Ethical and Regulatory Considerations through data governance mechanisms, ethical reviews, and bias assessment procedures.
Sensitivity Analysis through variation of LLM configurations, prompts, architecture decisions, datasets, and if applicable human participant backgrounds.
… Unchanged text
Study Types
Researchers MUST follow this guideline for all study types. Transparently reporting limitations and mitigations is a universal requirement, but specific concerns vary by study type. For LLMs as AnnotatorsS1: LLMs as Annotators , researchers MUST discuss potential biases in label assignment, label reliability limitations, and sensitivity of annotations to prompt wording and model choice. For LLMs as JudgesS2: LLMs as Judges , researchers MUST address measurement validity concerns, known biases such as position bias or verbosity bias, and the extent to which LLM judgments align with human expert assessments. For LLMs for SynthesisS3: LLMs for Synthesis , researchers MUST discuss the risk of contextual misinterpretation, potential loss of nuance in summarized or aggregated outputs, and reflexivity limitations inherent in using an LLM for qualitative interpretation. For LLMs as SubjectsS4: LLMs as Subjects , researchers MUST discuss the fundamental inability of LLMs to truly simulate human behavior, the risk of stereotype amplification, and the limited ecological validity of simulated responses. For Studying LLM UsageS5: Studying LLM Usage , researchers MUST discuss generalizability constraints across different tools and user populations, and acknowledge how observed usage patterns may not transfer to other contexts. For LLMs for ToolsS6: LLMs for Tools , researchers MUST discuss replicability constraints arising from dependencies on commercial models, the impact of model updates on tool behavior, and limitations of the evaluation setup. For Benchmarking LLMsS7: Benchmarking LLMs , researchers MUST discuss potential data contamination, benchmark scope limitations, and the extent to which benchmark performance generalizes to real-world tasks.
Advice for Reviewers
Reviewers should verify that the limitation section is comprehensive and appropriate for the specific study type, checking that: (1) limitations address the specific threats relevant to the study type (e.g., label reliability for annotation studies, simulation fidelity for studies using LLMs as subjects); (2) mitigations are concrete and correspond to identified limitations rather than being generic statements; (3) the impact of LLM non-determinism on findings is discussed; (4) generalizability constraints—across models, configurations, time periods, and populations—are acknowledged. When important limitations are missing, reviewers should request they be added. The absence of a limitation section, or one that is formulaic or insufficiently specific, is a more serious concern than any individual missing limitation and may warrant a major revision.
Rationale and key recommendations for each guideline. ● = MUST, ○ = SHOULD.
ID
Rationale
Core Recommendations
G1
Transparency enables informed assessment of scope and limitations.
● Declare which LLM, how it was used, and where in the research process.
G2
Reproducibility requires precise identification of the system used in a study.
● Report exact version, date, configuration, and fine-tuning details.
G3
Static artifacts, from system architecture to prompt templates, determine what the model sees on every call and must be documented in full.
● Describe full architecture, infrastructure,all prompts or prompt templates (with development strategy and reuse), context files, tool catalog, agent architecture, and retrieval mechanisms. ○ For studies of LLM usage (S5), describe the tool architecture only to the extent accessible.
G4
Runtime traces make LLM and agent behavior verifiable despite non-determinism and commercial-tool opacity.
● For studies of LLM usage (S5), share full interaction logs subject to privacy constraints. ○ Otherwise, share interaction logs, runtime traces, and plans where feasible.
● Justify metric and benchmark choices; discuss their validity. ○ Repeat experiments and report descriptive statistics.
G6
Reproducibility depends on access to the model under study.
● For benchmarking studies (S7), include an open LLM among the models evaluated. ○ Otherwise, include an open LLM as a baseline; provide a replication package.
G7
Automated metrics alone cannot ensure validity of subjective constructs.
● Define the measured construct and share any custom measurement instruments. ○ Validate against human judgment with inter-rater reliability, where applicable.
G8
Honest acknowledgment of threats strengthens a study.
● Report threats by validity type. ○ Employ mitigations where possible.
Acknowledgments Changed
2026.03
Acknowledgments
Our study types and guidelines are based on discussion sessions with researchers at the 2024 International Software Engineering Research Network (ISERN) meeting and at the 2nd Copenhagen Symposium on Human-Centered Software Engineering AI (supported by the Carlsberg Foundation with grant CF24-0693 and the Alfred P. Sloan Foundation with grant G-2024-22586 to Daniel Russo). LLM-based tools, including ChatGPT (GPT-5) and Claude Code (Opus 4.6), were used interactively for language editing and structural revision suggestions across all sections of this paper. All LLM-generated suggestions were reviewed and revised by the authors.
2026.04
Acknowledgments
Our study types and guidelines are based on discussion sessions with researchers at the 2024 International Software Engineering Research Network (ISERN) meeting and at the 2nd Copenhagen Symposium on Human-Centered Software Engineering AI (supported by the Carlsberg Foundation with grant CF24-0693 and the Alfred P. Sloan Foundation with grant G-2024-22586 to Daniel Russo). LLM-based tools, including ChatGPT (GPT-5) and Claude Code (Opus 4.6 and 4.7), were used interactively for language editing and structural revision suggestions across all sections of this paper. All LLM-generated suggestions were reviewed and revised by the authors.
Reporting Checklist Changed
2026.03
Reporting Checklist
The following checklist, inspired by CONSORT (Schulz, Altman, and Moher 2010), summarizes actionable items from the guidelines based on the tl;dr sections. The checklist is organized along typical paper sections. Items marked ● are requirements (MUST ); items marked ○ are recommendations (SHOULD ). Each item references its source guideline (G1–G8). Items annotated with PAPER or SUPPLEMENTARY MATERIAL indicate where information should be reported; unmarked items may be reported in either.
… Unchanged text
Introduction
● Disclose any use of LLMs in the empirical study, specifying which LLM, how, and where it was used (G1).
○ Report the purpose of using LLMs, automated tasks, and expected benefits in the PAPER (G1).
Research Design and Methods
… Unchanged text
Model Selection and Configuration
● Report the exact LLM model or tool version, configuration, and experiment date in the PAPER (G2).
● For fine-tuned models, describe the fine-tuning goal, dataset, and procedure in the PAPER (G2).
○ Report default parameters and explain model and version choices (G2).
○ For quantized models, report the quantization level (e.g., 4-bit, 8-bit) and method (e.g., GPTQ or AWQ) (G2).
○ Compare base and fine-tuned models using suitable metrics and benchmarks; share fine-tuning data and weights as SUPPLEMENTARY MATERIAL (or justify in the PAPER why they cannot be shared) (G2).
Architecture
● Describe the full architecture of LLM-based tools in the PAPER, including the role of the LLM, interactions with other components, and overall system behavior (G3).
● Forensemblearchitectures, explain the coordinationlogicbetweenmodels in the PAPER (G3).
● If autonomous agents are used, specify agent roles, reasoning frameworks, and communication flows (G3).
● Report hosting, hardware setup, and latency implications (G3).
● Fortoolsusingretrievaloraugmentationmethods,describedatasources,integrationmechanisms, and updateandversioningstrategies (G3).
○Includearchitecturaldiagrams and justifydesigndecisions (G3).
● Publish all prompts or, when using templates, prompt templates with representative instances, including their structure, content, formatting, and dynamic components, as SUPPLEMENTARY MATERIAL (G4).
● Document input handling and token optimization strategies when prompts are long or complex (G4).
○ If full prompt disclosure is not feasible, provide summaries or examples (G4).
○ Report prompt revisions and pilottestinginsights (G4).
○ Include full interaction logs (prompts and responses) as SUPPLEMENTARY MATERIAL if privacy and confidentiality can be ensured (G4).
○ For agentic systems, reportallcontextfiles (e.g., AGENTS.md) usedtoconfigureAIagents as SUPPLEMENTARY MATERIAL (G4).
○ For agentic systems, report developedplansandgeneralizeinteractionlogstoincludehuman-agentinteractiontraces as SUPPLEMENTARY MATERIAL(G4).
Human Validation
● If using human validation, define the measured construct (e.g., usability, maintainability) and describe the measurement instrument in the PAPER (G5).
● When developing or adapting measurement instruments, share them (G5).
○ Consider human validation early in the study design and build on established reference models for human-LLM comparison (G5).
○ Validate LLM judgments against human judgment, report aggregation methods, and assess human-LLM agreement (G5).
○ Control for confounding factors andconductpoweranalysistoensurestatisticalrobustness (G5).
Benchmarks and Metrics
● Justify all benchmark and metric choices in the PAPER (G7).
● Explain in the PAPER why the selected metrics are suitable for the specific study (G7).
○ Include an open LLM as a baseline when using commercial models and report inter-model agreement (G6).
○ Summarize benchmark structure, task types, and limitations (G7).
○ Justify the number of experiment repetitions, for example through a power analysis or by monitoring convergence of descriptive statistics (G7).
Reproducibility, Ethics, and Resources
● ProviderawLLMoutputsforreproducibilityanddiscuss sensitive data handling (G8).
○ Justify LLM usage in light of its resource demands (G8).
○ If full datasharingis not possible, includeasubsetofthevalidationdata for partial replication (G8).
○ Provide a full replication package with step-by-step instructions as SUPPLEMENTARY MATERIAL (G6).
Results
● If comparing models or tools, use appropriate inferential statistics (e.g., hypothesis tests, effect sizes) rather than relying solely on summary statistics (G7).
○ Repeat experiments due to the inherent non-determinism of LLMs and report the result distribution using descriptive statistics (G7).
○ Use traditional (non-LLM) baselines for comparison where possible (G7).
○ Report established metrics to make study results comparable; additional metrics may be reported where appropriate (G7).
… Unchanged text
Limitations and Threats to Validity
● Describe measurement constructs and methods; disclose any data leakage risks and avoid leaking evaluation data into LLM improvement pipelines (G8).
● Acknowledge non-disclosed confidential or proprietary components as reproducibility limitations (G3).
● Transparently report study limitations, including the impact of non-determinism and generalizability constraints (G8).
● Specify whether generalization across LLMs or across time was assessed, and discuss model and version differences (G8).
○ Employ and report strategies to mitigate identified validity and reproducibility threats, such as replication packages, human validation, longitudinal re-runs, triangulation, and sensitivity analysis (G8).
2026.04
Reporting Checklist
The following checklist, inspired by CONSORT (Schulz, Altman, and Moher 2010), summarizes actionable items from the guidelines based on the tl;dr sections. The checklist is organized along typical paper sections. Items marked ● are requirements (MUST ); items marked ○ are recommendations (SHOULD ). Each item references its source guideline (G1–G8). Items annotated with PAPER or SUPPLEMENTARY MATERIAL indicate where information should be reported; unmarked items may be reported in either.
… Unchanged text
Introduction
● Disclose any use of LLMs in the empirical study, specifying which LLM, how, and where it was used (G1).
○ Report the purpose of using LLMs, automated tasks, and expected benefits in the PAPER (G1).
Research Design and Methods
… Unchanged text
Model Selection and Configuration
● Report the exact LLM model or tool version, configuration, and experiment date in the PAPER (G2).
● For fine-tuned models, describe the fine-tuning goal, dataset, and procedure in the PAPER (G2).
○ Report default parameters and explain model and version choices (G2).
○ For quantized models, report the quantization level (e.g., 4-bit, 8-bit) and method (e.g., GPTQ or AWQ) (G2).
○ Compare base and fine-tuned models using suitable metrics and benchmarks; share fine-tuning data and weights as SUPPLEMENTARY MATERIAL (or justify in the PAPER why they cannot be shared) (G2).
System and Prompt Design
● Describe the full architecture of LLM-based tools in the PAPER, including the role of the LLM, interactions with other components, and overall system behavior (G3).
● Describe any context-file mechanisms used (e.g., AGENTS.md, CLAUDE.md) in the PAPER (G3).
● Summarize which tools and skills were exposed to the model in the PAPER (G3).
● Specify whether zero-shot, one-shot, or few-shot prompting was used in the PAPER (G3).
● If autonomous agents are used, specify agent roles, reasoning frameworks, and communication flows (G3).
● Report hosting, hardware setup, and latency implications (G3).
● Specify prompt reuse across models and configurations (G3).
● For retrieval-augmented generation (RAG) or related methods, describe how external data was retrieved, stored, and integrated (G3).
● For dynamically generated prompts, document the generation process thoroughly (G3).
● Publish all prompts or, when using templates, prompt templates with representative instances, including their structure, content, formatting, and dynamic components, as SUPPLEMENTARY MATERIAL (G3).
○ For ensemble architectures, explain the coordination logic between models in the PAPER (G3).
○ Justify design decisions (G3).
○ Where legally possible, release the source code of the implementation under an open-source license (G3).
○ Report data preprocessing, versioning, and update frequency for stored data used for context augmentation (G3).
○ Describe prompt development rationale and selection process (G3).
○ Document input handling and token optimization strategies when prompts are long or complex (G3).
○ For user-authored prompts, describe how they were collected and analyzed (G3).
○ If full prompt disclosure is not feasible, provide summaries or examples (G3).
○ Report prompt evolution and any LLM-suggested refinements (G3).
○ Include context-file contents as SUPPLEMENTARY MATERIAL (G3).
○ Include the complete list of tools and skills (names with one-line purposes), tool schemas, skill definitions, sub-agent definitions, and connected MCP servers as SUPPLEMENTARY MATERIAL (G3).
Session Traces
○ Include full interaction logs (prompts and responses) as SUPPLEMENTARY MATERIAL if privacy and confidentiality can be ensured (G4).
○ For agentic systems, include interaction logs covering exchanges between humans and the agent and between external tools and the agent (human-in-the-loop feedback, approvals, refinements)as SUPPLEMENTARY MATERIAL (G4).
○ For agentic systems, report the complete runtime trace as SUPPLEMENTARY MATERIAL, including for each entry the tool or artifact name, arguments, result, and ordering, and which configured artifacts (skills, context files, sub-agents) were activated (G4).
○ For agentic systems, report developed plans as SUPPLEMENTARY MATERIAL if available (G4).
Human Validation
● If using human validation, define the measured construct (e.g., usability, maintainability) and describe the measurement instrument in the PAPER (G7).
● When developing or adapting measurement instruments, share them (G7).
○ Consider human validation early in the study design and build on established reference models for human-LLM comparison (G7).
○ Validate LLM judgments against human judgment, report aggregation methods, and assess human-LLM agreement (G7).
○ Discuss and, where feasible, control for confounding factors(G7).
Benchmarks and Metrics
● Justify all benchmark and metric choices in the PAPER (G5).
● Explain in the PAPER why the selected metrics are suitable for the specific study (G5).
○ Include an open LLM as a baseline when using commercial models and report inter-model agreement (G6).
○ Summarize benchmark structure, task types, and limitations (G5).
○ Justify the number of experiment repetitions, for example through a power analysis or by monitoring convergence of descriptive statistics (G5).
Reproducibility, Ethics, and Resources
● For studies involving sensitive data, discuss data governance mechanisms compliant with applicable jurisdictional obligations (G8).
○ Justify LLM usage in light of its resource demands (G8).
○ Where fullsharing of prompts, traces, or datasets is not feasible, share representative examples for partial replicability (G8).
○ Provide a full replication package with step-by-step instructions as SUPPLEMENTARY MATERIAL (G6).
Results
● If comparing models or tools, use appropriate inferential statistics (e.g., hypothesis tests, effect sizes) rather than relying solely on summary statistics (G5).
○ Repeat experiments due to the inherent non-determinism of LLMs and report the result distribution using descriptive statistics (G5).
○ Use traditional (non-LLM) baselines for comparison where possible (G5).
○ Report established metrics to make study results comparable; additional metrics may be reported where appropriate (G5).
… Unchanged text
Limitations and Threats to Validity
● Describe measurement constructs and methods; disclose any data leakage risks and avoid leaking evaluation data into LLM improvement pipelines (G8).
● Acknowledge non-disclosed confidential or proprietary components as reproducibility limitations (G3).
● Transparently report study limitations, including the impact of non-determinism and generalizability constraints (G8).
● Specify whether generalization across LLMs or across time was assessed, and discuss model and version differences (G8).
○ Employ and report strategies to mitigate identified validity and reproducibility threats, such as replication packages, human validation, longitudinal re-runs, triangulation, and sensitivity analysis (G8).
Agarwal, Rishabh, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Marc G. Bellemare. 2021. “Deep Reinforcement Learning at the Edge of the Statistical Precipice.” In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, Virtual, edited by Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, 29304–20. https://proceedings.neurips.cc/paper/2021/hash/f514cec81cb148559cf475e7426eed5e-Abstract.html.
Ahmed, Nur, Muntasir Wahed, and Neil C. Thompson. 2023. “The Growing Influence of Industry in AI Research.”Science 379 (6635): 884–86. https://doi.org/10.1126/science.ade2420.
Ahmed, Toufique, Premkumar T. Devanbu, Christoph Treude, and Michael Pradel. 2025. “Can LLMs Replace Manual Annotation of Software Engineering Artifacts?” In 22nd IEEE/ACM International Conference on Mining Software Repositories, MSR@ICSE 2025, Ottawa, ON, Canada, April 28-29, 2025, 526–38. IEEE. https://doi.org/10.1109/MSR66628.2025.00086.
Ahuja, Sanchit, Varun Gumma, and Sunayana Sitaram. 2024. “Contamination Report for Multilingual Benchmarks.”CoRR abs/2410.16186. https://doi.org/10.48550/ARXIV.2410.16186.
Angermeir, Florian, Maximilian Amougou, Mark Kreitz, Andreas Bauer, Matthias Linhuber, Davide Fucci, Fabiola Moyón Constante, Daniel Méndez, and Tony Gorschek. 2025. “Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies.”CoRR abs/2510.25506. https://doi.org/10.48550/ARXIV.2510.25506.
Argyle, Lisa P., Ethan C. Busby, Nancy Fulda, Joshua Gubler, Christopher Michael Rytting, and David Wingate. 2022. “Out of One, Many: Using Language Models to Simulate Human Samples.”CoRR abs/2209.06899. https://doi.org/10.48550/ARXIV.2209.06899.
Austin, Jacob, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, et al. 2021. “Program Synthesis with Large Language Models.”CoRR abs/2108.07732. https://arxiv.org/abs/2108.07732.
Azanza, Maider, Juanan Pereira, Arantza Irastorza, and Aritz Galdos. 2024. “Can LLMs Facilitate Onboarding Software Developers? An Ongoing Industrial Case Study.” In 36th International Conference on Software Engineering Education and Training, CSEE&t 2024, 1–6. IEEE. https://doi.org/10.1109/CSEET62301.2024.10662989.
Bano, Muneera, Hashini Gunatilake, and Rashina Hoda. 2025. “What Does a Software Engineer Look Like? Exploring Societal Stereotypes in LLMs.” In 47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Society, ICSE-SEIS 2025, Ottawa, ON, Canada, April 27 - May 3, 2025, 173–84. IEEE. https://doi.org/10.1109/ICSE-SEIS66351.2025.00023.
Bano, Muneera, Rashina Hoda, Didar Zowghi, and Christoph Treude. 2024. “Large Language Models for Qualitative Research in Software Engineering: Exploring Opportunities and Challenges.”Autom. Softw. Eng. 31 (1): 8. https://doi.org/10.1007/S10515-023-00407-8.
Barros, Cauã Ferreira, Bruna Borges Azevedo, Valdemar Vicente Graciano Neto, Mohamad Kassab, Marcos Kalinowski, Hugo Alexandre Dantas do Nascimento, and Michelle C. G. S. P. Bandeira. 2025. “Large Language Model for Qualitative Research: A Systematic Mapping Study.” In IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering, WSESE@ICSE 2025, Ottawa, ON, Canada, May 3, 2025, 48–55. IEEE. https://doi.org/10.1109/WSESE66602.2025.00015.
Bavaresco, Anna, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, et al. 2024. “LLMs Instead of Human Judges? A Large Scale Empirical Study Across 20 NLP Evaluation Tasks.”CoRR abs/2406.18403. https://doi.org/10.48550/ARXIV.2406.18403.
Begg, C., M. Cho, S. Eastwood, R. Horton, D. Moher, I. Olkin, R. Pitkin, et al. 1996. “Improving the Quality of Reporting of Randomized Controlled Trials. The CONSORT statement.”JAMA 276 (8): 637–39. https://doi.org/10.1001/jama.276.8.637.
Bjarnason, Bjarni Haukur, André Silva, and Martin Monperrus. 2026. “On Randomness in Agentic Evals.”https://arxiv.org/abs/2602.07150.
Blackwell, Robert E., Jon Barry, and Anthony G. Cohn. 2024. “Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scores.”https://arxiv.org/abs/2410.03492.
Borsboom, Denny. 2005. Measuring the Mind: Conceptual Issues in Contemporary Psychometrics. Cambridge University Press. https://doi.org/10.1017/CBO9780511490026.
Cai, Yuchen, Aashish Yadavally, Abhishek Mishra, Genesis Montejo, and Tien N. Nguyen. 2024. “Programming Assistant for Exception Handling with CodeBERT.” In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024, 94:1–13. ACM. https://doi.org/10.1145/3597503.3639188.
Cao, Jialun, Yuk-Kit Chan, Zixuan Ling, Wenxuan Wang, Shuqing Li, Mingwei Liu, Ruixi Qiao, et al. 2025. “How Should We Build a Benchmark? Revisiting 274 Code-Related Benchmarks for LLMs.”CoRR abs/2501.10711. https://arxiv.org/abs/2501.10711.
Chen, Lingjiao, Matei Zaharia, and James Zou. 2024. “How Is ChatGPT’s Behavior Changing over Time?”Harvard Data Science Review 6 (2). https://doi.org/10.1162/99608f92.5317da47.
Chen, Mark, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, et al. 2021. “Evaluating Large Language Models Trained on Code.”CoRR abs/2107.03374. https://arxiv.org/abs/2107.03374.
Chen, Yizheng, Zhoujie Ding, Lamya Alowain, Xinyun Chen, and David A. Wagner. 2023. “DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection.” In Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses, RAID 2023, Hong Kong, China, October 16-18, 2023, 654–68. ACM. https://doi.org/10.1145/3607199.3607242.
Choi, Hyeong Kyu, Maxim Khanov, Hongxin Wei, and Yixuan Li. 2025. “How Contaminated Is Your Benchmark? Quantifying Dataset Leakage in Large Language Models with Kernel Divergence.”CoRR abs/2502.00678. https://doi.org/10.48550/ARXIV.2502.00678.
Dhar, Rudra, Karthik Vaidhyanathan, and Vasudeva Varma. 2024. “Can LLMs Generate Architectural Design Decisions? - an Exploratory Empirical Study.” In 21st IEEE International Conference on Software Architecture, ICSA 2024, Hyderabad, India, June 4-8, 2024, 79–89. IEEE. https://doi.org/10.1109/ICSA59870.2024.00016.
Du, Xueying, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2023. “ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-Level Code Generation.”CoRR abs/2308.01861. https://doi.org/10.48550/ARXIV.2308.01861.
… Unchanged text
Dybå, Tore, Vigdis By Kampenes, and Dag I. K. Sjøberg. 2006. “A Systematic Review of Statistical Power in Software Engineering Experiments.”Inf. Softw. Technol. 48 (8): 745–55. https://doi.org/10.1016/J.INFSOF.2005.08.009.
Eghbali, Aryaz, and Michael Pradel. 2022. “CrystalBLEU: Precisely and Efficiently Measuring the Similarity of Code.” In 37th IEEE/ACM International Conference on Automated Software Engineering, ASE 2022, 28:1–12. ACM. https://doi.org/10.1145/3551349.3556903.
El-Hajjami, Abdelkarim, and Camille Salinesi. 2025. “How Good Are Synthetic Requirements? Evaluating LLM-Generated Datasets for AI4RE.”CoRR abs/2506.21138. https://doi.org/10.48550/ARXIV.2506.21138.
Gallegos, Isabel O., Ryan A. Rossi, Joe Barrow, Md. Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen Ahmed. 2024. “Bias and Fairness in Large Language Models: A Survey.”Computational Linguistics 50: 1097–1179. https://doi.org/10.1162/coli_a_00524.
Gallifant, Jack, Majid Afshar, Saleem Ameen, Yindalon Aphinyanaphongs, Shan Chen, Giovanni Cacciamani, Dina Demner-Fushman, et al. 2025. “The TRIPOD-LLM Reporting Guideline for Studies Using Large Language Models.”Nature Medicine 31 (1): 60–69. https://doi.org/10.1038/s41591-024-03425-5.
Gerosa, Marco Aurélio, Bianca Trinkenreich, Igor Steinmacher, and Anita Sarma. 2024. “Can AI Serve as a Substitute for Human Subjects in Software Engineering Research?”Autom. Softw. Eng. 31 (1): 13. https://doi.org/10.1007/S10515-023-00409-6.
Gilardi, Fabrizio, Meysam Alizadeh, and Maël Kubli. 2023. “ChatGPT Outperforms Crowd Workers for Text-Annotation Tasks.”Proceedings of the National Academy of Sciences 120 (30): e2305016120. https://doi.org/10.1073/pnas.2305016120.
Gonçalves, José, Miguel Silva, Bernardo Cabral, Tiago Dias, Eva Maia, Isabel Praça, Ricardo Severino, and Luı́s Lino Ferreira. 2025. “Evaluating LLaMA 3.2 for Software Vulnerability Detection.” In Cybersecurity - 9th European Interdisciplinary Cybersecurity Conference, EICC 2025, Rennes, France, June 18-19, 2025, Proceedings, edited by Isabel Praça, Simona Bernardi, and Pedro R. M. Inácio, 2500:38–51. Communications in Computer and Information Science. Springer. https://doi.org/10.1007/978-3-031-94855-8\_3.
Guba, Egon G. 1981. “Criteria for Assessing the Trustworthiness of Naturalistic Inquiries.”ECTJ 29 (2): 75–91. https://doi.org/10.1007/BF02766777.
Guo, Daya, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, et al. 2024. “DeepSeek-Coder: When the Large Language Model Meets Programming - the Rise of Code Intelligence.”CoRR abs/2401.14196. https://doi.org/10.48550/ARXIV.2401.14196.
Harding, Jacqueline, William D’Alessandro, N. G. Laskowski, and Robert Long. 2024. “AI Language Models Cannot Replace Human Research Participants.”AI Soc. 39 (5): 2603–5. https://doi.org/10.1007/S00146-023-01725-X.
He, Junda, Jieke Shi, Terry Yue Zhuo, Christoph Treude, Jiamou Sun, Zhenchang Xing, Xiaoning Du, and David Lo. 2025. “LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead.”CoRR abs/2510.24367. https://doi.org/10.48550/ARXIV.2510.24367.
He, Zeyu, Chieh-Yang Huang, Chien-Kuang Cornelia Ding, Shaurya Rohatgi, and Ting-Hao Kenneth Huang. 2024. “If in a Crowdsourced Data Annotation Pipeline, a GPT-4.” In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI 2024, edited by Florian ’Floyd’Mueller, Penny Kyburz, Julie R. Williamson, Corina Sas, Max L. Wilson, Phoebe O. Toups Dugas, and Irina Shklovski, 1040:1–25. ACM. https://doi.org/10.1145/3613904.3642834.
Holtzman, Ari, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. “The Curious Case of Neural Text Degeneration.” In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. https://openreview.net/forum?id=rygGQyrFvH.
Hou, Xinyi, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. “Large Language Models for Software Engineering: A Systematic Literature Review.”ACM Trans. Softw. Eng. Methodol. 33 (8): 1–79. https://doi.org/10.1145/3695988.
Hu, Xing, Feifei Niu, Junkai Chen, Xin Zhou, Junwei Zhang, Junda He, Xin Xia, and David Lo. 2025. “Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Tasks.”CoRR abs/2505.08903. https://doi.org/10.48550/ARXIV.2505.08903.
Huang, Fan, Haewoon Kwak, and Jisun An. 2023. “Is ChatGPT Better Than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech.” In Companion Proceedings of the ACM Web Conference 2023, WWW 2023, edited by Ying Ding, Jie Tang, Juan F. Sequeda, Lora Aroyo, Carlos Castillo, and Geert-Jan Houben, 294–97. ACM. https://doi.org/10.1145/3543873.3587368.
Huang, Jiangping, Bochen Yi, Weisong Sun, Bangrui Wan, Yang Xu, Yebo Feng, Wenguang Ye, and Qinjun Qin. 2024. “Enhancing Review Classification via LLM-Based Data Annotation and Multi-Perspective Feature Representation Learning.”SSRN Electronic Journal, 1–15. https://doi.org/10.2139/ssrn.5002351.
Husain, Hamel, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. “CodeSearchNet Challenge: Evaluating the State of Semantic Code Search.”CoRR abs/1909.09436. http://arxiv.org/abs/1909.09436.
Hymel, Cory, and Hiroe Johnson. 2025. “Analysis of LLMs Vs Human Experts in Requirements Engineering.”CoRR abs/2501.19297. https://doi.org/10.48550/ARXIV.2501.19297.
Jahic, Jasmin, and Ashkan Sami. 2024. “State of Practice: LLMs in Software Engineering and Software Architecture.” In 21st IEEE International Conference on Software Architecture, ICSA 2024 - Companion, Hyderabad, India, June 4-8, 2024, 311–18. IEEE. https://doi.org/10.1109/ICSA-C63560.2024.00059.
Jain, Naman, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. “LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code.”CoRR abs/2403.07974. https://doi.org/10.48550/ARXIV.2403.07974.
Jimenez, Carlos E., John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. “SWE-Bench: Can Language Models Resolve Real-World Github Issues?” In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum?id=VTF8yNQM66.
Jowsey, Tanisha, Virginia Braun, Victoria Clarke, Victoria Clarke, Deborah Lupton, and Michelle Fine. 2025. “We Reject the Use of Generative Artificial Intelligence for Reflexive Qualitative Research.”SSRN. https://doi.org/10.2139/ssrn.5676462.
Kang, Sungmin, Juyeon Yoon, and Shin Yoo. 2023. “Large Language Models Are Few-Shot Testers: Exploring LLM-Based General Bug Reproduction.” In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, 2312–23. IEEE. https://doi.org/10.1109/ICSE48619.2023.00194.
Khojah, Ranim, Mazen Mohamad, Philipp Leitner, and Francisco Gomes de Oliveira Neto. 2024. “Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice.”Proc. ACM Softw. Eng. 1 (FSE): 1819–40. https://doi.org/10.1145/3660788.
Khraisha, Qusai, Sophie Put, Johanna Kappenberg, Azza Warraitch, and Kristin Hadfield. 2024. “Can Large Language Models Replace Humans in Systematic Reviews? Evaluating GPT-4’s Efficacy in Screening and Extracting Data from Peer-Reviewed and Grey Literature in Multiple Languages.”Research Synthesis Methods 15 (4): 616–26. https://doi.org/10.1002/jrsm.1715.
Kistowski, Jóakim v., Jeremy A. Arnold, Karl Huppler, Klaus-Dieter Lange, John L. Henning, and Paul Cao. 2015. “How to Build a Benchmark.” In Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering, 333–36. ICPE ’15. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/2668930.2688819.
Krippendorff, Klaus. 2018. Content Analysis: An Introduction to Its Methodology. 4th ed. SAGE Publications.
Kübler, Jonas M., Kailash Budhathoki, Matthäus Kleindessner, Xiong Zhou, Junming Yin, Ashish Khetan, and George Karypis. 2026. “When LLMs Get Significantly Worse: A Statistical Approach to Detect Model Degradations.” In Proc. Of the International Conference on Learning Representations (ICLR).
Kulal, Sumith, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. 2019. “SPoC: Search-Based Pseudocode to Code.”CoRR abs/1906.04908. http://arxiv.org/abs/1906.04908.
Lachaux, Marie-Anne, Baptiste Rozière, Lowik Chanussot, and Guillaume Lample. 2020. “Unsupervised Translation of Programming Languages.”CoRR abs/2006.03511. https://arxiv.org/abs/2006.03511.
Li, David, Kartik Gupta, Mousumi Bhaduri, Paul Sathiadoss, Sahir Bhatnagar, and Jaron Chong. 2024. “Comparing GPT-3.5 and GPT-4 Accuracy and Drift in Radiology Diagnosis Please Cases.”Radiology 310 (1): e232411. https://doi.org/10.1148/radiol.232411.
Li, Raymond, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, et al. 2023. “StarCoder: May the Source Be with You!”CoRR abs/2305.06161. https://doi.org/10.48550/ARXIV.2305.06161.
Liang, Jenny T, Carmen Badea, Christian Bird, Robert DeLine, Denae Ford, Nicole Forsgren, and Thomas Zimmermann. 2024. “Can Gpt-4 Replicate Empirical Software Engineering Research?”Proceedings of the ACM on Software Engineering 1 (FSE): 1330–53.
Lu, Qinghua, Liming Zhu, Xiwei Xu, Zhenchang Xing, and Jon Whittle. 2024. “Toward Responsible AI in the Era of Generative AI:A Reference Architecture for Designing Foundation Model-Based Systems.”IEEE Softw. 41 (6): 91–100. https://doi.org/10.1109/MS.2024.3406333.
Lubos, Sebastian, Alexander Felfernig, Thi Ngoc Trang Tran, Damian Garber, Merfat El Mansi, Seda Polat Erdeniz, and Viet-Man Le. 2024. “Leveraging LLMs for the Quality Assurance of Software Requirements.” In 32nd IEEE International Requirements Engineering Conference, RE 2024, Reykjavik, Iceland, June 24-28, 2024, edited by Grischa Liebel, Irit Hadar, and Paola Spoletini, 389–97. IEEE. https://doi.org/10.1109/RE59067.2024.00046.
Madampe, Kashumi, John Grundy, Rashina Hoda, and Humphrey O. Obie. 2024. “The Struggle Is Real! The Agony of Recruiting Participants for Empirical Software Engineering Studies.” In 2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), Liverpool, UK, September 2-6, 2024, 417–22. IEEE. https://doi.org/10.1109/VL/HCC60511.2024.00065.
McDonald, Nora, Sarita Schoenebeck, and Andrea Forte. 2019. “Reliability and Inter-Rater Reliability in Qualitative Research: Norms and Guidelines for CSCW and HCI Practice.”Proc. ACM Hum. Comput. Interact. 3 (CSCW): 72:1–23. https://doi.org/10.1145/3359174.
Mitu, Narcis Eduard, and George Teodor Mitu. 2024. “The Hidden Cost of AI: Carbon Footprint and Mitigation Strategies.”Revista de Stiinte Politice. Revue Des Sciences Politiques 84: 9–16. https://doi.org/10.2139/ssrn.5036344.
Mohamed, Suad, Abdullah Parvin, and Esteban Parra. 2024. “Chatting with AI: Deciphering Developer Conversations with ChatGPT.” In 21st IEEE/ACM International Conference on Mining Software Repositories, MSR 2024, Lisbon, Portugal, April 15-16, 2024, edited by Diomidis Spinellis, Alberto Bacchelli, and Eleni Constantinou, 187–91. ACM. https://doi.org/10.1145/3643991.3645078.
Mohsenimofidi, Seyedmoein, Matthias Galster, Christoph Treude, and Sebastian Baltes. 2026. “Context Engineering for AI Agents in Open-Source Software.” In Proceedings of the 23rd IEEE/ACM International Conference on Mining Software Repositories (MSR 2026). https://arxiv.org/abs/2510.21413.
Montes, Cristina Martinez, Robert Feldt, Cristina Miguel Martos, Sofia Ouhbi, Shweta Premanandan, and Daniel Graziotin. 2025. “Large Language Models in Thematic Analysis: Prompt Engineering, Evaluation, and Guidelines for Qualitative Software Engineering Research.”CoRR abs/2510.18456. https://doi.org/10.48550/ARXIV.2510.18456.
Morais Leça, Matheus de, Lucas Valença, Reydne Santos, and Ronnie de Souza Santos. 2025. “Applications and Implications of Large Language Models in Qualitative Analysis: A New Frontier for Empirical Software Engineering.” In IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering, WSESE@ICSE 2025, Ottawa, ON, Canada, May 3, 2025, 36–43. IEEE. https://doi.org/10.1109/WSESE66602.2025.00013.
Moumoula, Micheline Bénédicte, Abdoul Kader Kaboré, Jacques Klein, and Tegawendé F. Bissyandé. 2024. “Large Language Models for Cross-Language Code Clone Detection.”CoRR abs/2408.04430. https://doi.org/10.48550/ARXIV.2408.04430.
Pan, Qian, Zahra Ashktorab, Michael Desmond, Martin Santillan Cooper, James Johnson, Rahul Nair, Elizabeth Daly, and Werner Geyer. 2024. “Human-Centered Design Recommendations for LLM-as-a-Judge.” In ACL 2024 Workshop HuCLLM. arXiv. https://doi.org/10.48550/arXiv.2407.03479.
Pangakis, Nicholas, Samuel Wolken, and Neil Fasching. 2023. “Automated Annotation with Generative AI Requires Validation.”CoRR abs/2306.00176. https://doi.org/10.48550/ARXIV.2306.00176.
Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. “Bleu: A Method for Automatic Evaluation of Machine Translation.” In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, 311–18. ACL. https://doi.org/10.3115/1073083.1073135.
Pezeshkpour, Pouya, and Estevam Hruschka. 2024. “Large Language Models Sensitivity to the Order of Options in Multiple-Choice Questions.” In Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024, edited by Kevin Duh, Helena Gómez-Adorno, and Steven Bethard, NAACL 2024:2006–17. Findings of ACL. Association for Computational Linguistics. https://doi.org/10.18653/V1/2024.FINDINGS-NAACL.130.
Rabbi, Md. Fazle, Arifa I. Champa, Minhaz Fahim Zibran, and Md. Rakibul Islam. 2024. “AI Writes, We Analyze: The ChatGPT Python Code Saga.” In 21st IEEE/ACM International Conference on Mining Software Repositories, MSR 2024, Lisbon, Portugal, April 15-16, 2024, edited by Diomidis Spinellis, Alberto Bacchelli, and Eleni Constantinou, 177–81. ACM. https://doi.org/10.1145/3643991.3645076.
Ralph, Paul, Nauman bin Ali, Sebastian Baltes, Domenico Bianculli, Jessica Diaz, Yvonne Dittrich, Neil Ernst, et al. 2021. “Empirical Standards for Software Engineering Research.”https://arxiv.org/abs/2010.03525.
Ralph, Paul, Miikka Kuutila, Hera Arif, and Bimpe Ayoola. 2024. “Teaching Software Metrology: The Science of Measurement for Software Engineering.” In Handbook on Teaching Empirical Software Engineering, edited by Daniel Méndez, Paris Avgeriou, Marcos Kalinowski, and Nauman Bin Ali, 101–54. Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-71769-7\_5.
Ralph, Paul, and Ewan D. Tempero. 2018. “Construct Validity in Software Engineering Research and Software Metrics.” In Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering, EASE2018, edited by Austen Rainer, Stephen G. MacDonell, and Jacky W. Keung, 13–23. ACM. https://doi.org/10.1145/3210459.3210461.
Rasheed, Zeeshan, Muhammad Waseem, Aakash Ahmad, Kai-Kristian Kemell, Xiaofeng Wang, Anh Nguyen-Duc, and Pekka Abrahamsson. 2024. “Can Large Language Models Serve as Data Analysts? A Multi-Agent Assisted Approach for Qualitative Data Analysis.”CoRR abs/2402.01386. https://doi.org/10.48550/ARXIV.2402.01386.
Reiss, Michael V. 2023. “Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark.”CoRR abs/2304.11085. https://doi.org/10.48550/ARXIV.2304.11085.
Ren, Shuo, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. “CodeBLEU: A Method for Automatic Evaluation of Code Synthesis.”CoRR abs/2009.10297. https://arxiv.org/abs/2009.10297.
Renze, Matthew. 2024. “The Effect of Sampling Temperature on Problem Solving in Large Language Models.” In Findings of the Association for Computational Linguistics: EMNLP 2024, 7346–56. https://doi.org/10.18653/v1/2024.findings-emnlp.432.
Richards, Jonan, and Mairieli Wessel. 2024. “What You Need Is What You Get: Theory of Mind for an LLM-Based Code Understanding Assistant.” In IEEE International Conference on Software Maintenance and Evolution, ICSME 2024, 666–71. IEEE. https://doi.org/10.1109/ICSME58944.2024.00070.
Ronanki, Krishna, Christian Berger, and Jennifer Horkoff. 2023. “Investigating ChatGPT’s Potential to Assist in Requirements Elicitation Processes.” In 2023 49th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), 354–61. IEEE.
Rozière, Baptiste, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, et al. 2023. “Code Llama: Open Foundation Models for Code.”CoRR abs/2308.12950. https://doi.org/10.48550/ARXIV.2308.12950.
Russo, Daniel. 2024. “Navigating the Complexity of Generative AI Adoption in Software Engineering.”ACM Transactions on Software Engineering and Methodology 33 (5): 1–50.
Sallou, June, Thomas Durieux, and Annibale Panichella. 2024. “Breaking the Silence: The Threats of Using LLMs in Software Engineering.” In Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results, NIER@ICSE 2024, Lisbon, Portugal, April 14-20, 2024, 102–6. ACM. https://doi.org/10.1145/3639476.3639764.
Santos, Adrian, Sira Vegas, Markku Oivo, and Natalia Juristo. 2021. “A Procedure and Guidelines for Analyzing Groups of Software Engineering Replications.”IEEE Trans. Software Eng. 47 (9): 1742–63. https://doi.org/10.1109/TSE.2019.2935720.
Schäfer, Max, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024. “An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation.”IEEE Trans. Software Eng. 50 (1): 85–105. https://doi.org/10.1109/TSE.2023.3334955.
Schilke, Oliver, and Martin Reimann. 2025. “The Transparency Dilemma: How AI Disclosure Erodes Trust.”Organizational Behavior and Human Decision Processes 188: 104405. https://doi.org/10.1016/j.obhdp.2025.104405.
Schneider, Kurt, Farnaz Fotrousi, and Rebekka Wohlrab. 2025. “A Reference Model for Empirically Comparing LLMs with Humans.” In 47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Society, ICSE-SEIS 2025, Ottawa, ON, Canada, April 27 - May 3, 2025, 130–34. IEEE. https://doi.org/10.1109/ICSE-SEIS66351.2025.00018.
Schröder, Sarah, Thekla Morgenroth, Ulrike Kuhl, Valerie Vaquet, and Benjamin Paaßen. 2025. “Large Language Models Do Not Simulate Human Psychology.”CoRR abs/2508.06950. https://doi.org/10.48550/ARXIV.2508.06950.
Schroeder, Kayla, and Zach Wood-Doughty. 2024. “Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge.”CoRR abs/2412.12509. https://doi.org/10.48550/ARXIV.2412.12509.
Schulhoff, Sander, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, et al. 2024. “The Prompt Report: A Systematic Survey of Prompting Techniques.”CoRR abs/2406.06608. https://doi.org/10.48550/ARXIV.2406.06608.
Schulz, Kenneth F., Douglas G. Altman, and David Moher. 2010. “CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomised Trials.”BMJ 340: c332. https://doi.org/10.1136/bmj.c332.
Schwartz, Roy, Jesse Dodge, Noah A. Smith, and Oren Etzioni. 2020. “Green AI.”Communications of the ACM 63 (12): 54–63. https://doi.org/10.1145/3381831.
Sclar, Melanie, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. “Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I Learned to Start Worrying about Prompt Formatting.” In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum?id=RIu5lyNXjT.
Shull, Forrest, Janice Singer, and Dag I. K. Sjøberg, eds. 2008. Guide to Advanced Empirical Software Engineering. Springer. https://doi.org/10.1007/978-1-84800-044-5.
Silva, André, and Martin Monperrus. 2025. “RepairBench: Leaderboard of Frontier Models for Program Repair.” In IEEE/ACM International Workshop on Large Language Models for Code, LLM4Code@ICSE 2025, Ottawa, ON, Canada, May 3, 2025, 9–16. IEEE. https://doi.org/10.1109/LLM4CODE66737.2025.00006.
Sjøberg, Dag I. K., and Gunnar Rye Bergersen. 2023. “Construct Validity in Software Engineering.”IEEE Trans. Software Eng. 49 (3): 1374–96. https://doi.org/10.1109/TSE.2022.3176725.
Song, Yifan, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. 2025. “The Good, the Bad, and the Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism.” In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2025 - Volume 1: Long Papers, edited by Luis Chiruzzo, Alan Ritter, and Lu Wang, 4195–4206. Association for Computational Linguistics. https://doi.org/10.18653/V1/2025.NAACL-LONG.211.
Sonnekalb, Tim, Bernd Gruner, Clemens-Alexander Brust, and Patrick Mäder. 2022. “Generalizability of Code Clone Detection on CodeBERT.” In 37th IEEE/ACM International Conference on Automated Software Engineering, ASE 2022, 143:1–3. ACM. https://doi.org/10.1145/3551349.3561165.
Strubell, Emma, Ananya Ganesh, and Andrew McCallum. 2019. “Energy and Policy Considerations for Deep Learning in NLP.” In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 3645–50. https://doi.org/10.18653/v1/P19-1355.
Sullivan, Gail M, and Joan Sargeant. 2011. “Qualities of Qualitative Research: Part I.”J Grad Med Educ 3 (4): 449–52.
Sumers, Theodore R., Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. 2024. “Cognitive Architectures for Language Agents.”Trans. Mach. Learn. Res. 2024. https://openreview.net/forum?id=1i6ZCvflQJ.
Takerngsaksiri, Wannita, Jirat Pasuksmit, Patanamon Thongtanunam, Chakkrit Tantithamthavorn, Ruixiong Zhang, Fan Jiang, Jing Li, Evan Cook, Kun Chen, and Ming Wu. 2025. “Human-in-the-Loop Software Development Agents.” In 47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice, SEIP@ICSE 2025, Ottawa, ON, Canada, April 27 - May 3, 2025, 342–52. IEEE. https://doi.org/10.1109/ICSE-SEIP66354.2025.00036.
Tam, Thomas Yu Chow, Sonish Sivarajkumar, Sumit Kapoor, Alisa V Stolyar, Katelyn Polanska, Karleigh R McCarthy, Hunter Osterhoudt, et al. 2024. “A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Review.”NPJ Digital Medicine 7 (1): 258.
Wagner, Stefan, Marvin Muñoz Barón, Davide Falessi, and Sebastian Baltes. 2025. “Towards Evaluation Guidelines for Empirical Studies Involving LLMs.” In IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering, WSESE@ICSE 2025, May 3, 2025, 24–27. IEEE. https://doi.org/10.1109/WSESE66602.2025.00011.
Wan, Mengting, Tara Safavi, Sujay Kumar Jauhar, Yujin Kim, Scott Counts, Jennifer Neville, Siddharth Suri, et al. 2024. “TnT-LLM: Text Mining at Scale with Large Language Models.” In Proceedings of the 30th ACMSIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2024, edited by Ricardo Baeza-Yates and Francesco Bonchi, 5836–47. ACM. https://doi.org/10.1145/3637528.3671647.
Wang, Angelina, Jamie Morgenstern, and John P. Dickerson. 2025. “Large Language Models That Replace Human Participants Can Harmfully Misportray and Flatten Identity Groups.”Nature Machine Intelligence 7: 400–411. https://doi.org/10.1038/s42256-025-00986-z.
Wang, Chong, Kaifeng Huang, Jian Zhang, Yebo Feng, Lyuye Zhang, Yang Liu, and Xin Peng. 2024. “How and Why LLMs Use Deprecated APIs in Code Completion? An Empirical Study.”CoRR abs/2406.09834. https://doi.org/10.48550/ARXIV.2406.09834.
Wang, Fanyu, Chetan Arora, Yonghui Liu, Kaicheng Huang, Chakkrit Tantithamthavorn, Aldeida Aleti, Dishan Sambathkumar, and David Lo. 2025. “Multi-Modal Requirements Data-Based Acceptance Criteria Generation Using LLMs.” In 40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025, 3334–45. IEEE. https://doi.org/10.1109/ASE63991.2025.00275.
Wang, Shuohang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. “Want to Reduce Labeling Cost? GPT-3 Can Help.” In Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, edited by Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, EMNLP 2021:4195–205. Findings of ACL. Association for Computational Linguistics. https://doi.org/10.18653/V1/2021.FINDINGS-EMNLP.354.
Wang, Xinru, Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjie Miao. 2024. “Human-LLM Collaborative Annotation Through Effective Verification of LLM Labels.” In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI 2024, edited by Florian ’Floyd’Mueller, Penny Kyburz, Julie R. Williamson, Corina Sas, Max L. Wilson, Phoebe O. Toups Dugas, and Irina Shklovski, 303:1–21. ACM. https://doi.org/10.1145/3613904.3641960.
Widder, David Gray, Meredith Whittaker, and Sarah Myers West. 2024. “Why ‘Open’AI Systems Are Actually Closed, and Why This Matters.”Nature 635 (8040): 827–33.
Wiesinger, Julia, Patrick Marlow, and Vladimir Vuskovic. 2025. “Agents.”Google DeepMind. https://gemini.google.com.
Wohlin, Claes, Per Runeson, Martin Höst, Magnus C. Ohlsson, Björn Regnell, and Anders Wesslén. 2024. Experimentation in Software Engineering, Second Edition. Springer. https://doi.org/10.1007/978-3-662-69306-3.
Xia, Yuying, Haijian Shao, and Xing Deng. 2024. “VulCoBERT: A CodeBERT-Based System for Source Code Vulnerability Detection.” In 2024 International Conference on Generative Artificial Intelligence and Information Security, GAIIS 2024, Kuala Lumpur, Malaysia, May 10-12, 2024. ACM. https://doi.org/10.1145/3665348.3665391.
… Unchanged text
Xiao, Tao, Christoph Treude, Hideaki Hata, and Kenichi Matsumoto. 2024. “DevGPT: Studying Developer-ChatGPT Conversations.” In 21st IEEE/ACM International Conference on Mining Software Repositories, MSR 2024, Lisbon, Portugal, April 15-16, 2024, edited by Diomidis Spinellis, Alberto Bacchelli, and Eleni Constantinou, 227–30. ACM. https://doi.org/10.1145/3643991.3648400.
Xu, Cheng, Shuhao Guan, Derek Greene, and M. Tahar Kechadi. 2024. “Benchmark Data Contamination of Large Language Models: A Survey.”CoRR abs/2406.04244. https://doi.org/10.48550/ARXIV.2406.04244.
Xu, Ruoxi, Yingfei Sun, Mengjie Ren, Shiguang Guo, Ruotong Pan, Hongyu Lin, Le Sun, and Xianpei Han. 2024. “AI for Social Science and Social Science of AI:A Survey.”Inf. Process. Manag. 61 (2): 103665. https://doi.org/10.1016/J.IPM.2024.103665.
Yan, Litao, Alyssa Hwang, Zhiyuan Wu, and Andrew Head. 2024. “Ivie: Lightweight Anchored Explanations of Just-Generated Code.” In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI 2024, edited by Florian ’Floyd’Mueller, Penny Kyburz, Julie R. Williamson, Corina Sas, Max L. Wilson, Phoebe O. Toups Dugas, and Irina Shklovski, 140:1–15. ACM. https://doi.org/10.1145/3613904.3642239.
Yang, Guang, Yu Zhou, Xiang Chen, Xiangyu Zhang, Tingting Han, and Taolue Chen. 2023. “ExploitGen: Template-Augmented Exploit Code Generation Based on CodeBERT.”J. Syst. Softw. 197: 111577. https://doi.org/10.1016/J.JSS.2022.111577.
Yang, Yifan, Jiho Shin, Byeonggyu Choi, Minjun Park, Dayun Ju, Changmin Lee, Sanghyuk Chun, Dongjin Kang, Jiin Kim, and Sungroh Yoon. 2025. “From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence.”arXiv Preprint arXiv:2502.11827. https://arxiv.org/abs/2502.11827.
Yuan, Jiayi, Hao Li, Xinheng Ding, Wenya Xie, Yu-Jhe Li, Wentian Zhao, Kun Wan, Jing Shi, Xia Hu, and Zirui Liu. 2025. “Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference.” In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025. https://openreview.net/forum?id=Q3qAsZAEZw.
Zhao, Chenguang, Meirewuti Habule, and Wei Zhang. 2025. “Large Language Models (LLMs) as Research Subjects: Status, Opportunities and Challenges.”New Ideas in Psychology 79: 101167. https://doi.org/10.1016/j.newideapsych.2025.101167.
Zhou, Ruiyang, Lu Chen, and Kai Yu. 2024. “Is LLM a Reliable Reviewer? A Comprehensive Evaluation of LLM on Automatic Paper Reviewing Tasks.” In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, edited by Nicoletta Calzolari, Min-Yen Kan, Véronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, 9340–51. ELRA; ICCL. https://aclanthology.org/2024.lrec-main.816.
Zhou, Wangchunshu, Yuchen Eleanor Jiang, Long Li, Jialong Wu, Tiannan Wang, Shi Qiu, Jintian Zhang, et al. 2023. “Agents: An Open-Source Framework for Autonomous Language Agents.”CoRR abs/2309.07870. https://doi.org/10.48550/ARXIV.2309.07870.
Zhu, Yiming, Peixian Zhang, Ehsan ul Haq, Pan Hui, and Gareth Tyson. 2023. “Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks.”CoRR abs/2304.10145. https://doi.org/10.48550/ARXIV.2304.10145.
Agarwal, Rishabh, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Marc G. Bellemare. 2021. “Deep Reinforcement Learning at the Edge of the Statistical Precipice.” In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, Virtual, edited by Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, 29304–20. https://proceedings.neurips.cc/paper/2021/hash/f514cec81cb148559cf475e7426eed5e-Abstract.html.
Ahmed, Nur, Muntasir Wahed, and Neil C. Thompson. 2023. “The Growing Influence of Industry in AI Research.”Science 379 (6635): 884–86. https://doi.org/10.1126/science.ade2420.
Ahmed, Toufique, Premkumar T. Devanbu, Christoph Treude, and Michael Pradel. 2025. “Can LLMs Replace Manual Annotation of Software Engineering Artifacts?” In 22nd IEEE/ACM International Conference on Mining Software Repositories, MSR@ICSE 2025, Ottawa, ON, Canada, April 28-29, 2025, 526–38. IEEE. https://doi.org/10.1109/MSR66628.2025.00086.
Ahuja, Sanchit, Varun Gumma, and Sunayana Sitaram. 2024. “Contamination Report for Multilingual Benchmarks.”CoRR abs/2410.16186. https://doi.org/10.48550/ARXIV.2410.16186.
Angermeir, Florian, Maximilian Amougou, Mark Kreitz, Andreas Bauer, Matthias Linhuber, Davide Fucci, Fabiola Moyón Constante, Daniel Méndez, and Tony Gorschek. 2025. “Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies.”CoRR abs/2510.25506. https://doi.org/10.48550/ARXIV.2510.25506.
Argyle, Lisa P., Ethan C. Busby, Nancy Fulda, Joshua Gubler, Christopher Michael Rytting, and David Wingate. 2022. “Out of One, Many: Using Language Models to Simulate Human Samples.”CoRR abs/2209.06899. https://doi.org/10.48550/ARXIV.2209.06899.
Austin, Jacob, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, et al. 2021. “Program Synthesis with Large Language Models.”CoRR abs/2108.07732. https://arxiv.org/abs/2108.07732.
Azanza, Maider, Juanan Pereira, Arantza Irastorza, and Aritz Galdos. 2024. “Can LLMs Facilitate Onboarding Software Developers? An Ongoing Industrial Case Study.” In 36th International Conference on Software Engineering Education and Training, CSEE&t 2024, 1–6. IEEE. https://doi.org/10.1109/CSEET62301.2024.10662989.
Bano, Muneera, Hashini Gunatilake, and Rashina Hoda. 2025. “What Does a Software Engineer Look Like? Exploring Societal Stereotypes in LLMs.” In 47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Society, ICSE-SEIS 2025, Ottawa, ON, Canada, April 27 - May 3, 2025, 173–84. IEEE. https://doi.org/10.1109/ICSE-SEIS66351.2025.00023.
Bano, Muneera, Rashina Hoda, Didar Zowghi, and Christoph Treude. 2024. “Large Language Models for Qualitative Research in Software Engineering: Exploring Opportunities and Challenges.”Autom. Softw. Eng. 31 (1): 8. https://doi.org/10.1007/S10515-023-00407-8.
Barros, Cauã Ferreira, Bruna Borges Azevedo, Valdemar Vicente Graciano Neto, Mohamad Kassab, Marcos Kalinowski, Hugo Alexandre Dantas do Nascimento, and Michelle C. G. S. P. Bandeira. 2025. “Large Language Model for Qualitative Research: A Systematic Mapping Study.” In IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering, WSESE@ICSE 2025, Ottawa, ON, Canada, May 3, 2025, 48–55. IEEE. https://doi.org/10.1109/WSESE66602.2025.00015.
Bavaresco, Anna, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, et al. 2024. “LLMs Instead of Human Judges? A Large Scale Empirical Study Across 20 NLP Evaluation Tasks.”CoRR abs/2406.18403. https://doi.org/10.48550/ARXIV.2406.18403.
Begg, C., M. Cho, S. Eastwood, R. Horton, D. Moher, I. Olkin, R. Pitkin, et al. 1996. “Improving the Quality of Reporting of Randomized Controlled Trials. The CONSORT statement.”JAMA 276 (8): 637–39. https://doi.org/10.1001/jama.276.8.637.
Bjarnason, Bjarni Haukur, André Silva, and Martin Monperrus. 2026. “On Randomness in Agentic Evals.”https://arxiv.org/abs/2602.07150.
Blackwell, Robert E., Jon Barry, and Anthony G. Cohn. 2024. “Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scores.”https://arxiv.org/abs/2410.03492.
Borsboom, Denny. 2005. Measuring the Mind: Conceptual Issues in Contemporary Psychometrics. Cambridge University Press. https://doi.org/10.1017/CBO9780511490026.
Cai, Yuchen, Aashish Yadavally, Abhishek Mishra, Genesis Montejo, and Tien N. Nguyen. 2024. “Programming Assistant for Exception Handling with CodeBERT.” In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024, 94:1–13. ACM. https://doi.org/10.1145/3597503.3639188.
Cao, Jialun, Yuk-Kit Chan, Zixuan Ling, Wenxuan Wang, Shuqing Li, Mingwei Liu, Ruixi Qiao, et al. 2025. “How Should We Build a Benchmark? Revisiting 274 Code-Related Benchmarks for LLMs.”CoRR abs/2501.10711. https://arxiv.org/abs/2501.10711.
Chen, Lingjiao, Matei Zaharia, and James Zou. 2024. “How Is ChatGPT’s Behavior Changing over Time?”Harvard Data Science Review 6 (2). https://doi.org/10.1162/99608f92.5317da47.
Chen, Mark, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, et al. 2021. “Evaluating Large Language Models Trained on Code.”CoRR abs/2107.03374. https://arxiv.org/abs/2107.03374.
Chen, Yizheng, Zhoujie Ding, Lamya Alowain, Xinyun Chen, and David A. Wagner. 2023. “DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection.” In Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses, RAID 2023, Hong Kong, China, October 16-18, 2023, 654–68. ACM. https://doi.org/10.1145/3607199.3607242.
Choi, Hyeong Kyu, Maxim Khanov, Hongxin Wei, and Yixuan Li. 2025. “How Contaminated Is Your Benchmark? Quantifying Dataset Leakage in Large Language Models with Kernel Divergence.”CoRR abs/2502.00678. https://doi.org/10.48550/ARXIV.2502.00678.
Dhar, Rudra, Karthik Vaidhyanathan, and Vasudeva Varma. 2024. “Can LLMs Generate Architectural Design Decisions? - an Exploratory Empirical Study.” In 21st IEEE International Conference on Software Architecture, ICSA 2024, Hyderabad, India, June 4-8, 2024, 79–89. IEEE. https://doi.org/10.1109/ICSA59870.2024.00016.
… Unchanged text
Du, Xueying, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2023. “ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-Level Code Generation.”CoRR abs/2308.01861. https://doi.org/10.48550/ARXIV.2308.01861.
Dybå, Tore, Vigdis By Kampenes, and Dag I. K. Sjøberg. 2006. “A Systematic Review of Statistical Power in Software Engineering Experiments.”Inf. Softw. Technol. 48 (8): 745–55. https://doi.org/10.1016/J.INFSOF.2005.08.009.
Eghbali, Aryaz, and Michael Pradel. 2022. “CrystalBLEU: Precisely and Efficiently Measuring the Similarity of Code.” In 37th IEEE/ACM International Conference on Automated Software Engineering, ASE 2022, 28:1–12. ACM. https://doi.org/10.1145/3551349.3556903.
El-Hajjami, Abdelkarim, and Camille Salinesi. 2025. “How Good Are Synthetic Requirements? Evaluating LLM-Generated Datasets for AI4RE.”CoRR abs/2506.21138. https://doi.org/10.48550/ARXIV.2506.21138.
Gallegos, Isabel O., Ryan A. Rossi, Joe Barrow, Md. Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen Ahmed. 2024. “Bias and Fairness in Large Language Models: A Survey.”Computational Linguistics 50: 1097–1179. https://doi.org/10.1162/coli_a_00524.
Gallifant, Jack, Majid Afshar, Saleem Ameen, Yindalon Aphinyanaphongs, Shan Chen, Giovanni Cacciamani, Dina Demner-Fushman, et al. 2025. “The TRIPOD-LLM Reporting Guideline for Studies Using Large Language Models.”Nature Medicine 31 (1): 60–69. https://doi.org/10.1038/s41591-024-03425-5.
Galster, Matthias, Seyedmoein Mohsenimofidi, Jai Lal Lulla, Muhammad Auwal Abubakar, Christoph Treude, and Sebastian Baltes. 2026. “Configuring Agentic AI Coding Tools: An Exploratory Study.” In Proceedings of the 3rd ACM International Conference on AI-Powered Software (AIware 2026).
Gerosa, Marco Aurélio, Bianca Trinkenreich, Igor Steinmacher, and Anita Sarma. 2024. “Can AI Serve as a Substitute for Human Subjects in Software Engineering Research?”Autom. Softw. Eng. 31 (1): 13. https://doi.org/10.1007/S10515-023-00409-6.
Gilardi, Fabrizio, Meysam Alizadeh, and Maël Kubli. 2023. “ChatGPT Outperforms Crowd Workers for Text-Annotation Tasks.”Proceedings of the National Academy of Sciences 120 (30): e2305016120. https://doi.org/10.1073/pnas.2305016120.
Gonçalves, José, Miguel Silva, Bernardo Cabral, Tiago Dias, Eva Maia, Isabel Praça, Ricardo Severino, and Luı́s Lino Ferreira. 2025. “Evaluating LLaMA 3.2 for Software Vulnerability Detection.” In Cybersecurity - 9th European Interdisciplinary Cybersecurity Conference, EICC 2025, Rennes, France, June 18-19, 2025, Proceedings, edited by Isabel Praça, Simona Bernardi, and Pedro R. M. Inácio, 2500:38–51. Communications in Computer and Information Science. Springer. https://doi.org/10.1007/978-3-031-94855-8\_3.
Guba, Egon G. 1981. “Criteria for Assessing the Trustworthiness of Naturalistic Inquiries.”ECTJ 29 (2): 75–91. https://doi.org/10.1007/BF02766777.
Guo, Daya, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, et al. 2024. “DeepSeek-Coder: When the Large Language Model Meets Programming - the Rise of Code Intelligence.”CoRR abs/2401.14196. https://doi.org/10.48550/ARXIV.2401.14196.
Harding, Jacqueline, William D’Alessandro, N. G. Laskowski, and Robert Long. 2024. “AI Language Models Cannot Replace Human Research Participants.”AI Soc. 39 (5): 2603–5. https://doi.org/10.1007/S00146-023-01725-X.
He, Junda, Jieke Shi, Terry Yue Zhuo, Christoph Treude, Jiamou Sun, Zhenchang Xing, Xiaoning Du, and David Lo. 2025. “LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead.”CoRR abs/2510.24367. https://doi.org/10.48550/ARXIV.2510.24367.
He, Zeyu, Chieh-Yang Huang, Chien-Kuang Cornelia Ding, Shaurya Rohatgi, and Ting-Hao Kenneth Huang. 2024. “If in a Crowdsourced Data Annotation Pipeline, a GPT-4.” In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI 2024, edited by Florian ’Floyd’Mueller, Penny Kyburz, Julie R. Williamson, Corina Sas, Max L. Wilson, Phoebe O. Toups Dugas, and Irina Shklovski, 1040:1–25. ACM. https://doi.org/10.1145/3613904.3642834.
Holtzman, Ari, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. “The Curious Case of Neural Text Degeneration.” In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. https://openreview.net/forum?id=rygGQyrFvH.
Hou, Xinyi, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. “Large Language Models for Software Engineering: A Systematic Literature Review.”ACM Trans. Softw. Eng. Methodol. 33 (8): 1–79. https://doi.org/10.1145/3695988.
Hu, Xing, Feifei Niu, Junkai Chen, Xin Zhou, Junwei Zhang, Junda He, Xin Xia, and David Lo. 2025. “Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Tasks.”CoRR abs/2505.08903. https://doi.org/10.48550/ARXIV.2505.08903.
Huang, Fan, Haewoon Kwak, and Jisun An. 2023. “Is ChatGPT Better Than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech.” In Companion Proceedings of the ACM Web Conference 2023, WWW 2023, edited by Ying Ding, Jie Tang, Juan F. Sequeda, Lora Aroyo, Carlos Castillo, and Geert-Jan Houben, 294–97. ACM. https://doi.org/10.1145/3543873.3587368.
Huang, Jiangping, Bochen Yi, Weisong Sun, Bangrui Wan, Yang Xu, Yebo Feng, Wenguang Ye, and Qinjun Qin. 2024. “Enhancing Review Classification via LLM-Based Data Annotation and Multi-Perspective Feature Representation Learning.”SSRN Electronic Journal, 1–15. https://doi.org/10.2139/ssrn.5002351.
Husain, Hamel, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. “CodeSearchNet Challenge: Evaluating the State of Semantic Code Search.”CoRR abs/1909.09436. http://arxiv.org/abs/1909.09436.
Hymel, Cory, and Hiroe Johnson. 2025. “Analysis of LLMs Vs Human Experts in Requirements Engineering.”CoRR abs/2501.19297. https://doi.org/10.48550/ARXIV.2501.19297.
Jahic, Jasmin, and Ashkan Sami. 2024. “State of Practice: LLMs in Software Engineering and Software Architecture.” In 21st IEEE International Conference on Software Architecture, ICSA 2024 - Companion, Hyderabad, India, June 4-8, 2024, 311–18. IEEE. https://doi.org/10.1109/ICSA-C63560.2024.00059.
Jain, Naman, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. “LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code.”CoRR abs/2403.07974. https://doi.org/10.48550/ARXIV.2403.07974.
Jimenez, Carlos E., John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. “SWE-Bench: Can Language Models Resolve Real-World Github Issues?” In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum?id=VTF8yNQM66.
Jowsey, Tanisha, Virginia Braun, Victoria Clarke, Victoria Clarke, Deborah Lupton, and Michelle Fine. 2025. “We Reject the Use of Generative Artificial Intelligence for Reflexive Qualitative Research.”SSRN. https://doi.org/10.2139/ssrn.5676462.
Kang, Sungmin, Juyeon Yoon, and Shin Yoo. 2023. “Large Language Models Are Few-Shot Testers: Exploring LLM-Based General Bug Reproduction.” In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, 2312–23. IEEE. https://doi.org/10.1109/ICSE48619.2023.00194.
Khojah, Ranim, Mazen Mohamad, Philipp Leitner, and Francisco Gomes de Oliveira Neto. 2024. “Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice.”Proc. ACM Softw. Eng. 1 (FSE): 1819–40. https://doi.org/10.1145/3660788.
Khraisha, Qusai, Sophie Put, Johanna Kappenberg, Azza Warraitch, and Kristin Hadfield. 2024. “Can Large Language Models Replace Humans in Systematic Reviews? Evaluating GPT-4’s Efficacy in Screening and Extracting Data from Peer-Reviewed and Grey Literature in Multiple Languages.”Research Synthesis Methods 15 (4): 616–26. https://doi.org/10.1002/jrsm.1715.
Kistowski, Jóakim v., Jeremy A. Arnold, Karl Huppler, Klaus-Dieter Lange, John L. Henning, and Paul Cao. 2015. “How to Build a Benchmark.” In Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering, 333–36. ICPE ’15. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/2668930.2688819.
Krippendorff, Klaus. 2018. Content Analysis: An Introduction to Its Methodology. 4th ed. SAGE Publications.
Kübler, Jonas M., Kailash Budhathoki, Matthäus Kleindessner, Xiong Zhou, Junming Yin, Ashish Khetan, and George Karypis. 2026. “When LLMs Get Significantly Worse: A Statistical Approach to Detect Model Degradations.” In Proc. Of the International Conference on Learning Representations (ICLR).
Kulal, Sumith, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. 2019. “SPoC: Search-Based Pseudocode to Code.”CoRR abs/1906.04908. http://arxiv.org/abs/1906.04908.
Lachaux, Marie-Anne, Baptiste Rozière, Lowik Chanussot, and Guillaume Lample. 2020. “Unsupervised Translation of Programming Languages.”CoRR abs/2006.03511. https://arxiv.org/abs/2006.03511.
Li, David, Kartik Gupta, Mousumi Bhaduri, Paul Sathiadoss, Sahir Bhatnagar, and Jaron Chong. 2024. “Comparing GPT-3.5 and GPT-4 Accuracy and Drift in Radiology Diagnosis Please Cases.”Radiology 310 (1): e232411. https://doi.org/10.1148/radiol.232411.
Li, Raymond, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, et al. 2023. “StarCoder: May the Source Be with You!”CoRR abs/2305.06161. https://doi.org/10.48550/ARXIV.2305.06161.
Liang, Jenny T, Carmen Badea, Christian Bird, Robert DeLine, Denae Ford, Nicole Forsgren, and Thomas Zimmermann. 2024. “Can Gpt-4 Replicate Empirical Software Engineering Research?”Proceedings of the ACM on Software Engineering 1 (FSE): 1330–53.
Lu, Qinghua, Liming Zhu, Xiwei Xu, Zhenchang Xing, and Jon Whittle. 2024. “Toward Responsible AI in the Era of Generative AI:A Reference Architecture for Designing Foundation Model-Based Systems.”IEEE Softw. 41 (6): 91–100. https://doi.org/10.1109/MS.2024.3406333.
Lubos, Sebastian, Alexander Felfernig, Thi Ngoc Trang Tran, Damian Garber, Merfat El Mansi, Seda Polat Erdeniz, and Viet-Man Le. 2024. “Leveraging LLMs for the Quality Assurance of Software Requirements.” In 32nd IEEE International Requirements Engineering Conference, RE 2024, Reykjavik, Iceland, June 24-28, 2024, edited by Grischa Liebel, Irit Hadar, and Paola Spoletini, 389–97. IEEE. https://doi.org/10.1109/RE59067.2024.00046.
Madampe, Kashumi, John Grundy, Rashina Hoda, and Humphrey O. Obie. 2024. “The Struggle Is Real! The Agony of Recruiting Participants for Empirical Software Engineering Studies.” In 2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), Liverpool, UK, September 2-6, 2024, 417–22. IEEE. https://doi.org/10.1109/VL/HCC60511.2024.00065.
McDonald, Nora, Sarita Schoenebeck, and Andrea Forte. 2019. “Reliability and Inter-Rater Reliability in Qualitative Research: Norms and Guidelines for CSCW and HCI Practice.”Proc. ACM Hum. Comput. Interact. 3 (CSCW): 72:1–23. https://doi.org/10.1145/3359174.
Mitu, Narcis Eduard, and George Teodor Mitu. 2024. “The Hidden Cost of AI: Carbon Footprint and Mitigation Strategies.”Revista de Stiinte Politice. Revue Des Sciences Politiques 84: 9–16. https://doi.org/10.2139/ssrn.5036344.
Mohamed, Suad, Abdullah Parvin, and Esteban Parra. 2024. “Chatting with AI: Deciphering Developer Conversations with ChatGPT.” In 21st IEEE/ACM International Conference on Mining Software Repositories, MSR 2024, Lisbon, Portugal, April 15-16, 2024, edited by Diomidis Spinellis, Alberto Bacchelli, and Eleni Constantinou, 187–91. ACM. https://doi.org/10.1145/3643991.3645078.
Mohsenimofidi, Seyedmoein, Matthias Galster, Christoph Treude, and Sebastian Baltes. 2026. “Context Engineering for AI Agents in Open-Source Software.” In Proceedings of the 23rd IEEE/ACM International Conference on Mining Software Repositories (MSR 2026). https://arxiv.org/abs/2510.21413.
Montes, Cristina Martinez, Robert Feldt, Cristina Miguel Martos, Sofia Ouhbi, Shweta Premanandan, and Daniel Graziotin. 2025. “Large Language Models in Thematic Analysis: Prompt Engineering, Evaluation, and Guidelines for Qualitative Software Engineering Research.”CoRR abs/2510.18456. https://doi.org/10.48550/ARXIV.2510.18456.
Morais Leça, Matheus de, Lucas Valença, Reydne Santos, and Ronnie de Souza Santos. 2025. “Applications and Implications of Large Language Models in Qualitative Analysis: A New Frontier for Empirical Software Engineering.” In IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering, WSESE@ICSE 2025, Ottawa, ON, Canada, May 3, 2025, 36–43. IEEE. https://doi.org/10.1109/WSESE66602.2025.00013.
Moumoula, Micheline Bénédicte, Abdoul Kader Kaboré, Jacques Klein, and Tegawendé F. Bissyandé. 2024. “Large Language Models for Cross-Language Code Clone Detection.”CoRR abs/2408.04430. https://doi.org/10.48550/ARXIV.2408.04430.
Pan, Qian, Zahra Ashktorab, Michael Desmond, Martin Santillan Cooper, James Johnson, Rahul Nair, Elizabeth Daly, and Werner Geyer. 2024. “Human-Centered Design Recommendations for LLM-as-a-Judge.” In ACL 2024 Workshop HuCLLM. arXiv. https://doi.org/10.48550/arXiv.2407.03479.
Pangakis, Nicholas, Samuel Wolken, and Neil Fasching. 2023. “Automated Annotation with Generative AI Requires Validation.”CoRR abs/2306.00176. https://doi.org/10.48550/ARXIV.2306.00176.
Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. “Bleu: A Method for Automatic Evaluation of Machine Translation.” In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, 311–18. ACL. https://doi.org/10.3115/1073083.1073135.
Pezeshkpour, Pouya, and Estevam Hruschka. 2024. “Large Language Models Sensitivity to the Order of Options in Multiple-Choice Questions.” In Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024, edited by Kevin Duh, Helena Gómez-Adorno, and Steven Bethard, NAACL 2024:2006–17. Findings of ACL. Association for Computational Linguistics. https://doi.org/10.18653/V1/2024.FINDINGS-NAACL.130.
Rabbi, Md. Fazle, Arifa I. Champa, Minhaz Fahim Zibran, and Md. Rakibul Islam. 2024. “AI Writes, We Analyze: The ChatGPT Python Code Saga.” In 21st IEEE/ACM International Conference on Mining Software Repositories, MSR 2024, Lisbon, Portugal, April 15-16, 2024, edited by Diomidis Spinellis, Alberto Bacchelli, and Eleni Constantinou, 177–81. ACM. https://doi.org/10.1145/3643991.3645076.
Ralph, Paul, Nauman bin Ali, Sebastian Baltes, Domenico Bianculli, Jessica Diaz, Yvonne Dittrich, Neil Ernst, et al. 2021. “Empirical Standards for Software Engineering Research.”https://arxiv.org/abs/2010.03525.
Ralph, Paul, Miikka Kuutila, Hera Arif, and Bimpe Ayoola. 2024. “Teaching Software Metrology: The Science of Measurement for Software Engineering.” In Handbook on Teaching Empirical Software Engineering, edited by Daniel Méndez, Paris Avgeriou, Marcos Kalinowski, and Nauman Bin Ali, 101–54. Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-71769-7\_5.
Ralph, Paul, and Ewan D. Tempero. 2018. “Construct Validity in Software Engineering Research and Software Metrics.” In Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering, EASE2018, edited by Austen Rainer, Stephen G. MacDonell, and Jacky W. Keung, 13–23. ACM. https://doi.org/10.1145/3210459.3210461.
Rasheed, Zeeshan, Muhammad Waseem, Aakash Ahmad, Kai-Kristian Kemell, Xiaofeng Wang, Anh Nguyen-Duc, and Pekka Abrahamsson. 2024. “Can Large Language Models Serve as Data Analysts? A Multi-Agent Assisted Approach for Qualitative Data Analysis.”CoRR abs/2402.01386. https://doi.org/10.48550/ARXIV.2402.01386.
Reiss, Michael V. 2023. “Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark.”CoRR abs/2304.11085. https://doi.org/10.48550/ARXIV.2304.11085.
Ren, Shuo, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. “CodeBLEU: A Method for Automatic Evaluation of Code Synthesis.”CoRR abs/2009.10297. https://arxiv.org/abs/2009.10297.
Renze, Matthew. 2024. “The Effect of Sampling Temperature on Problem Solving in Large Language Models.” In Findings of the Association for Computational Linguistics: EMNLP 2024, 7346–56. https://doi.org/10.18653/v1/2024.findings-emnlp.432.
Richards, Jonan, and Mairieli Wessel. 2024. “What You Need Is What You Get: Theory of Mind for an LLM-Based Code Understanding Assistant.” In IEEE International Conference on Software Maintenance and Evolution, ICSME 2024, 666–71. IEEE. https://doi.org/10.1109/ICSME58944.2024.00070.
Ronanki, Krishna, Christian Berger, and Jennifer Horkoff. 2023. “Investigating ChatGPT’s Potential to Assist in Requirements Elicitation Processes.” In 2023 49th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), 354–61. IEEE.
Rozière, Baptiste, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, et al. 2023. “Code Llama: Open Foundation Models for Code.”CoRR abs/2308.12950. https://doi.org/10.48550/ARXIV.2308.12950.
Russo, Daniel. 2024. “Navigating the Complexity of Generative AI Adoption in Software Engineering.”ACM Transactions on Software Engineering and Methodology 33 (5): 1–50.
Sallou, June, Thomas Durieux, and Annibale Panichella. 2024. “Breaking the Silence: The Threats of Using LLMs in Software Engineering.” In Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results, NIER@ICSE 2024, Lisbon, Portugal, April 14-20, 2024, 102–6. ACM. https://doi.org/10.1145/3639476.3639764.
Santos, Adrian, Sira Vegas, Markku Oivo, and Natalia Juristo. 2021. “A Procedure and Guidelines for Analyzing Groups of Software Engineering Replications.”IEEE Trans. Software Eng. 47 (9): 1742–63. https://doi.org/10.1109/TSE.2019.2935720.
Schäfer, Max, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024. “An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation.”IEEE Trans. Software Eng. 50 (1): 85–105. https://doi.org/10.1109/TSE.2023.3334955.
Schilke, Oliver, and Martin Reimann. 2025. “The Transparency Dilemma: How AI Disclosure Erodes Trust.”Organizational Behavior and Human Decision Processes 188: 104405. https://doi.org/10.1016/j.obhdp.2025.104405.
Schneider, Kurt, Farnaz Fotrousi, and Rebekka Wohlrab. 2025. “A Reference Model for Empirically Comparing LLMs with Humans.” In 47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Society, ICSE-SEIS 2025, Ottawa, ON, Canada, April 27 - May 3, 2025, 130–34. IEEE. https://doi.org/10.1109/ICSE-SEIS66351.2025.00018.
Schröder, Sarah, Thekla Morgenroth, Ulrike Kuhl, Valerie Vaquet, and Benjamin Paaßen. 2025. “Large Language Models Do Not Simulate Human Psychology.”CoRR abs/2508.06950. https://doi.org/10.48550/ARXIV.2508.06950.
Schroeder, Kayla, and Zach Wood-Doughty. 2024. “Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge.”CoRR abs/2412.12509. https://doi.org/10.48550/ARXIV.2412.12509.
Schulhoff, Sander, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, et al. 2024. “The Prompt Report: A Systematic Survey of Prompting Techniques.”CoRR abs/2406.06608. https://doi.org/10.48550/ARXIV.2406.06608.
Schulz, Kenneth F., Douglas G. Altman, and David Moher. 2010. “CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomised Trials.”BMJ 340: c332. https://doi.org/10.1136/bmj.c332.
Schwartz, Roy, Jesse Dodge, Noah A. Smith, and Oren Etzioni. 2020. “Green AI.”Communications of the ACM 63 (12): 54–63. https://doi.org/10.1145/3381831.
Sclar, Melanie, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. “Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I Learned to Start Worrying about Prompt Formatting.” In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum?id=RIu5lyNXjT.
Shull, Forrest, Janice Singer, and Dag I. K. Sjøberg, eds. 2008. Guide to Advanced Empirical Software Engineering. Springer. https://doi.org/10.1007/978-1-84800-044-5.
Silva, André, and Martin Monperrus. 2025. “RepairBench: Leaderboard of Frontier Models for Program Repair.” In IEEE/ACM International Workshop on Large Language Models for Code, LLM4Code@ICSE 2025, Ottawa, ON, Canada, May 3, 2025, 9–16. IEEE. https://doi.org/10.1109/LLM4CODE66737.2025.00006.
Sjøberg, Dag I. K., and Gunnar Rye Bergersen. 2023. “Construct Validity in Software Engineering.”IEEE Trans. Software Eng. 49 (3): 1374–96. https://doi.org/10.1109/TSE.2022.3176725.
Song, Yifan, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. 2025. “The Good, the Bad, and the Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism.” In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2025 - Volume 1: Long Papers, edited by Luis Chiruzzo, Alan Ritter, and Lu Wang, 4195–4206. Association for Computational Linguistics. https://doi.org/10.18653/V1/2025.NAACL-LONG.211.
Sonnekalb, Tim, Bernd Gruner, Clemens-Alexander Brust, and Patrick Mäder. 2022. “Generalizability of Code Clone Detection on CodeBERT.” In 37th IEEE/ACM International Conference on Automated Software Engineering, ASE 2022, 143:1–3. ACM. https://doi.org/10.1145/3551349.3561165.
Strubell, Emma, Ananya Ganesh, and Andrew McCallum. 2019. “Energy and Policy Considerations for Deep Learning in NLP.” In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 3645–50. https://doi.org/10.18653/v1/P19-1355.
Sullivan, Gail M, and Joan Sargeant. 2011. “Qualities of Qualitative Research: Part I.”J Grad Med Educ 3 (4): 449–52.
Sumers, Theodore R., Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. 2024. “Cognitive Architectures for Language Agents.”Trans. Mach. Learn. Res. 2024. https://openreview.net/forum?id=1i6ZCvflQJ.
Takerngsaksiri, Wannita, Jirat Pasuksmit, Patanamon Thongtanunam, Chakkrit Tantithamthavorn, Ruixiong Zhang, Fan Jiang, Jing Li, Evan Cook, Kun Chen, and Ming Wu. 2025. “Human-in-the-Loop Software Development Agents.” In 47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice, SEIP@ICSE 2025, Ottawa, ON, Canada, April 27 - May 3, 2025, 342–52. IEEE. https://doi.org/10.1109/ICSE-SEIP66354.2025.00036.
Tam, Thomas Yu Chow, Sonish Sivarajkumar, Sumit Kapoor, Alisa V Stolyar, Katelyn Polanska, Karleigh R McCarthy, Hunter Osterhoudt, et al. 2024. “A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Review.”NPJ Digital Medicine 7 (1): 258.
Wagner, Stefan, Marvin Muñoz Barón, Davide Falessi, and Sebastian Baltes. 2025. “Towards Evaluation Guidelines for Empirical Studies Involving LLMs.” In IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering, WSESE@ICSE 2025, May 3, 2025, 24–27. IEEE. https://doi.org/10.1109/WSESE66602.2025.00011.
Wan, Mengting, Tara Safavi, Sujay Kumar Jauhar, Yujin Kim, Scott Counts, Jennifer Neville, Siddharth Suri, et al. 2024. “TnT-LLM: Text Mining at Scale with Large Language Models.” In Proceedings of the 30th ACMSIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2024, edited by Ricardo Baeza-Yates and Francesco Bonchi, 5836–47. ACM. https://doi.org/10.1145/3637528.3671647.
Wang, Angelina, Jamie Morgenstern, and John P. Dickerson. 2025. “Large Language Models That Replace Human Participants Can Harmfully Misportray and Flatten Identity Groups.”Nature Machine Intelligence 7: 400–411. https://doi.org/10.1038/s42256-025-00986-z.
Wang, Chong, Kaifeng Huang, Jian Zhang, Yebo Feng, Lyuye Zhang, Yang Liu, and Xin Peng. 2024. “How and Why LLMs Use Deprecated APIs in Code Completion? An Empirical Study.”CoRR abs/2406.09834. https://doi.org/10.48550/ARXIV.2406.09834.
Wang, Fanyu, Chetan Arora, Yonghui Liu, Kaicheng Huang, Chakkrit Tantithamthavorn, Aldeida Aleti, Dishan Sambathkumar, and David Lo. 2025. “Multi-Modal Requirements Data-Based Acceptance Criteria Generation Using LLMs.” In 40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025, 3334–45. IEEE. https://doi.org/10.1109/ASE63991.2025.00275.
Wang, Shuohang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. “Want to Reduce Labeling Cost? GPT-3 Can Help.” In Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, edited by Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, EMNLP 2021:4195–205. Findings of ACL. Association for Computational Linguistics. https://doi.org/10.18653/V1/2021.FINDINGS-EMNLP.354.
Wang, Xinru, Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjie Miao. 2024. “Human-LLM Collaborative Annotation Through Effective Verification of LLM Labels.” In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI 2024, edited by Florian ’Floyd’Mueller, Penny Kyburz, Julie R. Williamson, Corina Sas, Max L. Wilson, Phoebe O. Toups Dugas, and Irina Shklovski, 303:1–21. ACM. https://doi.org/10.1145/3613904.3641960.
Widder, David Gray, Meredith Whittaker, and Sarah Myers West. 2024. “Why ‘Open’AI Systems Are Actually Closed, and Why This Matters.”Nature 635 (8040): 827–33.
Wiesinger, Julia, Patrick Marlow, and Vladimir Vuskovic. 2025. “Agents.”Google DeepMind. https://gemini.google.com.
Wohlin, Claes, Per Runeson, Martin Höst, Magnus C. Ohlsson, Björn Regnell, and Anders Wesslén. 2024. Experimentation in Software Engineering, Second Edition. Springer. https://doi.org/10.1007/978-3-662-69306-3.
… Unchanged text
Xia, Yuying, Haijian Shao, and Xing Deng. 2024. “VulCoBERT: A CodeBERT-Based System for Source Code Vulnerability Detection.” In 2024 International Conference on Generative Artificial Intelligence and Information Security, GAIIS 2024, Kuala Lumpur, Malaysia, May 10-12, 2024. ACM. https://doi.org/10.1145/3665348.3665391.
Xiao, Tao, Christoph Treude, Hideaki Hata, and Kenichi Matsumoto. 2024. “DevGPT: Studying Developer-ChatGPT Conversations.” In 21st IEEE/ACM International Conference on Mining Software Repositories, MSR 2024, Lisbon, Portugal, April 15-16, 2024, edited by Diomidis Spinellis, Alberto Bacchelli, and Eleni Constantinou, 227–30. ACM. https://doi.org/10.1145/3643991.3648400.
Xu, Cheng, Shuhao Guan, Derek Greene, and M. Tahar Kechadi. 2024. “Benchmark Data Contamination of Large Language Models: A Survey.”CoRR abs/2406.04244. https://doi.org/10.48550/ARXIV.2406.04244.
Xu, Ruoxi, Yingfei Sun, Mengjie Ren, Shiguang Guo, Ruotong Pan, Hongyu Lin, Le Sun, and Xianpei Han. 2024. “AI for Social Science and Social Science of AI:A Survey.”Inf. Process. Manag. 61 (2): 103665. https://doi.org/10.1016/J.IPM.2024.103665.
Yan, Litao, Alyssa Hwang, Zhiyuan Wu, and Andrew Head. 2024. “Ivie: Lightweight Anchored Explanations of Just-Generated Code.” In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI 2024, edited by Florian ’Floyd’Mueller, Penny Kyburz, Julie R. Williamson, Corina Sas, Max L. Wilson, Phoebe O. Toups Dugas, and Irina Shklovski, 140:1–15. ACM. https://doi.org/10.1145/3613904.3642239.
Yang, Guang, Yu Zhou, Xiang Chen, Xiangyu Zhang, Tingting Han, and Taolue Chen. 2023. “ExploitGen: Template-Augmented Exploit Code Generation Based on CodeBERT.”J. Syst. Softw. 197: 111577. https://doi.org/10.1016/J.JSS.2022.111577.
Yang, Yifan, Jiho Shin, Byeonggyu Choi, Minjun Park, Dayun Ju, Changmin Lee, Sanghyuk Chun, Dongjin Kang, Jiin Kim, and Sungroh Yoon. 2025. “From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence.”arXiv Preprint arXiv:2502.11827. https://arxiv.org/abs/2502.11827.
Yuan, Jiayi, Hao Li, Xinheng Ding, Wenya Xie, Yu-Jhe Li, Wentian Zhao, Kun Wan, Jing Shi, Xia Hu, and Zirui Liu. 2025. “Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference.” In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025. https://openreview.net/forum?id=Q3qAsZAEZw.
Zhao, Chenguang, Meirewuti Habule, and Wei Zhang. 2025. “Large Language Models (LLMs) as Research Subjects: Status, Opportunities and Challenges.”New Ideas in Psychology 79: 101167. https://doi.org/10.1016/j.newideapsych.2025.101167.
Zhou, Ruiyang, Lu Chen, and Kai Yu. 2024. “Is LLM a Reliable Reviewer? A Comprehensive Evaluation of LLM on Automatic Paper Reviewing Tasks.” In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, edited by Nicoletta Calzolari, Min-Yen Kan, Véronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, 9340–51. ELRA; ICCL. https://aclanthology.org/2024.lrec-main.816.
Zhou, Wangchunshu, Yuchen Eleanor Jiang, Long Li, Jialong Wu, Tiannan Wang, Shi Qiu, Jintian Zhang, et al. 2023. “Agents: An Open-Source Framework for Autonomous Language Agents.”CoRR abs/2309.07870. https://doi.org/10.48550/ARXIV.2309.07870.
Zhu, Yiming, Peixian Zhang, Ehsan ul Haq, Pan Hui, and Gareth Tyson. 2023. “Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks.”CoRR abs/2304.10145. https://doi.org/10.48550/ARXIV.2304.10145.
Unchanged sections (11)Study Types Unchanged
2026.03
Study Types
Empirical guidelines are needed for SE studies involving LLMs to ensure the validity and reproducibility of their results. However, such guidelines must be tailored to different study types that each pose unique challenges. Therefore, we created a taxonomy of study types to contextualize our recommendations. We use the term study type to refer to categories of LLM involvement in empirical research, rather than to research methodologies (e.g., experiments, case studies). A single empirical study may involve multiple such study types. Each study type section starts with a description, followed by examples from the SE research community and beyond. The advantages and challenges of using LLMs are discussed in a summary subsection at the end of this section, synthesizing cross-cutting themes across all study types.
Empirical guidelines are needed for SE studies involving LLMs to ensure the validity and reproducibility of their results. However, such guidelines must be tailored to different study types that each pose unique challenges. Therefore, we created a taxonomy of study types to contextualize our recommendations. We use the term study type to refer to categories of LLM involvement in empirical research, rather than to research methodologies (e.g., experiments, case studies). A single empirical study may involve multiple such study types. Each study type section starts with a description, followed by examples from the SE research community and beyond. The advantages and challenges of using LLMs are discussed in a summary subsection at the end of this section, synthesizing cross-cutting themes across all study types.
LLMs as Tools for Software Engineering Researchers Unchanged
2026.03
LLMs as Tools for Software Engineering Researchers
LLMs can serve as tools to help researchers conduct empirical studies. They can automate various tasks such as data collection, pre-processing, and analysis. For example, LLMs can apply predefined coding guides to qualitative datasets ( LLMs as AnnotatorsS1: LLMs as Annotators ), assess the quality of software artifacts ( LLMs as JudgesS2: LLMs as Judges ), generate summaries of research papers ( LLMs for SynthesisS3: LLMs for Synthesis ), or simulate human behavior in empirical studies ( LLMs as SubjectsS4: LLMs as Subjects ). If LLMs could complete these tasks well enough, they would reduce the time and effort required to conduct a study. However, these applications each pose challenges to validity and reproducibility.
2026.04
LLMs as Tools for Software Engineering Researchers
LLMs can serve as tools to help researchers conduct empirical studies. They can automate various tasks such as data collection, pre-processing, and analysis. For example, LLMs can apply predefined coding guides to qualitative datasets ( LLMs as AnnotatorsS1: LLMs as Annotators ), assess the quality of software artifacts ( LLMs as JudgesS2: LLMs as Judges ), generate summaries of research papers ( LLMs for SynthesisS3: LLMs for Synthesis ), or simulate human behavior in empirical studies ( LLMs as SubjectsS4: LLMs as Subjects ). If LLMs could complete these tasks well enough, they would reduce the time and effort required to conduct a study. However, these applications each pose challenges to validity and reproducibility.
LLMs as Annotators (S1) Unchanged
2026.03
LLMs as Annotators (S1)
Description
In qualitative data analysis, manually annotating (“coding”) natural language text (e.g., requirements, interview transcripts, open-ended survey responses) is a time-consuming manual process (Bano et al. 2024). LLMs can augment human coding, suggest new codes, and label artifacts based on a pre-defined coding guide much faster than humans can (Z. He et al. 2024). The extent to which, or under what conditions, LLMs can perform these tasks effectively remains an open research question (T. Ahmed et al. 2025). Indeed, measuring their effectiveness is practically and philosophically challenging. From an interpretivist philosophical perspective, one cannot measure the quality of analysis by comparing one (human or machine) analyst’s work to another. From a realist perspective, triangulating across multiple human judges, LLM judges, and other data sources (e.g., whether a pull request is marked as having resolved an issue) improves confidence in the findings but does not prove that any one judge is valid.
Examples
J. Huang et al. (2024)(J. Huang et al. 2024) utilized multiple LLMs for joint annotation of mobile application reviews. They used three models of comparable size with an absolute majority voting rule (i.e., a label is only accepted if it receives more than half of the total votes from the models). This approach slightly outperformed the best individual model tested. Meanwhile, T. Ahmed et al. (2025)(T. Ahmed et al. 2025) examined LLMs as annotators in SE research across five datasets, six LLMs, and ten annotation tasks. They found that model-model agreement strongly correlates with human-model agreement; models performed poorly in tasks where humans also frequently disagreed. They proposed to use model confidence scores to identify specific samples that could be safely delegated to LLMs, potentially reducing human annotation effort without compromising inter-rater agreement.
2026.04
LLMs as Annotators (S1)
Description
In qualitative data analysis, manually annotating (“coding”) natural language text (e.g., requirements, interview transcripts, open-ended survey responses) is a time-consuming manual process (Bano et al. 2024). LLMs can augment human coding, suggest new codes, and label artifacts based on a pre-defined coding guide much faster than humans can (Z. He et al. 2024). The extent to which, or under what conditions, LLMs can perform these tasks effectively remains an open research question (T. Ahmed et al. 2025). Indeed, measuring their effectiveness is practically and philosophically challenging. From an interpretivist philosophical perspective, one cannot measure the quality of analysis by comparing one (human or machine) analyst’s work to another. From a realist perspective, triangulating across multiple human judges, LLM judges, and other data sources (e.g., whether a pull request is marked as having resolved an issue) improves confidence in the findings but does not prove that any one judge is valid.
Examples
J. Huang et al. (2024)(J. Huang et al. 2024) utilized multiple LLMs for joint annotation of mobile application reviews. They used three models of comparable size with an absolute majority voting rule (i.e., a label is only accepted if it receives more than half of the total votes from the models). This approach slightly outperformed the best individual model tested. Meanwhile, T. Ahmed et al. (2025)(T. Ahmed et al. 2025) examined LLMs as annotators in SE research across five datasets, six LLMs, and ten annotation tasks. They found that model-model agreement strongly correlates with human-model agreement; models performed poorly in tasks where humans also frequently disagreed. They proposed to use model confidence scores to identify specific samples that could be safely delegated to LLMs, potentially reducing human annotation effort without compromising inter-rater agreement.
LLMs as Judges (S2) Unchanged
2026.03
LLMs as Judges (S2)
Description
LLMs can rate properties of software artifacts (e.g. assess code readability, adherence to coding standards, or comment quality) or sort multiple solutions by some attribute. These kinds of judgments are distinct from annotating unstructured text (see LLMs as AnnotatorsS1: LLMs as Annotators ).
Example(s)
Lubos et al. (2024)(Lubos et al. 2024) leveraged Llama-2 to evaluate the quality of software requirements statements. They prompted the LLM with the text below, where the words in braces reflect the study parameters:
Your task is to evaluate the quality of a software requirement. Evaluate
whether the following requirement is quality_characteristic.
quality_characteristic means: quality_characteristic_explanation The
evaluation result must be: ’yes’ or ’no’. Request: Based on the
following description of the project: project_description Evaluate the
quality of the following requirement: requirement. Explain your decision
and suggest an improved version.
They evaluated LLM output against expert human judges and found moderate agreement for simple requirements and poor agreement for more complex requirements. In contrast, F. Wang et al. (2025)(F. Wang et al. 2025) used an LLM to generate acceptance criteria for user stories, and provided a rubric to an LLM to judge the generated acceptance criteria on interpretable scales (0 to 4).
2026.04
LLMs as Judges (S2)
Description
LLMs can rate properties of software artifacts (e.g. assess code readability, adherence to coding standards, or comment quality) or sort multiple solutions by some attribute. These kinds of judgments are distinct from annotating unstructured text (see LLMs as AnnotatorsS1: LLMs as Annotators ).
Example(s)
Lubos et al. (2024)(Lubos et al. 2024) leveraged Llama-2 to evaluate the quality of software requirements statements. They prompted the LLM with the text below, where the words in braces reflect the study parameters:
Your task is to evaluate the quality of a software requirement. Evaluate
whether the following requirement is quality_characteristic.
quality_characteristic means: quality_characteristic_explanation The
evaluation result must be: ’yes’ or ’no’. Request: Based on the
following description of the project: project_description Evaluate the
quality of the following requirement: requirement. Explain your decision
and suggest an improved version.
They evaluated LLM output against expert human judges and found moderate agreement for simple requirements and poor agreement for more complex requirements. In contrast, F. Wang et al. (2025)(F. Wang et al. 2025) used an LLM to generate acceptance criteria for user stories, and provided a rubric to an LLM to judge the generated acceptance criteria on interpretable scales (0 to 4).
LLMs for Synthesis (S3) Unchanged
2026.03
LLMs for Synthesis (S3)
Description
Unlike annotation (see LLMs as AnnotatorsS1: LLMs as Annotators ), which focuses on categorizing or labeling individual data points, synthesis refers to the process of integrating and interpreting information from multiple sources to generate higher-level insights, identify patterns across datasets, and develop conceptual frameworks or theories. LLMs may be able to support synthesis tasks in SE research by processing and distilling information from qualitative data sources. Although synthesis in the preceding notion refers to abstraction and interpretation across multiple data sources, the term is sometimes also used to refer to generating synthetic content (e.g., source code, bug-fix pairs, requirements, etc.) that are then used in downstream tasks to train, fine-tune, or evaluate existing models or tools. In this case, the synthesis is done primarily using the LLM and its training data; the input is limited to basic instructions and examples.
Example(s)
Published examples of applying LLMs for synthesis in SE remain scarce; however, some recent work in other domains is instructive (Bano et al. 2024). Barros et al. (2025)(Barros et al. 2025) conducted a systematic mapping study on using LLMs for qualitative research and found that most studies focused on the healthcare and social sciences. In these studies, LLMs supported qualitative methods, such as grounded theory and thematic analysis, by aiding in pattern identification. In SE, de Morais Leca et al. (Morais Leça et al. 2025) explored how LLMs have been applied for qualitative data analysis (QDA) and proposed general strategies and guidelines for their application. Ornelas et al. (Ornelas et al. 2025) complemented this perspective by studying the opportunities and limitations of introducing LLM-based support into QDA, and by formulating recommendations for embedding human–AI collaboration across the thematic analysis phases. Building on these, subsequent work has proposed hybrid frameworks combining LLM support with human-led QDA. Rashid et al. (Rasheed et al. 2024) designed an LLM-driven multi-agent system that integrates AI with human decision-making to automate qualitative data analysis methods. Their system generated initial codes, developed themes, and summarized text. Similarly, Montes et al. (Montes et al. 2025) compared the performance of humans and LLMs in coding, theme development, definition, and refinement, creating guidelines for a hybrid-LLM framework. Finally, El-Hajjami and Salinesi (2025)’s work is an example of using LLMs to create synthetic datasets. They present an approach to generate synthetic requirements, showing that they “can match or surpass human-authored requirements for specific classification tasks” (El-Hajjami and Salinesi 2025).
2026.04
LLMs for Synthesis (S3)
Description
Unlike annotation (see LLMs as AnnotatorsS1: LLMs as Annotators ), which focuses on categorizing or labeling individual data points, synthesis refers to the process of integrating and interpreting information from multiple sources to generate higher-level insights, identify patterns across datasets, and develop conceptual frameworks or theories. LLMs may be able to support synthesis tasks in SE research by processing and distilling information from qualitative data sources. Although synthesis in the preceding notion refers to abstraction and interpretation across multiple data sources, the term is sometimes also used to refer to generating synthetic content (e.g., source code, bug-fix pairs, requirements, etc.) that are then used in downstream tasks to train, fine-tune, or evaluate existing models or tools. In this case, the synthesis is done primarily using the LLM and its training data; the input is limited to basic instructions and examples.
Example(s)
Published examples of applying LLMs for synthesis in SE remain scarce; however, some recent work in other domains is instructive (Bano et al. 2024). Barros et al. (2025)(Barros et al. 2025) conducted a systematic mapping study on using LLMs for qualitative research and found that most studies focused on the healthcare and social sciences. In these studies, LLMs supported qualitative methods, such as grounded theory and thematic analysis, by aiding in pattern identification. In SE, de Morais Leca et al. (Morais Leça et al. 2025) explored how LLMs have been applied for qualitative data analysis (QDA) and proposed general strategies and guidelines for their application. Ornelas et al. (Ornelas et al. 2025) complemented this perspective by studying the opportunities and limitations of introducing LLM-based support into QDA, and by formulating recommendations for embedding human–AI collaboration across the thematic analysis phases. Building on these, subsequent work has proposed hybrid frameworks combining LLM support with human-led QDA. Rashid et al. (Rasheed et al. 2024) designed an LLM-driven multi-agent system that integrates AI with human decision-making to automate qualitative data analysis methods. Their system generated initial codes, developed themes, and summarized text. Similarly, Montes et al. (Montes et al. 2025) compared the performance of humans and LLMs in coding, theme development, definition, and refinement, creating guidelines for a hybrid-LLM framework. Finally, El-Hajjami and Salinesi (2025)’s work is an example of using LLMs to create synthetic datasets. They present an approach to generate synthetic requirements, showing that they “can match or surpass human-authored requirements for specific classification tasks” (El-Hajjami and Salinesi 2025).
LLMs as Subjects (S4) Unchanged
2026.03
LLMs as Subjects (S4)
Description
In empirical studies, data is collected from participants through methods such as surveys, interviews, or controlled experiments. LLMs can serve as virtual subjects by simulating human behavior and interactions. If LLMs can generate responses that approximate those of human participants, they could be valuable for research involving user interactions, collaborative coding environments, and software usability assessments (Zhao, Habule, and Zhang 2025). To achieve this, prompt engineering techniques are widely employed; for instance, the Personas Pattern(Kong et al. 2023) involves tailoring LLM responses to align with predefined profiles or roles that emulate specific user archetypes. To serve as virtual subjects, generated responses should be indistinguishable from human-produced texts, consistent with the attitudes and sociodemographic information of the conditioning context (e.g., junior vs. senior developers), naturally aligned with the form, tone, and content of the simulated scenario, and reflect patterns in relationships between ideas, demographics, and behavior observed in comparable human data (Argyle et al. 2022).
Example(s)
R. Xu et al. (2024)(R. Xu et al. 2024) compiled a list of ways LLMs can support social science research, some of which transfer to empirical SE research. For example, LLMs can emulate human responses and behaviors in simulated interviews and focus groups Gerosa et al. (2024)(Gerosa et al. 2024). Similarly, Bano, Gunatilake, and Hoda (2025)(Bano, Gunatilake, and Hoda 2025), investigated biases in LLM-generated candidate profiles in SE recruitment processes. They found biases favoring male candidates, lighter skin tones, and slim physiques, particularly for senior roles. LLMs may be able to simulate end-user feedback and behavior in usability studies, identify usability issues and offering suggestions for improvement based on predefined user personas.
2026.04
LLMs as Subjects (S4)
Description
In empirical studies, data is collected from participants through methods such as surveys, interviews, or controlled experiments. LLMs can serve as virtual subjects by simulating human behavior and interactions. If LLMs can generate responses that approximate those of human participants, they could be valuable for research involving user interactions, collaborative coding environments, and software usability assessments (Zhao, Habule, and Zhang 2025). To achieve this, prompt engineering techniques are widely employed; for instance, the Personas Pattern(Kong et al. 2023) involves tailoring LLM responses to align with predefined profiles or roles that emulate specific user archetypes. To serve as virtual subjects, generated responses should be indistinguishable from human-produced texts, consistent with the attitudes and sociodemographic information of the conditioning context (e.g., junior vs. senior developers), naturally aligned with the form, tone, and content of the simulated scenario, and reflect patterns in relationships between ideas, demographics, and behavior observed in comparable human data (Argyle et al. 2022).
Example(s)
R. Xu et al. (2024)(R. Xu et al. 2024) compiled a list of ways LLMs can support social science research, some of which transfer to empirical SE research. For example, LLMs can emulate human responses and behaviors in simulated interviews and focus groups Gerosa et al. (2024)(Gerosa et al. 2024). Similarly, Bano, Gunatilake, and Hoda (2025)(Bano, Gunatilake, and Hoda 2025), investigated biases in LLM-generated candidate profiles in SE recruitment processes. They found biases favoring male candidates, lighter skin tones, and slim physiques, particularly for senior roles. LLMs may be able to simulate end-user feedback and behavior in usability studies, identify usability issues and offering suggestions for improvement based on predefined user personas.
LLMs as Tools for Software Engineers Unchanged
2026.03
LLMs as Tools for Software Engineers
LLM-based assistants have become a widely adopted tool for software engineers (Stack Overflow 2025), supporting them in various tasks such as code generation and debugging. Researchers have studied how software engineers use LLMs ( Studying LLM UsageS5: Studying LLM Usage ), developed new tools that integrate LLMs ( LLMs for ToolsS6: LLMs for Tools ), and benchmarked LLMs for software engineering tasks ( Benchmarking LLMsS7: Benchmarking LLMs ).
2026.04
LLMs as Tools for Software Engineers
LLM-based assistants have become a widely adopted tool for software engineers (Stack Overflow 2025), supporting them in various tasks such as code generation and debugging. Researchers have studied how software engineers use LLMs ( Studying LLM UsageS5: Studying LLM Usage ), developed new tools that integrate LLMs ( LLMs for ToolsS6: LLMs for Tools ), and benchmarked LLMs for software engineering tasks ( Benchmarking LLMsS7: Benchmarking LLMs ).
Studying LLM Usage in Software Engineering (S5) Unchanged
2026.03
Studying LLM Usage in Software Engineering (S5)
Description
Studying how software engineers use LLMs and LLM-based tools is important for understanding the current state of practice in SE. Researchers can observe software engineers’ usage of LLM-based tools in the field, or study if and how they adopt such tools, their usage patterns, as well as perceived benefits and challenges. Surveys, interviews, observational studies, or analysis of usage logs can provide insights into how LLMs are integrated into development processes, how they influence decision making, and what factors affect their acceptance and effectiveness. Such studies can inform improvements for existing LLM-based tools, motivate the design of novel tools, or derive best practices for LLM-assisted software engineering. They can also uncover risks or deficiencies of existing tools.
Example(s)
Based on a convergent mixed-methods study, Russo (2024) has found that early adoption of generative AI by software engineers is primarily driven by compatibility with existing workflows (Russo 2024). Khojah et al. (2024) investigated the use of ChatGPT (GPT-3.5) by professional software engineers in a week-long observational study (Khojah et al. 2024). They found that most developers do not use the code generated by ChatGPT directly but instead use the output as a guide to implement their own solutions. Azanza et al. (2024)’s case study found that LLMs could “enhance personalized, instant onboarding support; however, relying on proprietary external LLMs poses significant data privacy risks” (Azanza et al. 2024). Jahic and Sami (2024) found that most participants from 15 software companies they surveyed had already adopted AI (especially ChatGPT) for SE tasks; but cited copyright and privacy issues, as well as inconsistent or low-quality outputs, as barriers to adoption (Jahic and Sami 2024). Retrospective studies that analyze data generated while developers use LLMs can provide additional insights into human-LLM interactions. For example, researchers can employ data mining methods to build large-scale conversation datasets, such as the DevGPT dataset introduced by Xiao et al. (2024)(Xiao et al. 2024). Conversations can then be analyzed using quantitative (Rabbi et al. 2024) and qualitative (Mohamed, Parvin, and Parra 2024) analysis methods.
2026.04
Studying LLM Usage in Software Engineering (S5)
Description
Studying how software engineers use LLMs and LLM-based tools is important for understanding the current state of practice in SE. Researchers can observe software engineers’ usage of LLM-based tools in the field, or study if and how they adopt such tools, their usage patterns, as well as perceived benefits and challenges. Surveys, interviews, observational studies, or analysis of usage logs can provide insights into how LLMs are integrated into development processes, how they influence decision making, and what factors affect their acceptance and effectiveness. Such studies can inform improvements for existing LLM-based tools, motivate the design of novel tools, or derive best practices for LLM-assisted software engineering. They can also uncover risks or deficiencies of existing tools.
Example(s)
Based on a convergent mixed-methods study, Russo (2024) has found that early adoption of generative AI by software engineers is primarily driven by compatibility with existing workflows (Russo 2024). Khojah et al. (2024) investigated the use of ChatGPT (GPT-3.5) by professional software engineers in a week-long observational study (Khojah et al. 2024). They found that most developers do not use the code generated by ChatGPT directly but instead use the output as a guide to implement their own solutions. Azanza et al. (2024)’s case study found that LLMs could “enhance personalized, instant onboarding support; however, relying on proprietary external LLMs poses significant data privacy risks” (Azanza et al. 2024). Jahic and Sami (2024) found that most participants from 15 software companies they surveyed had already adopted AI (especially ChatGPT) for SE tasks; but cited copyright and privacy issues, as well as inconsistent or low-quality outputs, as barriers to adoption (Jahic and Sami 2024). Retrospective studies that analyze data generated while developers use LLMs can provide additional insights into human-LLM interactions. For example, researchers can employ data mining methods to build large-scale conversation datasets, such as the DevGPT dataset introduced by Xiao et al. (2024)(Xiao et al. 2024). Conversations can then be analyzed using quantitative (Rabbi et al. 2024) and qualitative (Mohamed, Parvin, and Parra 2024) analysis methods.
LLMs for New Software Engineering Tools (S6) Unchanged
2026.03
LLMs for New Software Engineering Tools (S6)
Description
LLMs are being integrated into new tools that support software engineers in their daily tasks (e.g., to assist in code comprehension (Yan et al. 2024) and test case generation (Schäfer et al. 2024)). One way of integrating LLM-based tools into software engineers’ workflows is using GenAI agents. Unlike traditional LLM-based tools, these agents are capable of acting autonomously and proactively, are often tailored to meet specific user needs, and can interact with external environments (Takerngsaksiri et al. 2025; Wiesinger, Marlow, and Vuskovic 2025; Y. Yang et al. 2025). From an architectural perspective, GenAI agents can be implemented in various ways (Wiesinger, Marlow, and Vuskovic 2025). However, they generally share three key components: (1) a reasoning mechanism that guides the LLM (often enabled by advanced prompt engineering), (2) a set of tools to interact with external systems (e.g., APIs or databases), and (3) a user communication interface that extends beyond traditional chat-based interactions (Richards and Wessel 2024; Sumers et al. 2024; W. Zhou et al. 2023). Researchers can also test and compare different tool architectures to increase artifact quality and developer satisfaction.
Example(s)
Yan et al. (2024) proposed IVIE, a tool integrated into the VS Code graphical interface that generates and explains code using LLMs (Yan et al. 2024). The authors focused more on the presentation, providing a user-friendly interface to interact with the LLM. Schäfer et al. (2024)(Schäfer et al. 2024) presented a large-scale empirical evaluation on the effectiveness of LLMs for automated unit test generation. They presented TestPilot, a tool that implements an approach in which the LLM is provided with prompts that include the signature and implementation of a function under test, along with usage examples extracted from the documentation. Richards and Wessel (2024) introduced a preliminary GenAI agent designed to assist developers in understanding source code by incorporating a reasoning component grounded in the theory of mind (Richards and Wessel 2024).
2026.04
LLMs for New Software Engineering Tools (S6)
Description
LLMs are being integrated into new tools that support software engineers in their daily tasks (e.g., to assist in code comprehension (Yan et al. 2024) and test case generation (Schäfer et al. 2024)). One way of integrating LLM-based tools into software engineers’ workflows is using GenAI agents. Unlike traditional LLM-based tools, these agents are capable of acting autonomously and proactively, are often tailored to meet specific user needs, and can interact with external environments (Takerngsaksiri et al. 2025; Wiesinger, Marlow, and Vuskovic 2025; Y. Yang et al. 2025). From an architectural perspective, GenAI agents can be implemented in various ways (Wiesinger, Marlow, and Vuskovic 2025). However, they generally share three key components: (1) a reasoning mechanism that guides the LLM (often enabled by advanced prompt engineering), (2) a set of tools to interact with external systems (e.g., APIs or databases), and (3) a user communication interface that extends beyond traditional chat-based interactions (Richards and Wessel 2024; Sumers et al. 2024; W. Zhou et al. 2023). Researchers can also test and compare different tool architectures to increase artifact quality and developer satisfaction.
Example(s)
Yan et al. (2024) proposed IVIE, a tool integrated into the VS Code graphical interface that generates and explains code using LLMs (Yan et al. 2024). The authors focused more on the presentation, providing a user-friendly interface to interact with the LLM. Schäfer et al. (2024)(Schäfer et al. 2024) presented a large-scale empirical evaluation on the effectiveness of LLMs for automated unit test generation. They presented TestPilot, a tool that implements an approach in which the LLM is provided with prompts that include the signature and implementation of a function under test, along with usage examples extracted from the documentation. Richards and Wessel (2024) introduced a preliminary GenAI agent designed to assist developers in understanding source code by incorporating a reasoning component grounded in the theory of mind (Richards and Wessel 2024).
Benchmarking LLMs for Software Engineering Tasks (S7) Unchanged
2026.03
Benchmarking LLMs for Software Engineering Tasks (S7)
Description
Benchmarking is the process of evaluating an LLM’s performance on standardized tasks, using standardized metrics, on a standardized dataset. The LLM output is compared to a supposed ground truth from the benchmark dataset. Typical tasks include code generation, code summarization, code completion, and code repair (Y. Yang et al. 2025), but also natural language processing tasks, such as anaphora resolution (i.e., the task of identifying the referring expression of a word or phrase occurring earlier in the text). Metrics may include general metrics for text generation, such as ROUGE, BLEU, or METEOR(Hou et al. 2024), or task-specific metrics, such as CodeBLEU for code generation. Benchmarking requires high-quality reference datasets.
Example(s)
In SE, benchmarking may include evaluating an LLM’s ability to produce accurate and reliable outputs for a given input (usually a task description, possibly accompanied by data obtained from curated real-world projects or from synthetic SE-specific datasets). RepairBench(Silva and Monperrus 2025), for example, contains 574 buggy Java methods and their corresponding fixed versions, which can be used to evaluate the performance of LLMs in code repair tasks. It uses the Plausible@1 metric (i.e., the probability that the first generated patch passes all test cases) and the AST Match@1 metric (i.e., the probability that the abstract syntax tree of the first generated patch matches the ground truth patch). SWE-Bench(Jimenez et al. 2024) is a more generic benchmark that contains 2,294 SE Python tasks extracted from GitHub pull requests. To score the LLM’s task performance, the benchmark validates whether the generated patch successfully compiles and calculates the percentage of passed test cases. Meanwhile, HumanEval(M. Chen et al. 2021) is often used to assess code generation.
2026.04
Benchmarking LLMs for Software Engineering Tasks (S7)
Description
Benchmarking is the process of evaluating an LLM’s performance on standardized tasks, using standardized metrics, on a standardized dataset. The LLM output is compared to a supposed ground truth from the benchmark dataset. Typical tasks include code generation, code summarization, code completion, and code repair (Y. Yang et al. 2025), but also natural language processing tasks, such as anaphora resolution (i.e., the task of identifying the referring expression of a word or phrase occurring earlier in the text). Metrics may include general metrics for text generation, such as ROUGE, BLEU, or METEOR(Hou et al. 2024), or task-specific metrics, such as CodeBLEU for code generation. Benchmarking requires high-quality reference datasets.
Example(s)
In SE, benchmarking may include evaluating an LLM’s ability to produce accurate and reliable outputs for a given input (usually a task description, possibly accompanied by data obtained from curated real-world projects or from synthetic SE-specific datasets). RepairBench(Silva and Monperrus 2025), for example, contains 574 buggy Java methods and their corresponding fixed versions, which can be used to evaluate the performance of LLMs in code repair tasks. It uses the Plausible@1 metric (i.e., the probability that the first generated patch passes all test cases) and the AST Match@1 metric (i.e., the probability that the abstract syntax tree of the first generated patch matches the ground truth patch). SWE-Bench(Jimenez et al. 2024) is a more generic benchmark that contains 2,294 SE Python tasks extracted from GitHub pull requests. To score the LLM’s task performance, the benchmark validates whether the generated patch successfully compiles and calculates the percentage of passed test cases. Meanwhile, HumanEval(M. Chen et al. 2021) is often used to assess code generation.
Conclusion Unchanged
2026.03
Conclusion
LLM non-determinism, opaque training data, and rapidly evolving models threaten the reproducibility and replicability of empirical SE studies. This paper presents a taxonomy of seven study types that organizes how LLMs are used in SE research, together with eight guidelines for designing and reporting such studies. Each guideline distinguishes requirements (MUST ) from recommended practices (SHOULD ) and is contextualized by the study types it applies to. Our guidelines recommend that researchers:
declare LLM usage and role;
report model versions, configurations, and customizations;
document the tool architecture beyond the model;
disclose prompts, their development, and, where feasible, interaction logs;
validate LLM outputs with humans;
include an open LLM as a baseline;
use suitable baselines, benchmarks, and metrics; and
articulate limitations and mitigations, including costs, potential biases, and environmental impact.
An applicability matrix maps guidelines to study types, and a reporting checklist (Section 7) supports authors and reviewers. The study types and guidelines were derived collaboratively by 22 co-authors, following a process similar to that of guidelines in other disciplines (see, e.g., (Begg et al. 1996)).
Following the quality dimensions of the SIGSOFT Empirical Standards(Ralph et al. 2021), we consider our guidelines (1) comprehensive, as they cover seven study types that we identified and were derived from 22 co-authors’ collective expertise; (2) useful, as they provide actionable recommendations with an applicability matrix and a concrete reporting checklist (Section 7); (3) well-argued, as each guideline is motivated by a dedicated rationale and supported by references to the literature; and (4) integrated with previous work, as they build on and complement established empirical SE standards while addressing LLM-specific challenges. Adopting these practices will not eliminate LLM non-determinism, but it should make claims auditable and results easier to replicate and compare across studies. Because models and tools evolve, we maintain the guidelines as a living resource and invite the community to refine and extend them at llm-guidelines.org.
2026.04
Conclusion
LLM non-determinism, opaque training data, and rapidly evolving models threaten the reproducibility and replicability of empirical SE studies. This paper presents a taxonomy of seven study types that organizes how LLMs are used in SE research, together with eight guidelines for designing and reporting such studies. Each guideline distinguishes requirements (MUST ) from recommended practices (SHOULD ) and is contextualized by the study types it applies to. Our guidelines recommend that researchers:
declare LLM usage and role;
report model versions, configurations, and customizations;
document the tool architecture beyond the model;
disclose prompts, their development, and, where feasible, interaction logs;
validate LLM outputs with humans;
include an open LLM as a baseline;
use suitable baselines, benchmarks, and metrics; and
articulate limitations and mitigations, including costs, potential biases, and environmental impact.
An applicability matrix maps guidelines to study types, and a reporting checklist (Section 7) supports authors and reviewers. The study types and guidelines were derived collaboratively by 22 co-authors, following a process similar to that of guidelines in other disciplines (see, e.g., (Begg et al. 1996)).
Following the quality dimensions of the SIGSOFT Empirical Standards(Ralph et al. 2021), we consider our guidelines (1) comprehensive, as they cover seven study types that we identified and were derived from 22 co-authors’ collective expertise; (2) useful, as they provide actionable recommendations with an applicability matrix and a concrete reporting checklist (Section 7); (3) well-argued, as each guideline is motivated by a dedicated rationale and supported by references to the literature; and (4) integrated with previous work, as they build on and complement established empirical SE standards while addressing LLM-specific challenges. Adopting these practices will not eliminate LLM non-determinism, but it should make claims auditable and results easier to replicate and compare across studies. Because models and tools evolve, we maintain the guidelines as a living resource and invite the community to refine and extend them at llm-guidelines.org.