Back to Changelog

Changes from 2026.04 to 2026.05

This page compares the paper body, appendices, and references. Deletions are struck through; additions are underlined. Sections follow the order of the newer release.

28 changed sections.

Introduction Changed

2026.04

Introduction

Since the release of ChatGPT in November 2022, large language models (LLMs) have rapidly become a major focus of software engineering (SE) research (Hou et al. 2024), yet the reproducibility and replicability of empirical studies involving LLMs remains uncertain. Recent findings indicate a low reproducibility of SE studies involving LLMs (Angermeir et al. 2025). Three characteristics of LLMs make this particularly challenging: their inherent non-determinism causes variability across runs, with slight changes leading to significantly different results (Schroeder and Wood-Doughty 2024; Song et al. 2025; Agarwal et al. 2021; Bjarnason, Silva, and Monperrus 2026); commercial models evolve beyond version identifiers, so reported performance can change over time (L. Chen, Zaharia, and Zou 2024); and even for “open” models, training data and fine-tuning details often remain undisclosed (Gibney 2024). Moreover, prompt formatting choices alone can shift accuracy by tens of percentage points (Sclar et al. 2024), and configured parameters such as temperature affect output variability (Renze 2024). Hence, not reporting these settings directly affects reproducibility.

Although the SE research community has developed guidelines for conducting and reporting specific types of empirical studies such as controlled experiments (e.g., (Wohlin et al. 2024; Shull, Singer, and Sjøberg 2008)), their replications (e.g., (Santos et al. 2021)), or empirical studies in general (e.g., the ACM SIGSOFT Empirical Standards (Ralph et al. 2021)), none of these address the LLM-specific aspects described above. Previously, a position paper highlighted these issues (Wagner et al. 2025), but there was no comprehensive community-developed guidance for designing and reporting empirical studies involving LLMs in SE.

Unchanged text

Therefore, we present community-developed guidelines for conducting and reporting studies involving LLMs in SE research, co-developed by 22 researchers. After outlining our Scope, we introduce a taxonomy of Study Types, then present eight Guidelines. We complement these with an applicability matrix mapping guidelines to study types and a reporting checklist for authors and reviewers. For each study type and guideline, we identify relevant examples, both within and outside of SE research. We maintain the guidelines online as a living resource for the community to use and shape (llm-guidelines.org).

2026.05

Introduction

Since the release of ChatGPT in November 2022, large language models (LLMs) have been adopted widely across software engineering (SE) research (Hou et al. 2024), yet the reproducibility and replicability of empirical studies involving LLMs remains uncertain. Recent findings indicate a low reproducibility of SE studies involving LLMs (Angermeir et al. 2025). Three characteristics of LLMs make this particularly challenging: their inherent non-determinism causes variability across runs (Song et al. 2025; Bjarnason, Silva, and Monperrus 2026), with slight changes leading to substantially different results (Agarwal et al. 2021; Schroeder and Wood-Doughty 2024); commercial models evolve beyond version identifiers, so reported performance can change over time (L. Chen, Zaharia, and Zou 2024); and even for “open” models, training data and fine-tuning details often remain undisclosed (Gibney 2024). Moreover, prompt formatting choices alone can shift accuracy by tens of percentage points (Sclar et al. 2024), and configured parameters such as temperature affect output variability (Renze 2024). Hence, not reporting these settings directly affects reproducibility.

Although the SE research community has developed guidelines for conducting and reporting specific types of empirical studies such as controlled experiments (e.g., (Wohlin et al. 2024; Shull, Singer, and Sjøberg 2008)), their replications (e.g., (Santos et al. 2021)), or empirical studies in general (e.g., the ACM SIGSOFT Empirical Standards (Ralph et al. 2021)), none of these address the LLM-specific aspects described above. Previously, a position paper highlighted these issues (Wagner et al. 2025), but there was no comprehensive community-developed guidance for designing and reporting empirical studies involving LLMs in SE.

Unchanged text

Therefore, we present community-developed guidelines for conducting and reporting studies involving LLMs in SE research, co-developed by 22 researchers. After outlining our Scope, we introduce a taxonomy of Study Types, then present eight Guidelines. We complement these with an applicability matrix mapping guidelines to study types and a reporting checklist for authors and reviewers. For each study type and guideline, we identify relevant examples, both within and outside of SE research. We maintain the guidelines online as a living resource for the community to use and shape (llm-guidelines.org).

Scope and Conventions Changed

2026.04

Scope of Guidelines

Software Engineering as our Target Discipline

We target SE research because existing cross-disciplinary guidelines do not address its needs. For instance, Gallifant et al.’s (Gallifant et al. 2025) guidelines for LLM use in healthcare research include discipline-specific items irrelevant to most SE studies (e.g., clinical-care usability) while lacking guidance on tool architectures, benchmarking, and code-specific evaluation that SE research requires. Moreover, SE research employs a wide variety of empirical methods (Ralph et al. 2021; Wohlin et al. 2024). Our work builds on Sallou, Durieux, and Panichella (2024)’s vision paper on mitigating validity threats in LLM-based SE research (Sallou, Durieux, and Panichella 2024) and our own position paper (Wagner et al. 2025), providing more detailed advice for a wider variety of research methods organized into a taxonomy of Study Types (Section 4).

Focus on Text-Based Use Cases

While multi-modal foundation models that use or generate images, audio, or video may also support SE research and practice, we focus on textual use cases of LLMs (e.g., in natural language or programming languages). Many of our guidelines—particularly those concerning model reporting, prompt documentation, and reproducibility—are likely applicable to multi-modal settings as well, but we leave their explicit validation to future work.

Focus on Direct Development or Research Support

While researchers may use LLMs for many peripheral tasks (e.g., proof-reading, spell-checking, translation), our guidelines focus on their direct role in empirical research and engineering practice. For engineers, we focus on the use of LLMs to automate SE tasks, that is, artificial intelligence (AI) for software engineering (AI4SE) (see LLMs for SE ). This includes agentic systems that autonomously plan and execute multi-step tasks using LLMs (see  System and Prompt Design G3: System and Prompt Design  ). For researchers, we focus on the use of LLMs to automate empirical research tasks such as data collection, processing, or analysis (see LLMs for Research ).

Unchanged text

Researchers as our Target Audience

Our guidelines are intended to help SE researchers design, plan, conduct, and report empirical studies involving LLMs, and to support scholarly peer review of such studies. Each guideline includes an Advice for Reviewers subsection with targeted assessment suggestions. Our guidelines focus on what to report and how; they complement but do not replace methodological guidance for designing specific types of empirical studies.

Reporting Locations

We use PAPER to refer to the manuscript PDF, including any appendices bound with it. We use SUPPLEMENTARY MATERIAL to refer to any artifact, replication package, dataset, or other resource external to the manuscript PDF (e.g., hosted on Zenodo or Figshare). What belongs in the main body versus an appendix is a presentation choice for authors and venues; what belongs inside the manuscript versus outside it is the reporting-location decision the guidelines make. When a recommendation does not specify a location, either PAPER or SUPPLEMENTARY MATERIAL is acceptable.

How to Navigate this Paper

This paper is structured to support different reading strategies depending on the reader’s goal. Researchers planning a new study may start with the taxonomy of study types (see Study Types ) to identify which types apply to their planned work, then consult Table 3 to determine which guidelines are requirements (MUST ) and which are recommendations (SHOULD ) for those study types. Each guideline section opens with a tl;dr summary in a shaded box, allowing readers to quickly assess relevance before reading the full text. Researchers writing up results may prefer to start with the checklist in Appendix 7, which organizes actionable items by typical paper sections (Introduction, Research Design and Methods, Results, etc.). Table 2 provides a quick reference to all eight guidelines, and Table 4 maps each guideline’s rationale to its key recommendations. Reviewers can use the Advice for Reviewers subsection at the end of each guideline for targeted guidance on assessing manuscripts. Table 3 helps reviewers identify which guidelines apply to the study type under review.

2026.05

Scope and Conventions

Software Engineering as our Target Discipline

We target SE research because existing cross-disciplinary guidelines do not address its needs. For instance, Gallifant et al. (2025)’s (Gallifant et al. 2025) guidelines for LLM use in healthcare research include discipline-specific items irrelevant to most SE studies (e.g., clinical-care usability) while lacking guidance on tool architectures, benchmarking, and code-specific evaluation that SE research requires. Moreover, SE research employs a wide variety of empirical methods (Ralph et al. 2021; Wohlin et al. 2024). Our work builds on Sallou, Durieux, and Panichella (2024)’s vision paper on mitigating validity threats in LLM-based SE research (Sallou, Durieux, and Panichella 2024) and our own position paper (Wagner et al. 2025), providing more detailed advice for a wider variety of research methods organized into a taxonomy of Study Types.

Focus on Text-Based Use Cases

While multi-modal foundation models that use or generate images, audio, or video may also support SE research and practice, we focus on textual use cases of LLMs (e.g., in natural language or programming languages). Many of our guidelines, particularly those concerning model reporting, prompt documentation, and reproducibility, are likely applicable to multi-modal settings as well, but we leave their explicit validation to future work.

Focus on Direct Development or Research Support

While researchers may use LLMs for many peripheral tasks (e.g., proof-reading, spell-checking, translation), our guidelines focus on their direct role in empirical research and engineering practice. For engineers, we focus on the use of LLMs to automate SE tasks, that is, artificial intelligence (AI) for software engineering (AI4SE) (see LLMs for SE ). This includes agentic systems that autonomously plan and execute multi-step tasks using LLMs (see System and Prompt Design ). For researchers, we focus on the use of LLMs to automate empirical research tasks such as data collection, processing, or analysis (see LLMs for Research ).

Unchanged text

Researchers as our Target Audience

Our guidelines are intended to help SE researchers design, plan, conduct, and report empirical studies involving LLMs, and to support scholarly peer review of such studies. Each guideline includes an Advice for Reviewers subsection with targeted assessment suggestions. Our guidelines focus on what to report and how; they complement but do not replace methodological guidance for designing specific types of empirical studies.

Reporting Locations

We use paper to refer to the manuscript PDF, including any appendices bound with it. We use supplementary material to refer to any artifact, replication package, dataset, or other resource external to the manuscript PDF (e.g., hosted on Zenodo or Figshare). What belongs in the main body versus an appendix is a presentation choice for authors and venues; what belongs inside the manuscript versus outside it is the reporting-location decision the guidelines make. When a recommendation does not specify a location, either paper or supplementary material is acceptable.

Navigating These Guidelines

These guidelines are structured to support different reading strategies depending on the reader’s goal. Researchers planning a new study may start with the taxonomy of study types (see Study Types ) to identify which types apply to their planned work, then consult the applicability matrix to determine which guidelines are requirements (must ) and which are recommendations (should ) for those study types. Each guideline section opens with a brief summary in a shaded box, allowing readers to quickly assess relevance before reading the full text. Researchers writing up results may prefer to start with the checklist, which organizes actionable items by typical paper sections (Introduction, Research Design and Methods, Results, etc.). Reviewers can use the Advice for Reviewers subsection at the end of each guideline for targeted guidance on assessing manuscripts.

Methodology Changed

2026.04

Methodology

This project was initiated at the 2024 meeting of the International Software Engineering Research Network (ISERN) in Barcelona, Spain, where the first and last author organized a session on guidelines for empirical studies involving LLMs. These discussions led to a position paper at WSESE 2025 (Wagner et al. 2025), which was presented at the Copenhagen Symposium on Human-Centered Software Engineering AI (CHASEAI 2024), forming a workstream to collaboratively refine and extend the preliminary guidelines. Our process follows an established tradition of developing reporting guidelines through expert collaboration, as exemplified by CONSORT for clinical trials (Begg et al. 1996) and, more recently, TRIPOD-LLM for LLM-based prediction studies in medicine (Gallifant et al. 2025).

We held bi-weekly meetings, in which we decided on the structure of the study type taxonomy and the guidelines. During these discussions, we added a new study type ( LLMs for Synthesis S3: LLMs for Synthesis  ), refined and extended the guideline sections (in particular Version and Configuration G2: Version and Configuration , System and Prompt Design G3: System and Prompt Design , Session Traces G4: Session Traces  ) and covered Benchmarks and Metrics G5: Benchmarks and Metrics  as well as Limitations and Mitigations G8: Limitations and Mitigations .

Then, at least two co-authors were assigned to each study type and guideline to refine and extend the content, followed by a peer review and refinement phase. Afterwards, the first author reviewed all sections and discussed potential changes with the topic leads. A requirement for co-authorship was that each author contributed or reviewed content and, most importantly, read the complete guidelines and confirmed that they stand behind all recommendations, not only their own contributions.

Unchanged text

To select illustrative examples for each study type and guideline, co-author pairs independently searched for relevant papers using academic databases. This initial pool was extended based on suggestions from other co-authors. Identified papers were assessed by one author and subsequently reviewed by a second author; papers that did not add new insights beyond those already covered were not included. Multiple review rounds were then carried out on the guidelines as a whole, during which selected papers were cross-checked by additional authors and, where necessary, further examples were added or replaced. The examples are intended to be illustrative, not the result of a systematic review; the involvement of multiple authors at different stages helped mitigate individual selection bias.

After completing the internal review, we invited external experts in empirical SE to review the study types and guidelines. During the review process of the Empirical Software Engineering journal, the structure and content were improved based on reviewers’ suggestions. We self-assessed our guidelines against the quality dimensions of the SIGSOFT Empirical Standards (Ralph et al. 2021); this assessment is reported in the conclusion.

2026.05

Methodology

This project was initiated at the 2024 meeting of the International Software Engineering Research Network (ISERN) in Barcelona, Spain, where the first and last author organized a session on guidelines for empirical studies involving LLMs. These discussions led to a position paper at WSESE 2025 (Wagner et al. 2025), which was presented at the Copenhagen Symposium on Human-Centered Software Engineering AI (CHASEAI 2024), forming a workstream to collaboratively refine and extend the preliminary guidelines. Our process follows an established tradition of developing reporting guidelines through expert collaboration, as exemplified by CONSORT for clinical trials (Begg et al. 1996) and, more recently, TRIPOD-LLM for LLM-based prediction studies in medicine (Gallifant et al. 2025).

We held bi-weekly meetings, in which we decided on the structure of the study type taxonomy and the guidelines. During these discussions, we added a new study type (LLMs for Synthesis ), refined and extended the guideline sections (in particular Version and Configuration, System and Prompt Design, Session Traces ) and covered Benchmarks and Metrics as well as Limitations and Mitigations.

Then, at least two co-authors were assigned to each study type and guideline to refine and extend the content, followed by a peer review and refinement phase. Afterwards, the first author reviewed all sections and discussed potential changes with the topic leads. A requirement for co-authorship was that each author contributed or reviewed content and, most importantly, read the complete guidelines and confirmed that they stand behind all recommendations, not only their own contributions.

Unchanged text

To select illustrative examples for each study type and guideline, co-author pairs independently searched for relevant papers using academic databases. This initial pool was extended based on suggestions from other co-authors. Identified papers were assessed by one author and subsequently reviewed by a second author; papers that did not add new insights beyond those already covered were not included. Multiple review rounds were then carried out on the guidelines as a whole, during which selected papers were cross-checked by additional authors and, where necessary, further examples were added or replaced. The examples are intended to be illustrative, not the result of a systematic review; the involvement of multiple authors at different stages helped mitigate individual selection bias.

After completing the internal review, we invited external experts in empirical SE to review the study types and guidelines. During the review process of the Empirical Software Engineering journal, the structure and content were improved based on reviewers’ suggestions. We self-assessed our guidelines against the quality dimensions of the SIGSOFT Empirical Standards (Ralph et al. 2021); this assessment is reported in the conclusion.

Study Types Changed

2026.04

Study Types

Empirical guidelines are needed for SE studies involving LLMs to ensure the validity and reproducibility of their results. However, such guidelines must be tailored to different study types that each pose unique challenges. Therefore, we created a taxonomy of study types to contextualize our recommendations. We use the term study type to refer to categories of LLM involvement in empirical research, rather than to research methodologies (e.g., experiments, case studies). A single empirical study may involve multiple such study types. Each study type section starts with a description, followed by examples from the SE research community and beyond. The advantages and challenges of using LLMs are discussed in a summary subsection at the end of this section, synthesizing cross-cutting themes across all study types.

2026.05

Study Types

Empirical guidelines are needed for SE studies involving LLMs to ensure the validity and reproducibility of their results. However, such guidelines must be tailored to different study types that each pose unique challenges. Therefore, we created a taxonomy of study types to contextualize our recommendations. We use the term study type to refer to categories of LLM involvement in empirical research, rather than to research methodologies (e.g., experiments, case studies). A single empirical study may involve multiple such study types. Each study type section starts with a description, followed by examples from the SE research community and beyond. The advantages and challenges of using LLMs are discussed in a summary subsection at the end of this section, synthesizing cross-cutting themes across all study types.

LLMs as Tools for Software Engineering Researchers Changed

2026.04

LLMs as Tools for Software Engineering Researchers

LLMs can serve as tools to help researchers conduct empirical studies. They can automate various tasks such as data collection, pre-processing, and analysis. For example, LLMs can apply predefined coding guides to qualitative datasets ( LLMs as Annotators S1: LLMs as Annotators  ), assess the quality of software artifacts ( LLMs as Judges S2: LLMs as Judges  ), generate summaries of research papers ( LLMs for Synthesis S3: LLMs for Synthesis  ), or simulate human behavior in empirical studies ( LLMs as Subjects S4: LLMs as Subjects  ). If LLMs could complete these tasks well enough, they would reduce the time and effort required to conduct a study. However, these applications each pose challenges to validity and reproducibility.

2026.05

LLMs as Tools for Software Engineering Researchers

LLMs can serve as tools to help researchers conduct empirical studies. They can automate various tasks such as data collection, pre-processing, and analysis. For example, LLMs can apply predefined coding guides to qualitative datasets (LLMs as Annotators ), assess the quality of software artifacts (LLMs as Judges ), generate summaries of research papers (LLMs for Synthesis ), or simulate human behavior in empirical studies (LLMs as Subjects ). If LLMs could complete these tasks well enough, they would reduce the time and effort required to conduct a study. However, these applications each pose challenges to validity and reproducibility.

LLMs as Annotators Changed

2026.04

LLMs as Annotators (S1)

Description

In qualitative data analysis, manually annotating (“coding”) natural language text (e.g., requirements, interview transcripts, open-ended survey responses) is a time-consuming manual process (Bano et al. 2024). LLMs can augment human coding, suggest new codes, and label artifacts based on a pre-defined coding guide much faster than humans can (Z. He et al. 2024). The extent to which, or under what conditions, LLMs can perform these tasks effectively remains an open research question (T. Ahmed et al. 2025). Indeed, measuring their effectiveness is practically and philosophically challenging. From an interpretivist philosophical perspective, one cannot measure the quality of analysis by comparing one (human or machine) analyst’s work to another. From a realist perspective, triangulating across multiple human judges, LLM judges, and other data sources (e.g., whether a pull request is marked as having resolved an issue) improves confidence in the findings but does not prove that any one judge is valid.

Examples

J. Huang et al. (2024) (J. Huang et al. 2024) utilized multiple LLMs for joint annotation of mobile application reviews. They used three models of comparable size with an absolute majority voting rule (i.e., a label is only accepted if it receives more than half of the total votes from the models). This approach slightly outperformed the best individual model tested. Meanwhile, T. Ahmed et al. (2025) (T. Ahmed et al. 2025) examined LLMs as annotators in SE research across five datasets, six LLMs, and ten annotation tasks. They found that model-model agreement strongly correlates with human-model agreement; models performed poorly in tasks where humans also frequently disagreed. They proposed to use model confidence scores to identify specific samples that could be safely delegated to LLMs, potentially reducing human annotation effort without compromising inter-rater agreement.

2026.05

LLMs as Annotators

Description

In qualitative data analysis, manually annotating (“coding”) natural language text (e.g., requirements, interview transcripts, open-ended survey responses) is a time-consuming manual process (Bano et al. 2024). LLMs can augment human coding, suggest new codes, and label artifacts based on a pre-defined coding guide much faster than humans can (Z. He et al. 2024). The extent to which, or under what conditions, LLMs can perform these tasks effectively remains an open research question (T. Ahmed et al. 2025). Indeed, measuring their effectiveness is practically and philosophically challenging. From an interpretivist philosophical perspective, one cannot measure the quality of analysis by comparing one (human or machine) analyst’s work to another. From a realist perspective, triangulating across multiple human judges, LLM judges, and other data sources (e.g., whether a pull request is marked as having resolved an issue) improves confidence in the findings but does not prove that any one judge is valid.

Examples

J. Huang et al. (2024) (J. Huang et al. 2024) used multiple LLMs for joint annotation of mobile application reviews. They used three models of comparable size with an absolute majority voting rule (i.e., a label is only accepted if it receives more than half of the total votes from the models). This approach slightly outperformed the best individual model tested. Meanwhile, T. Ahmed et al. (2025) (T. Ahmed et al. 2025) examined LLMs as annotators in SE research across five datasets, six LLMs, and ten annotation tasks. They found that model-model agreement strongly correlates with human-model agreement; models performed poorly in tasks where humans also frequently disagreed. They proposed to use model confidence scores to identify specific samples that could be safely delegated to LLMs, potentially reducing human annotation effort without compromising inter-rater agreement.

LLMs as Judges Changed

2026.04

LLMs as Judges (S2)

Description

LLMs can rate properties of software artifacts (e.g. assess code readability, adherence to coding standards, or comment quality) or sort multiple solutions by some attribute. These kinds of judgments are distinct from annotating unstructured text (see LLMs as Annotators S1: LLMs as Annotators  ).

Example(s)

Lubos et al. (2024) (Lubos et al. 2024) leveraged Llama-2 to evaluate the quality of software requirements statements. They prompted the LLM with the text below, where the words in braces reflect the study parameters:

Your task is to evaluate the quality of a software requirement. Evaluate
whether the following requirement is quality_characteristic.
quality_characteristic means: quality_characteristic_explanation The
evaluation result must be: ’yes’ or ’no’. Request: Based on the
following description of the project: project_description Evaluate the
quality of the following requirement: requirement. Explain your decision
and suggest an improved version.

They evaluated LLM output against expert human judges and found moderate agreement for simple requirements and poor agreement for more complex requirements. In contrast, F. Wang et al. (2025) (F. Wang et al. 2025) used an LLM to generate acceptance criteria for user stories, and provided a rubric to an LLM to judge the generated acceptance criteria on interpretable scales (0 to 4).

2026.05

LLMs as Judges

Description

LLMs can rate properties of software artifacts (e.g. assess code readability, adherence to coding standards, or comment quality) or sort multiple solutions by some attribute. These kinds of judgments are distinct from annotating unstructured text (see LLMs as Annotators ).

Examples

Lubos et al. (2024) (Lubos et al. 2024) used Llama-2 to evaluate the quality of software requirements statements. They prompted the LLM with the text below, where the words in braces reflect the study parameters:

Your task is to evaluate the quality of a software requirement. Evaluate
whether the following requirement is quality_characteristic.
quality_characteristic means: quality_characteristic_explanation The
evaluation result must be: ’yes’ or ’no’. Request: Based on the
following description of the project: project_description Evaluate the
quality of the following requirement: requirement. Explain your decision
and suggest an improved version.

They evaluated LLM output against expert human judges and found moderate agreement for simple requirements and poor agreement for more complex requirements. In contrast, F. Wang et al. (2025) (F. Wang et al. 2025) used an LLM to generate acceptance criteria for user stories, and provided a rubric to an LLM to judge the generated acceptance criteria on interpretable scales (0 to 4).

LLMs for Synthesis Changed

2026.04

LLMs for Synthesis (S3)

Description

Unlike annotation (see LLMs as Annotators S1: LLMs as Annotators  ), which focuses on categorizing or labeling individual data points, synthesis refers to the process of integrating and interpreting information from multiple sources to generate higher-level insights, identify patterns across datasets, and develop conceptual frameworks or theories. LLMs may be able to support synthesis tasks in SE research by processing and distilling information from qualitative data sources. Although synthesis in the preceding notion refers to abstraction and interpretation across multiple data sources, the term is sometimes also used to refer to generating synthetic content (e.g., source code, bug-fix pairs, requirements, etc.) that are then used in downstream tasks to train, fine-tune, or evaluate existing models or tools. In this case, the synthesis is done primarily using the LLM and its training data; the input is limited to basic instructions and examples.

Example(s)

Published examples of applying LLMs for synthesis in SE remain scarce; however, some recent work in other domains is instructive (Bano et al. 2024). Barros et al. (2025) (Barros et al. 2025) conducted a systematic mapping study on using LLMs for qualitative research and found that most studies focused on the healthcare and social sciences. In these studies, LLMs supported qualitative methods, such as grounded theory and thematic analysis, by aiding in pattern identification. In SE, de Morais Leca et al. (Morais Leça et al. 2025) explored how LLMs have been applied for qualitative data analysis (QDA) and proposed general strategies and guidelines for their application. Ornelas et al. (Ornelas et al. 2025) complemented this perspective by studying the opportunities and limitations of introducing LLM-based support into QDA, and by formulating recommendations for embedding human–AI collaboration across the thematic analysis phases. Building on these, subsequent work has proposed hybrid frameworks combining LLM support with human-led QDA. Rashid et al. (Rasheed et al. 2024) designed an LLM-driven multi-agent system that integrates AI with human decision-making to automate qualitative data analysis methods. Their system generated initial codes, developed themes, and summarized text. Similarly, Montes et al. (Montes et al. 2025) compared the performance of humans and LLMs in coding, theme development, definition, and refinement, creating guidelines for a hybrid-LLM framework. Finally, El-Hajjami and Salinesi (2025)’s work is an example of using LLMs to create synthetic datasets. They present an approach to generate synthetic requirements, showing that they “can match or surpass human-authored requirements for specific classification tasks” (El-Hajjami and Salinesi 2025).

2026.05

LLMs for Synthesis

Description

Unlike annotation (see LLMs as Annotators ), which focuses on categorizing or labeling individual data points, synthesis refers to the process of integrating and interpreting information from multiple sources to generate higher-level insights, identify patterns across datasets, and develop conceptual frameworks or theories. LLMs may be able to support synthesis tasks in SE research by processing and distilling information from qualitative data sources. Although synthesis in the preceding notion refers to abstraction and interpretation across multiple data sources, the term is sometimes also used to refer to generating synthetic content (e.g., source code, bug-fix pairs, or requirements) that is then used in downstream tasks to train, fine-tune, or evaluate existing models or tools. In this case, the synthesis is done primarily using the LLM and its training data; the input is limited to basic instructions and examples.

Examples

Published examples of applying LLMs for synthesis in SE remain scarce; however, some recent work in other domains is instructive (Bano et al. 2024). Barros et al. (2025) (Barros et al. 2025) conducted a systematic mapping study on using LLMs for qualitative research and found that most studies focused on the healthcare and social sciences. In these studies, LLMs supported qualitative methods, such as grounded theory and thematic analysis, by aiding in pattern identification. In SE, Morais Leça et al. (2025) (Morais Leça et al. 2025) explored how LLMs have been applied for qualitative data analysis (QDA) and proposed general strategies and guidelines for their application. Ornelas et al. (2025) (Ornelas et al. 2025) complemented this perspective by studying the opportunities and limitations of introducing LLM-based support into QDA, and by formulating recommendations for embedding human–AI collaboration across the thematic analysis phases. Building on these, subsequent work has proposed hybrid frameworks combining LLM support with human-led QDA. Rasheed et al. (2024) (Rasheed et al. 2024) designed an LLM-driven multi-agent system that integrates AI with human decision-making to automate qualitative data analysis methods. Their system generated initial codes, developed themes, and summarized text. Similarly, Montes et al. (2025) (Montes et al. 2025) compared the performance of humans and LLMs in coding, theme development, definition, and refinement, creating guidelines for a hybrid-LLM framework. Finally, El-Hajjami and Salinesi (2025)’s work is an example of using LLMs to create synthetic datasets. They present an approach to generate synthetic requirements, showing that they “can match or surpass human-authored requirements for specific classification tasks” (El-Hajjami and Salinesi 2025).

LLMs as Subjects Changed

2026.04

LLMs as Subjects (S4)

Description

In empirical studies, data is collected from participants through methods such as surveys, interviews, or controlled experiments. LLMs can serve as virtual subjects by simulating human behavior and interactions. If LLMs can generate responses that approximate those of human participants, they could be valuable for research involving user interactions, collaborative coding environments, and software usability assessments (Zhao, Habule, and Zhang 2025). To achieve this, prompt engineering techniques are widely employed; for instance, the Personas Pattern (Kong et al. 2023) involves tailoring LLM responses to align with predefined profiles or roles that emulate specific user archetypes. To serve as virtual subjects, generated responses should be indistinguishable from human-produced texts, consistent with the attitudes and sociodemographic information of the conditioning context (e.g., junior vs. senior developers), naturally aligned with the form, tone, and content of the simulated scenario, and reflect patterns in relationships between ideas, demographics, and behavior observed in comparable human data (Argyle et al. 2022).

Example(s)

R. Xu et al. (2024) (R. Xu et al. 2024) compiled a list of ways LLMs can support social science research, some of which transfer to empirical SE research. For example, LLMs can emulate human responses and behaviors in simulated interviews and focus groups Gerosa et al. (2024) (Gerosa et al. 2024). Similarly, Bano, Gunatilake, and Hoda (2025) (Bano, Gunatilake, and Hoda 2025), investigated biases in LLM-generated candidate profiles in SE recruitment processes. They found biases favoring male candidates, lighter skin tones, and slim physiques, particularly for senior roles. LLMs may be able to simulate end-user feedback and behavior in usability studies, identify usability issues and offering suggestions for improvement based on predefined user personas.

2026.05

LLMs as Subjects

Description

In empirical studies, data is collected from participants through methods such as surveys, interviews, or controlled experiments. LLMs can serve as virtual subjects by simulating human behavior and interactions. If LLMs can generate responses that approximate those of human participants, they could be valuable for research involving user interactions, collaborative coding environments, and software usability assessments (Zhao, Habule, and Zhang 2025). To achieve this, prompt engineering techniques are widely employed; for instance, the Personas Pattern (Kong et al. 2024) involves tailoring LLM responses to align with predefined profiles or roles that emulate specific user archetypes. To serve as virtual subjects, generated responses should be indistinguishable from human-produced texts, consistent with the attitudes and sociodemographic information of the conditioning context (e.g., junior vs. senior developers), naturally aligned with the form, tone, and content of the simulated scenario, and reflect patterns in relationships between ideas, demographics, and behavior observed in comparable human data (Argyle et al. 2022).

Examples

R. Xu et al. (2024) (R. Xu et al. 2024) compiled a list of ways LLMs can support social science research, some of which transfer to empirical SE research. For example, LLMs can emulate human responses and behaviors in simulated interviews and focus groups (Gerosa et al. 2024). Similarly, Bano, Gunatilake, and Hoda (2025) (Bano, Gunatilake, and Hoda 2025), investigated biases in LLM-generated candidate profiles in SE recruitment processes. They found biases favoring male candidates, lighter skin tones, and slim physiques, particularly for senior roles. LLMs may be able to simulate end-user feedback and behavior in usability studies, identify usability issues, and offer suggestions for improvement based on predefined user personas.

LLMs as Tools for Software Engineers Changed

2026.04

LLMs as Tools for Software Engineers

LLM-based assistants have become a widely adopted tool for software engineers (Stack Overflow 2025), supporting them in various tasks such as code generation and debugging. Researchers have studied how software engineers use LLMs ( Studying LLM Usage S5: Studying LLM Usage  ), developed new tools that integrate LLMs ( LLMs for Tools S6: LLMs for Tools  ), and benchmarked LLMs for software engineering tasks ( Benchmarking LLMs S7: Benchmarking LLMs  ).

2026.05

LLMs as Tools for Software Engineers

LLM-based assistants have become a widely adopted tool for software engineers (Stack Overflow 2025), supporting them in various tasks such as code generation and debugging. Researchers have studied how software engineers use LLMs (Studying LLM Usage ), developed new tools that integrate LLMs (LLMs for Tools ), and benchmarked LLMs for software engineering tasks (Benchmarking LLMs ).

Studying LLM Usage in Software Engineering Changed

2026.04

Studying LLM Usage in Software Engineering (S5)

Description

Studying how software engineers use LLMs and LLM-based tools is important for understanding the current state of practice in SE. Researchers can observe software engineers’ usage of LLM-based tools in the field, or study if and how they adopt such tools, their usage patterns, as well as perceived benefits and challenges. Surveys, interviews, observational studies, or analysis of usage logs can provide insights into how LLMs are integrated into development processes, how they influence decision making, and what factors affect their acceptance and effectiveness. Such studies can inform improvements for existing LLM-based tools, motivate the design of novel tools, or derive best practices for LLM-assisted software engineering. They can also uncover risks or deficiencies of existing tools.

Example(s)

Based on a convergent mixed-methods study, Russo (2024) has found that early adoption of generative AI by software engineers is primarily driven by compatibility with existing workflows (Russo 2024). Khojah et al. (2024) investigated the use of ChatGPT (GPT-3.5) by professional software engineers in a week-long observational study (Khojah et al. 2024). They found that most developers do not use the code generated by ChatGPT directly but instead use the output as a guide to implement their own solutions. Azanza et al. (2024)’s case study found that LLMs could “enhance personalized, instant onboarding support; however, relying on proprietary external LLMs poses significant data privacy risks” (Azanza et al. 2024). Jahic and Sami (2024) found that most participants from 15 software companies they surveyed had already adopted AI (especially ChatGPT) for SE tasks; but cited copyright and privacy issues, as well as inconsistent or low-quality outputs, as barriers to adoption (Jahic and Sami 2024). Retrospective studies that analyze data generated while developers use LLMs can provide additional insights into human-LLM interactions. For example, researchers can employ data mining methods to build large-scale conversation datasets, such as the DevGPT dataset introduced by Xiao et al. (2024) (Xiao et al. 2024). Conversations can then be analyzed using quantitative (Rabbi et al. 2024) and qualitative (Mohamed, Parvin, and Parra 2024) analysis methods.

2026.05

Studying LLM Usage in Software Engineering

Description

Studying how software engineers use LLMs and LLM-based tools is important for understanding the current state of practice in SE. Researchers can observe software engineers’ usage of LLM-based tools in the field, or study if and how they adopt such tools, their usage patterns, as well as perceived benefits and challenges. Surveys, interviews, observational studies, or analysis of usage logs can provide insights into how LLMs are integrated into development processes, how they influence decision making, and what factors affect their acceptance and effectiveness. Such studies can inform improvements for existing LLM-based tools, motivate the design of novel tools, or derive best practices for LLM-assisted software engineering. They can also uncover risks or deficiencies of existing tools.

Examples

Based on a convergent mixed-methods study, Russo (2024) has found that early adoption of generative AI by software engineers is primarily driven by compatibility with existing workflows (Russo 2024). Khojah et al. (2024) investigated the use of ChatGPT (GPT-3.5) by professional software engineers in a week-long observational study (Khojah et al. 2024). They found that most developers do not use the code generated by ChatGPT directly but instead use the output as a guide to implement their own solutions. Azanza et al. (2024)’s case study found that LLMs could “enhance personalized, instant onboarding support; however, relying on proprietary external LLMs poses significant data privacy risks” (Azanza et al. 2024). Jahic and Sami (2024) found that most participants from 15 software companies they surveyed had already adopted AI (especially ChatGPT) for SE tasks; but cited copyright and privacy issues, as well as inconsistent or low-quality outputs, as barriers to adoption (Jahic and Sami 2024). Retrospective studies that analyze data generated while developers use LLMs can provide additional insights into human-LLM interactions. For example, researchers can employ data mining methods to build large-scale conversation datasets, such as the DevGPT dataset introduced by Xiao et al. (2024) (Xiao et al. 2024). Conversations can then be analyzed using quantitative (Rabbi et al. 2024) and qualitative (Mohamed, Parvin, and Parra 2024) analysis methods.

LLMs for New Software Engineering Tools Changed

2026.04

LLMs for New Software Engineering Tools (S6)

Description

LLMs are being integrated into new tools that support software engineers in their daily tasks (e.g., to assist in code comprehension (Yan et al. 2024) and test case generation (Schäfer et al. 2024)). One way of integrating LLM-based tools into software engineers’ workflows is using GenAI agents. Unlike traditional LLM-based tools, these agents are capable of acting autonomously and proactively, are often tailored to meet specific user needs, and can interact with external environments (Takerngsaksiri et al. 2025; Wiesinger, Marlow, and Vuskovic 2025; Y. Yang et al. 2025). From an architectural perspective, GenAI agents can be implemented in various ways (Wiesinger, Marlow, and Vuskovic 2025). However, they generally share three key components: (1) a reasoning mechanism that guides the LLM (often enabled by advanced prompt engineering), (2) a set of tools to interact with external systems (e.g., APIs or databases), and (3) a user communication interface that extends beyond traditional chat-based interactions (Richards and Wessel 2024; Sumers et al. 2024; W. Zhou et al. 2023). Researchers can also test and compare different tool architectures to increase artifact quality and developer satisfaction.

Example(s)

Yan et al. (2024) proposed IVIE, a tool integrated into the VS Code graphical interface that generates and explains code using LLMs (Yan et al. 2024). The authors focused more on the presentation, providing a user-friendly interface to interact with the LLM. Schäfer et al. (2024) (Schäfer et al. 2024) presented a large-scale empirical evaluation on the effectiveness of LLMs for automated unit test generation. They presented TestPilot, a tool that implements an approach in which the LLM is provided with prompts that include the signature and implementation of a function under test, along with usage examples extracted from the documentation. Richards and Wessel (2024) introduced a preliminary GenAI agent designed to assist developers in understanding source code by incorporating a reasoning component grounded in the theory of mind (Richards and Wessel 2024).

2026.05

LLMs for New Software Engineering Tools

Description

LLMs are being integrated into new tools that support software engineers in their daily tasks (e.g., to assist in code comprehension (Yan et al. 2024) and test case generation (Schäfer et al. 2024)). One way of integrating LLM-based tools into software engineers’ workflows is using GenAI agents. Unlike traditional LLM-based tools, these agents are capable of acting autonomously and proactively, are often tailored to meet specific user needs, and can interact with external environments (Wiesinger, Marlow, and Vuskovic 2025; Y. Yang et al. 2025). From an architectural perspective, GenAI agents can be implemented in various ways (Wiesinger, Marlow, and Vuskovic 2025). However, they generally share three key components: (1) a reasoning mechanism that guides the LLM (often enabled by advanced prompt engineering), (2) a set of tools to interact with external systems (e.g., APIs or databases), and (3) a user communication interface that extends beyond traditional chat-based interactions (Sumers et al. 2024; W. Zhou et al. 2023). Researchers can also test and compare different tool architectures to increase artifact quality and developer satisfaction.

Examples

Yan et al. (2024) proposed IVIE, a tool integrated into the VS Code graphical interface that generates and explains code using LLMs (Yan et al. 2024). The authors focused more on the presentation, providing a user-friendly interface to interact with the LLM. Schäfer et al. (2024) (Schäfer et al. 2024) presented a large-scale empirical evaluation on the effectiveness of LLMs for automated unit test generation. They presented TestPilot, a tool that implements an approach in which the LLM is provided with prompts that include the signature and implementation of a function under test, along with usage examples extracted from the documentation. Richards and Wessel (2024) introduced a preliminary GenAI agent designed to assist developers in understanding source code by incorporating a reasoning component grounded in the theory of mind (Richards and Wessel 2024). Takerngsaksiri et al. (2025) (Takerngsaksiri et al. 2025) present HULA, a multi-agent system deployed in Atlassian JIRA that lets engineers refine LLM-generated coding plans and source code, and report acceptance and modification rates from real users.

Benchmarking LLMs for Software Engineering Tasks Changed

2026.04

Benchmarking LLMs for Software Engineering Tasks (S7)

Description

Benchmarking is the process of evaluating an LLM’s performance on standardized tasks, using standardized metrics, on a standardized dataset. The LLM output is compared to a supposed ground truth from the benchmark dataset. Typical tasks include code generation, code summarization, code completion, and code repair (Y. Yang et al. 2025), but also natural language processing tasks, such as anaphora resolution (i.e., the task of identifying the referring expression of a word or phrase occurring earlier in the text). Metrics may include general metrics for text generation, such as ROUGE, BLEU, or METEOR (Hou et al. 2024), or task-specific metrics, such as CodeBLEU for code generation. Benchmarking requires high-quality reference datasets.

Example(s)

In SE, benchmarking may include evaluating an LLM’s ability to produce accurate and reliable outputs for a given input (usually a task description, possibly accompanied by data obtained from curated real-world projects or from synthetic SE-specific datasets). RepairBench (Silva and Monperrus 2025), for example, contains 574 buggy Java methods and their corresponding fixed versions, which can be used to evaluate the performance of LLMs in code repair tasks. It uses the Plausible@1 metric (i.e., the probability that the first generated patch passes all test cases) and the AST Match@1 metric (i.e., the probability that the abstract syntax tree of the first generated patch matches the ground truth patch). SWE-Bench (Jimenez et al. 2024) is a more generic benchmark that contains 2,294 SE Python tasks extracted from GitHub pull requests. To score the LLM’s task performance, the benchmark validates whether the generated patch successfully compiles and calculates the percentage of passed test cases. Meanwhile, HumanEval (M. Chen et al. 2021) is often used to assess code generation.

2026.05

Benchmarking LLMs for Software Engineering Tasks

Description

Benchmarking is the process of evaluating an LLM’s performance on standardized tasks, using standardized metrics, on a standardized dataset. The LLM output is compared to a supposed ground truth from the benchmark dataset. Typical tasks include code generation, code summarization, code completion, and code repair (Y. Yang et al. 2025), but also natural language processing tasks, such as anaphora resolution (i.e., the task of identifying the referring expression of a word or phrase occurring earlier in the text). Metrics may include general metrics for text generation, such as ROUGE, BLEU, or METEOR (Hou et al. 2024), or task-specific metrics, such as CodeBLEU for code generation. Benchmarking requires high-quality reference datasets.

Examples

In SE, benchmarking may include evaluating an LLM’s ability to produce accurate and reliable outputs for a given input (usually a task description, possibly accompanied by data obtained from curated real-world projects or from synthetic SE-specific datasets). RepairBench (Silva and Monperrus 2025), for example, contains 574 buggy Java methods and their corresponding fixed versions, which can be used to evaluate the performance of LLMs in code repair tasks. It uses the Plausible@1 metric (i.e., the probability that the first generated patch passes all test cases) and the AST Match@1 metric (i.e., the probability that the abstract syntax tree of the first generated patch matches the ground truth patch). SWE-Bench (Jimenez et al. 2024) is a more generic benchmark that contains 2,294 SE Python tasks extracted from GitHub pull requests. To score the LLM’s task performance, the benchmark validates whether the generated patch successfully compiles and calculates the percentage of passed test cases. Meanwhile, HumanEval (M. Chen et al. 2021) is often used to assess code generation.

Advantages and Challenges Changed

2026.04

Advantages and Challenges

Using LLMs in empirical SE research, whether as tools for researchers or for software engineers, offers several potential advantages but also raises fundamental challenges that cut across the study types described above.

Advantages

The primary advantage of using LLMs for research tasks is speed, cost reduction, and scalability. LLMs can annotate, judge, synthesize, and simulate faster and at lower cost than human researchers, with studies showing cost reductions of 50–96% on various natural language tasks (S. Wang et al. 2021) (e.g., Z. He et al. (2024) found that GPT-4 annotation required only two days and 122.08 USD compared to several weeks and 4,508 USD for a comparable MTurk pipeline (Z. He et al. 2024)). Similarly, augmenting or replacing human participants with LLM-generated virtual participants would reduce recruitment effort (Madampe et al. 2024). This efficiency unlocks scalability: qualitative research traditionally does not scale well to large samples, but LLMs allow researchers to analyze larger datasets (Bano et al. 2024; Barros et al. 2025; Morais Leça et al. 2025) and support larger judgment datasets.

LLMs can also automate tasks such as coding qualitative data, assessing artifact quality, and generating summaries, reducing cognitive demands and resources required for qualitative research. Some studies suggest that LLMs may improve consistency: ChatGPT’s accuracy exceeded crowd workers by approximately 25%, and LLMs can achieve higher inter-rater agreement than crowd workers and trained annotators (Gilardi, Alizadeh, and Kubli 2023). However, no compelling evidence supports claims that LLMs are less biased than human annotators (Gilardi, Alizadeh, and Kubli 2023).

LLMs could potentially provide access to otherwise-inaccessible research contexts. If virtual participants’ behavior is sufficiently similar to human participants, LLMs could access underrepresented and hard-to-reach populations, strengthen generalizability and inclusiveness, impute missing data (see LLMs for Synthesis S3: LLMs for Synthesis  ), and enable research that is ethically problematic with real humans (e.g., questions that would force a human to relive past trauma do not harm an LLM).

Studying real-world LLM-based tools allows researchers to understand the state of practice, uncovering usage patterns, adoption rates, and contextual factors, and generate hypotheses about how LLMs affect developer productivity, collaboration, and decision-making. From an engineering perspective, developing LLM-based tools requires less task-specific engineering than traditional SE approaches such as static analysis or symbolic execution, because a single model can handle diverse inputs without language-specific parsers or hand-crafted rules. Good benchmarks provide standardized evaluation and model comparison, reducing research effort and fostering open science by providing common ground for sharing data and results. Benchmarks built for specific SE tasks can help identify LLM weaknesses and support optimization and fine-tuning.

Challenges

The stochastic nature of LLM responses, where identical prompts may yield different outputs, undermines replicability across all study types, complicating interpretation of experimental results and test-retest reliability. More broadly, reliability is a persistent concern: conceptualizing an LLM-as-judge or LLM-as-annotator as a measurement instrument, researchers should expect validity, test-retest reliability, inter-rater reliability, minimal error, and measurement invariance (Ralph et al. 2024). However, LLMs show significant variability depending on the dataset and task (Pangakis, Wolken, and Fasching 2023), are sensitive to prompt variations (Reiss 2023) and option order (Pezeshkpour and Hruschka 2024), can behave differently when reviewing their own outputs (Panickssery, Bowman, and Feng 2024), and can be unreliable for high-stakes labeling (X. Wang et al. 2024). Crucially, reliability does not imply validity: a reliable LLM might be reliably inaccurate (R. Zhou, Chen, and Yu 2024), and for tasks with no single correct answer, the statistical framework pushes outputs toward the most likely answer, which may not be the best one (Bavaresco et al. 2024). See Version and Configuration G2: Version and Configuration  and Limitations and Mitigations G8: Limitations and Mitigations  for reporting guidance on replicability and reliability.

The rapid evolution of LLMs and LLM-based tools, combined with professionals rapidly learning how to employ them, complicates longitudinal comparisons and may quickly render study findings obsolete (e.g., GPT-4’s accuracy on a simple mathematical task dropped from 84% to 51% within three months (L. Chen, Zaharia, and Zou 2024)).

The prevalence of proprietary tools and opaque training data limits researchers’ ability to assess and mitigate biases, and enables benchmark contamination (Ahuja, Gumma, and Sitaram 2024). LLMs exhibit bias in multiple forms: tendencies to overestimate certain labels (Zhu et al. 2023), well-documented fairness issues (Gallegos et al. 2024), and the potential to reinforce prejudices. When simulating human participants, LLMs are “likely to both misportray and flatten the representations of demographic groups” (A. Wang, Morgenstern, and Dickerson 2025). See Open LLMs G6: Open LLMs  and Limitations and Mitigations G8: Limitations and Mitigations  for mitigation strategies.

Evaluation difficulty is inherent in studying LLM-based tools and benchmarks. Benchmarks may lack construct validity (Ralph et al. 2024), usually do not capture the full complexity of software engineering work (Chandra 2025), and may lead to overconfidence and overfitting (Wan et al. 2024). LLMs are also susceptible to adversarial manipulation where semantics-preserving changes may deceive them into accepting flawed artifacts (J. He et al. 2025). See Benchmarks and Metrics G5: Benchmarks and Metrics  for detailed guidance on benchmark design, including Cao et al. (2025)’s (Cao et al. 2025) guidelines for coding task benchmarks.

A fundamental challenge is philosophical and methodological incongruence with qualitative research. In a recent open letter, 419 experienced qualitative researchers argued “that analytic approaches such as reflexive thematic analysis are human research practices requiring a subjective, positioned, and reflexive researcher and therefore the use of GenAI in such approaches is not methodologically congruent” (Jowsey et al. 2025). LLMs lack the capacity for genuinely reflexive qualitative analysis because they operate on statistical prediction without understanding the meaning of the data being analyzed (Jowsey et al. 2025; Bano, Zowghi, and Whittle 2023; Montes et al. 2025). Effective use requires structured prompts and careful human oversight (Barros et al. 2025); see Human Validation G7: Human Validation  and Limitations and Mitigations G8: Limitations and Mitigations  for guidance.

There is insufficient evidence for the effectiveness of LLMs in most research roles. No compelling evidence exists that LLMs can accurately simulate human participants (Schröder et al. 2025) or reliably judge most relevant properties of SE artifacts; gathering such evidence for each specific usage may be quite difficult (Harding et al. 2024). LLMs may be better suited for augmenting rather than replacing human researchers (S. Wang et al. 2021; Z. He et al. 2024), but even then, limited evidence exists that augmentation increases effectiveness as well as efficiency (Schröder et al. 2025). When LLM outputs are incorrect, they can negatively affect human judgment (F. Huang, Kwak, and An 2023; Pan et al. 2024); see Human Validation G7: Human Validation  for mitigation strategies.

Pooling multiple outputs for majority voting improves reliability but increases cost and environmental impact (Reiss 2023; S. Wang et al. 2021). While open models are available, the most capable ones require substantial hardware; relying on cloud-based APIs introduces concerns related to data privacy, security, and replicability. See Limitations and Mitigations G8: Limitations and Mitigations  for sustainability considerations.

Field study findings face generalizability challenges because outcomes may be highly sensitive to individual differences, usage patterns, goals, and contexts. Field studies must be “dependable” (Sullivan and Sargeant 2011) beyond traditional validity criteria, which complicates methodology; see Limitations and Mitigations G8: Limitations and Mitigations  for a detailed threat taxonomy.

2026.05

Advantages and Challenges

Using LLMs in empirical SE research, whether as tools for researchers or for software engineers, offers several potential advantages but also raises fundamental challenges that cut across the study types described above.

Advantages

The primary advantage of using LLMs for research tasks is speed, cost reduction, and scalability. LLMs can annotate, judge, synthesize, and simulate faster and at lower cost than human researchers, with studies showing cost reductions of 50–96% on various natural language tasks (S. Wang et al. 2021) (e.g., Z. He et al. (2024) found that GPT-4 annotation required only two days and 122.08 USD compared to several weeks and 4,508 USD for a comparable MTurk pipeline (Z. He et al. 2024)). Similarly, augmenting or replacing human participants with LLM-generated virtual participants would reduce recruitment effort (Madampe et al. 2024). This efficiency unlocks scalability: qualitative research traditionally does not scale well to large samples, but LLMs allow researchers to analyze larger datasets (Bano et al. 2024; Barros et al. 2025; Morais Leça et al. 2025) and support larger judgment datasets.

LLMs can also automate tasks such as coding qualitative data, assessing artifact quality, and generating summaries, reducing cognitive demands and resources required for qualitative research. Some studies suggest that LLMs may improve consistency: ChatGPT’s accuracy exceeded crowd workers by approximately 25%, and LLMs can achieve higher inter-rater agreement than crowd workers and trained annotators (Gilardi, Alizadeh, and Kubli 2023). However, no compelling evidence supports claims that LLMs are less biased than human annotators (Gilardi, Alizadeh, and Kubli 2023).

LLMs could potentially provide access to otherwise-inaccessible research contexts. If virtual participants’ behavior is sufficiently similar to human participants, LLMs could access underrepresented and hard-to-reach populations, strengthen generalizability and inclusiveness, impute missing data (see LLMs for Synthesis ), and enable research that is ethically problematic with real humans (e.g., questions that would force a human to relive past trauma do not harm an LLM).

Studying real-world LLM-based tools allows researchers to understand the state of practice, uncovering usage patterns, adoption rates, and contextual factors, and generate hypotheses about how LLMs affect developer productivity, collaboration, and decision-making. From an engineering perspective, developing LLM-based tools requires less task-specific engineering than traditional SE approaches such as static analysis or symbolic execution, because a single model can handle diverse inputs without language-specific parsers or hand-crafted rules. Good benchmarks provide standardized evaluation and model comparison, reducing research effort and supporting open science by providing common ground for sharing data and results. Benchmarks built for specific SE tasks can help identify LLM weaknesses and support optimization and fine-tuning.

Challenges

The stochastic nature of LLM responses, where identical prompts may yield different outputs, undermines replicability across all study types, complicating interpretation of experimental results and test-retest reliability. More broadly, reliability is a persistent concern: conceptualizing an LLM-as-judge or LLM-as-annotator as a measurement instrument, researchers should expect validity, test-retest reliability, inter-rater reliability, minimal error, and measurement invariance (Ralph et al. 2024). However, LLMs show substantial variability depending on the dataset and task (Pangakis, Wolken, and Fasching 2023), are sensitive to prompt variations (Reiss 2023) and option order (Pezeshkpour and Hruschka 2024), can behave differently when reviewing their own outputs (Panickssery, Bowman, and Feng 2024), and can be unreliable for high-stakes labeling (X. Wang et al. 2024). Crucially, reliability does not imply validity: a reliable LLM might be reliably inaccurate (R. Zhou, Chen, and Yu 2024), and for tasks with no single correct answer, the statistical framework pushes outputs toward the most likely answer, which may not be the best one (Bavaresco et al. 2024). See Version and Configuration and Limitations and Mitigations for reporting guidance on replicability and reliability.

The rapid evolution of LLMs and LLM-based tools, combined with professionals rapidly learning how to employ them, complicates longitudinal comparisons and may quickly render study findings obsolete (e.g., GPT-4’s accuracy on a simple mathematical task dropped from 84% to 51% within three months (L. Chen, Zaharia, and Zou 2024)). Findings tied to a specific model version may not transfer to later releases even within the same model family, complicating reproducibility and longitudinal comparisons.

The prevalence of proprietary tools and opaque training data limits researchers’ ability to assess and mitigate biases, and enables benchmark contamination (Ahuja, Gumma, and Sitaram 2024). LLMs exhibit bias in multiple forms: tendencies to overestimate certain labels (Zhu et al. 2023), well-documented fairness issues (Gallegos et al. 2024), and the potential to reinforce prejudices. When simulating human participants, LLMs are “likely to both misportray and flatten the representations of demographic groups” (A. Wang, Morgenstern, and Dickerson 2025). See Open LLMs and Limitations and Mitigations for mitigation strategies.

Evaluation difficulty is inherent in studying LLM-based tools and benchmarks. Benchmarks may lack construct validity (Ralph et al. 2024), usually do not capture the full complexity of software engineering work (Chandra 2025), and may lead to overconfidence and overfitting (Wan et al. 2024). LLMs are also susceptible to adversarial manipulation where semantics-preserving changes may deceive them into accepting flawed artifacts (Junda He et al. 2026). See Benchmarks and Metrics for detailed guidance on benchmark design, including Cao et al. (2026)’s (Cao et al. 2026) guidelines for coding task benchmarks.

A fundamental challenge is philosophical and methodological incongruence with qualitative research. In a recent open letter, 419 experienced qualitative researchers argued “that analytic approaches such as reflexive thematic analysis are human research practices requiring a subjective, positioned, and reflexive researcher and therefore the use of GenAI in such approaches is not methodologically congruent” (Jowsey et al. 2025). LLMs lack the capacity for genuinely reflexive qualitative analysis because they operate on statistical prediction without understanding the meaning of the data being analyzed (Bano, Zowghi, and Whittle 2023; Montes et al. 2025). Effective use requires structured prompts and careful human oversight (Barros et al. 2025); see Human Validation and Limitations and Mitigations for guidance.

There is insufficient evidence for the effectiveness of LLMs in most research roles. No compelling evidence exists that LLMs can accurately simulate human participants (Schröder et al. 2025) or reliably judge most relevant properties of SE artifacts; gathering such evidence for each specific usage may be quite difficult (Harding et al. 2024). LLMs may be better suited for augmenting rather than replacing human researchers (S. Wang et al. 2021; Z. He et al. 2024), but even then, limited evidence exists that augmentation increases effectiveness as well as efficiency (Schröder et al. 2025). When LLM outputs are incorrect, they can negatively affect human judgment (F. Huang, Kwak, and An 2023; Pan et al. 2024); see Human Validation for mitigation strategies.

Majority voting across multiple outputs improves reliability but increases cost and environmental impact (Reiss 2023; S. Wang et al. 2021). While open models are available, the most capable ones require substantial hardware; relying on cloud-based APIs introduces concerns related to data privacy, security, and replicability. See Limitations and Mitigations for sustainability considerations.

Field study findings face generalizability challenges because outcomes may be highly sensitive to individual differences, usage patterns, goals, and contexts. Field studies must be “dependable” (Sullivan and Sargeant 2011) beyond traditional validity criteria, which complicates methodology; see Limitations and Mitigations for a detailed threat taxonomy.

Guidelines Changed

2026.04

Guidelines

A primary goal of our guidelines is to enable reproducibility and replicability of empirical SE studies involving LLMs. As repeating LLM-focused research to verify results lies somewhere between the ACM’s definitions of reproducibility (different team, same research artifacts) and repeatability (different team, different research artifacts) (ACM Publications Board 2020) due to potential model changes and imperfect research artifacts, we follow Angermeir et al. (2025) (Angermeir et al. 2025) and use the terms interchangeably. While previous guidelines regarding open science and empirical studies still apply, LLM-specific characteristics (e.g. inherent non-determinism (Song et al. 2025; Yuan et al. 2025), opaque and proprietary models) present additional replicability challenges, which, in turn, demand new guidance.

Each guideline below begins with a brief tl;dr summary, followed by its rationale, recommendations, examples, benefits, challenges, links to the relevant study types. The rationale articulates the underlying principle, i.e., why the guideline matters, while the recommendations provide concrete, actionable practices.

The guidelines further contain advice for reviewers. Broadly, reviewers should use our guidelines to help them interrogate the extent to which a manuscript’s authors have done what was practically possible to improve reproducibility. We must neither accept research absent reasonable efforts to improve reproducibility, nor reject research for failing to obtain an impossible goal. We borrow this principle of “reasonable efforts” from the SIGSOFT Empirical Standards (Ralph et al. 2021), where it applies to methodological rigor more generally. Of course, papers should acknowledge their limitations, but to determine whether these limitations are reasonable, reviewers should ask “have the authors done what they could to minimize limitations?” Our guidelines attempt to capture what “reasonable efforts” practically means for LLM-based studies.

To distinguish essential criteria from recommendations, our guidelines use two tiers. A MUST criterion is a requirement. Studies that intend to follow these guidelines are expected to meet all MUST criteria. A SHOULD criterion represents a desired practice that strengthens a study’s rigor or transparency. However, there may be valid reasons to deviate in particular circumstances (e.g., resource constraints, inapplicability to a specific study context or type). Nonetheless, studies that deviate from a SHOULD criterion should briefly justify the deviation and discuss its potential impact (e.g., on validity or reproducibility).

The following sections indicate which information we expect researchers to report, and whether it should be in the PAPER or SUPPLEMENTARY MATERIAL. Where a publication venue’s page limits hinder reporting all expected elements in the PAPER, it is better to report essential information in the SUPPLEMENTARY MATERIAL than not at all. The SUPPLEMENTARY MATERIAL should be published according to the ACM SIGSOFT Open Science Policies (Graziotin 2024).

Applicability of guidelines to study types. ●  = the guideline’s core recommendations MUST be followed for this study type, ○  = SHOULD be followed, – = are not directly applicable. Study type IDs: S1 =  LLMs as Annotators S1: LLMs as Annotators , S2 =  LLMs as Judges S2: LLMs as Judges , S3 =  LLMs for Synthesis S3: LLMs for Synthesis , S4 =  LLMs as Subjects S4: LLMs as Subjects , S5 =  Studying LLM Usage S5: Studying LLM Usage , S6 =  LLMs for Tools S6: LLMs for Tools , S7 =  Benchmarking LLMs S7: Benchmarking LLMs . Each guideline’s study-type-specific guidance is detailed in the corresponding subsection.
G1: Declare Usage ●  ●  ●  ●  ●  ●  ● 
G2: Model Version ●  ●  ●  ●  ●  ●  ● 
G3: Design ●  ●  ●  ●  ○  ●  ● 
G4: Traces ○  ○  ○  ○  ●  ○  ○ 
G5: Benchmarks ●  ○  ○  ○  ○  ○  ● 
G6: Open LLM ○  ○  ○  – – ○  ● 
G7: Human Valid. ○  ○  ○  ○  ○  ○  ○ 
G8: Limitations ●  ●  ●  ●  ●  ●  ● 

2026.05

Guidelines

A primary goal of our guidelines is to enable reproducibility and replicability of empirical SE studies involving LLMs. As repeating LLM-focused research to verify results lies somewhere between the ACM’s definitions of reproducibility (different team, same research artifacts) and repeatability (different team, different research artifacts) (ACM Publications Board 2020) due to potential model changes and imperfect research artifacts, we follow Angermeir et al. (2025) (Angermeir et al. 2025) and use the terms interchangeably. While previous guidelines regarding open science and empirical studies still apply, LLM-specific characteristics (e.g. inherent non-determinism (Song et al. 2025; Yuan et al. 2025), opaque and proprietary models) present additional replicability challenges, which, in turn, demand new guidance.

Each guideline below begins with a brief summary, followed by its rationale, recommendations, examples, benefits, challenges, links to the relevant study types. The rationale articulates the underlying principle, i.e., why the guideline matters, while the recommendations provide concrete, actionable practices.

The guidelines further contain advice for reviewers. Broadly, reviewers should use our guidelines to help them interrogate the extent to which a manuscript’s authors have done what was practically possible to improve reproducibility. We must neither accept research absent reasonable efforts to improve reproducibility, nor reject research for failing to obtain an impossible goal. We borrow this principle of “reasonable efforts” from the SIGSOFT Empirical Standards (Ralph et al. 2021), where it applies to methodological rigor more generally. Of course, papers should acknowledge their limitations, but to determine whether these limitations are reasonable, reviewers should ask “have the authors done what they could to minimize limitations?” Our guidelines attempt to capture what “reasonable efforts” practically means for LLM-based studies.

To distinguish essential criteria from recommendations, our guidelines use two tiers. A must criterion is a requirement. Studies that intend to follow these guidelines are expected to meet all must criteria. A should criterion represents a desired practice that strengthens a study’s rigor or transparency. However, there may be valid reasons to deviate in particular circumstances (e.g., resource constraints, inapplicability to a specific study context or type). Nonetheless, studies that deviate from a should criterion should briefly justify the deviation and discuss its potential impact (e.g., on validity or reproducibility).

The following sections indicate which information we expect researchers to report, and whether it should be in the paper or supplementary material. Where a publication venue’s page limits hinder reporting all expected elements in the paper, it is better to report essential information in the supplementary material than not at all. The supplementary material should be published according to the ACM SIGSOFT Open Science Policies (Graziotin 2024).

Declare LLM Usage and Role Changed

2026.04

Declare LLM Usage and Role (G1)

tl;dr: Researchers MUST disclose any use of LLMs to support empirical studies in their PAPER, specifying which LLM was used, how it was used, and where in the research process it was employed. They SHOULD report the exact purpose, the tasks that were automated, and the expected benefits in the PAPER. When the LLM plays a central role, the declaration SHOULD be prominent and detailed in the methodology section; for tangential uses, a brief statement in the methodology or acknowledgments suffices.

Unchanged text

Rationale

Transparency about LLM involvement is a prerequisite for informed assessment of a study’s scope, limitations, and potential biases. Without explicit disclosure, readers cannot evaluate how the LLM’s characteristics may have influenced the research process or its outcomes.

Recommendations

When conducting any kind of empirical study involving LLMs, researchers MUST clearly declare that an LLM was used (see Scope for what we consider relevant research support). This SHOULD be done in a suitable section of the PAPER, for example, in the introduction or research methods section. For authoring scientific articles, this transparency is, for example, required by the ACM Policy on Authorship: “The use of generative AI tools and technologies to create content is permitted but must be fully disclosed in the Work” (Association for Computing Machinery 2023).

Beyond generic declarations, researchers SHOULD report the exact purpose of using an LLM in a study, the tasks it was used to automate, and the expected benefits in the PAPER. A sufficient declaration specifies not only that an LLM was used, but also which LLM (name and version), how it was used (e.g., as an annotator, code generator, or judge), and where in the research process it was employed (e.g., data collection, analysis, or synthesis).

When the LLM plays a central role in the study (e.g., as the main tool being evaluated or as a core component of the research method), the declaration SHOULD be prominent and detailed, appearing in the methodology section with cross-references to the specific guidelines that apply (e.g., Sections Version and Configuration G2: Version and Configuration , System and Prompt Design G3: System and Prompt Design , and Session Traces G4: Session Traces  ). When the LLM’s role is more tangential (e.g., used for a single preprocessing step), a brief but explicit statement in the methodology or acknowledgments section is sufficient. In either case, the disclosure MUST be specific enough for readers to assess how the LLM’s involvement may affect the study’s validity and reproducibility.

Example(s)

The ACM Policy on Authorship (Association for Computing Machinery 2023) suggests disclosing GenAI usage in the acknowledgments section of the PAPER, advising to “err on the side of caution, and include a disclosure in the acknowledgments section of the Work” when uncertain about the need (Association for Computing Machinery 2023). For double-blind review, researchers can add a temporary “AI Disclosure” section where the acknowledgments would appear. An example of an LLM disclosure beyond writing support can be found in a recent paper by Lubos et al. (2024) (Lubos et al. 2024), in which they write in the methodology section:

“We conducted an LLM-based evaluation of requirements utilizing the Llama 2 language model with 70 billion parameters, fine-tuned to complete chat responses...”

A more contemporary declaration could similarly state:

“We used Claude Opus 4.7 via the Anthropic API to synthesize themes from interview transcripts, with all prompts and conversation logs published as SUPPLEMENTARY MATERIAL.”

Benefits

Transparency in the use of LLMs helps other researchers understand the context and scope of the study, facilitating better interpretation and comparison of the results. Beyond this declaration, we recommend researchers to be explicit about the LLM version they used (see Version and Configuration G2: Version and Configuration  ) and the LLM’s exact role.

Challenges

Declaring LLM usage requires only a brief statement and no additional experiments, making compliance straightforward. One challenge might be authors’ reluctance to disclose LLM usage for valid use cases, because they fear that AI-generated content makes reviewers think that the authors’ work is less original. In fact, there is evidence suggesting that AI disclosure can negatively affect trust in authors (Schilke and Reimann 2025). However, the ACM Policy on Authorship is very clear in that any use of GenAI tools to create content MUST be disclosed. Our guidelines focus on research support beyond proof-reading and writing support (see Scope ), but the threshold of what must be declared continues to evolve as organizations such as the ACM update their authorship policies.

Study Types

Researchers MUST follow this guideline for all study types. The specific focus of the declaration varies by study type. For LLMs as Annotators S1: LLMs as Annotators , LLMs as Judges S2: LLMs as Judges , LLMs for Synthesis S3: LLMs for Synthesis , and LLMs as Subjects S4: LLMs as Subjects , researchers MUST declare the specific role assigned to the LLM (e.g., annotator, judge, synthesizer, or simulated participant). For Studying LLM Usage S5: Studying LLM Usage , researchers MUST clarify which LLM(s) the observed participants used and under which conditions. For LLMs for Tools S6: LLMs for Tools , researchers MUST declare the LLM’s role within the tool architecture and its contribution to the tool’s functionality. For Benchmarking LLMs S7: Benchmarking LLMs , researchers MUST declare which LLMs were benchmarked and for which tasks.

Unchanged text

Advice for Reviewers

The most common problem with disclosure is incompleteness or vagueness about how the LLM was used. If the paper says “we used LLM X to help with task Y” without specifying how, reviewers should request clarification. Such requests are typically minor revisions unless the missing details may reveal methodological problems.

Using an LLM as a copyeditor becomes problematic when authors do not or cannot take responsibility for the resulting text. If a reviewer finds clear evidence that LLM-generated text was not carefully reviewed (e.g., text like “As a Large Language Model, I...”), this may warrant rejection. If a reviewer suspects an author cannot take responsibility for AI-generated text, they should raise the concern with their editor or program chair. Due process requires that authors not be accused of misconduct without clear evidence.

If undisclosed LLM use is suspected, the reviewer should similarly consult their editor or program chair. When the evidence is conclusive, the key question is the degree to which undisclosed use affects the study’s contribution, ranging from negligible (e.g., word choice in a single sentence) to severe (e.g., generating ostensibly empirical data or statistical analyses).

See Also

2026.05

Declare LLM Usage and Role

Summary: Researchers must disclose any use of LLMs to support empirical studies in their paper, specifying which LLM was used, how it was used, and where in the research process it was employed. They should report the exact purpose, the tasks that were automated, and the expected benefits in the paper. When the LLM is central to the study, the declaration should be prominent and detailed in the methodology section; for tangential uses, a brief statement in the methodology or acknowledgments suffices.

Unchanged text

Rationale

Transparency about LLM involvement is a prerequisite for informed assessment of a study’s scope, limitations, and potential biases. Without explicit disclosure, readers cannot evaluate how the LLM’s characteristics may have influenced the research process or its outcomes.

Recommendations

When conducting any kind of empirical study involving LLMs, researchers must clearly declare that an LLM was used (see Scope for what we consider relevant research support). This should be done in a suitable section of the paper, for example, in the introduction or research methods section; Cheng, Calhoun, and Reedy (2025) (Cheng, Calhoun, and Reedy 2025) argue specifically for the methods section, since acknowledgments are easily missed at the end of a paper. For authoring scientific articles, this transparency is, for example, required by the ACM Policy on Authorship: “The use of generative AI tools and technologies to create content is permitted but must be fully disclosed in the Work” (Association for Computing Machinery 2023).

Beyond generic declarations, researchers should report the exact purpose of using an LLM in a study, the tasks it was used to automate, and the expected benefits in the paper. A sufficient declaration specifies not only that an LLM was used, but also which LLM (name and version), how it was used (e.g., as an annotator, code generator, or judge), and where in the research process it was employed (e.g., data collection, analysis, or synthesis).

When the LLM is central to the study (e.g., as the main tool being evaluated or as a core component of the research method), the declaration should be prominent and detailed, appearing in the methodology section with cross-references to the specific guidelines that apply (e.g., Sections Version and Configuration, System and Prompt Design, and Session Traces ). When the LLM’s role is more tangential (e.g., used for a single preprocessing step), a brief but explicit statement in the methodology or acknowledgments section is sufficient. In either case, the disclosure must be specific enough for readers to assess how the LLM’s involvement may affect the study’s validity and reproducibility.

Examples

The ACM Policy on Authorship (Association for Computing Machinery 2023) suggests disclosing GenAI usage in the acknowledgments section of the paper, advising to “err on the side of caution, and include a disclosure in the acknowledgments section of the Work” when uncertain about the need (Association for Computing Machinery 2023). For double-blind review, researchers can add a temporary “AI Disclosure” section where the acknowledgments would appear. An example of an LLM disclosure beyond writing support can be found in a recent paper by Lubos et al. (2024) (Lubos et al. 2024), in which they write in the methodology section:

“We conducted an LLM-based evaluation of requirements utilizing the Llama 2 language model with 70 billion parameters, fine-tuned to complete chat responses...”

A more contemporary declaration could similarly state:

“We used Claude Opus 4.7 via the Anthropic API to synthesize themes from interview transcripts, with all prompts and conversation logs published as supplementary material.”

Benefits

Transparency in the use of LLMs helps other researchers understand the context and scope of the study, supporting better interpretation and comparison of the results. Beyond this declaration, we recommend researchers to be explicit about the LLM version they used (see Version and Configuration ) and the LLM’s exact role.

Challenges

Declaring LLM usage requires only a brief statement and no additional experiments, making compliance straightforward. One challenge might be authors’ reluctance to disclose LLM usage for valid use cases, because they fear that AI-generated content makes reviewers think that the authors’ work is less original. In fact, there is evidence suggesting that AI disclosure can negatively affect trust in authors (Schilke and Reimann 2025). However, the ACM Policy on Authorship is very clear in that any use of GenAI tools to create content must be disclosed. Our guidelines focus on research support beyond proof-reading and writing support (see Scope ), but the threshold of what must be declared continues to evolve as organizations such as the ACM update their authorship policies.

Study Types

Researchers must follow this guideline for all study types. The specific focus of the declaration varies by study type. For LLMs as Annotators, LLMs as Judges, LLMs for Synthesis, and LLMs as Subjects, researchers must declare the specific role assigned to the LLM (e.g., annotator, judge, synthesizer, or simulated participant). For Studying LLM Usage, researchers must clarify which LLM(s) the observed participants used and under which conditions. For LLMs for Tools, researchers must declare the LLM’s role within the tool architecture and its contribution to the tool’s functionality. For Benchmarking LLMs, researchers must declare which LLMs were benchmarked and for which tasks.

Unchanged text

Advice for Reviewers

The most common problem with disclosure is incompleteness or vagueness about how the LLM was used. If the paper says “we used LLM X to help with task Y” without specifying how, reviewers should request clarification. Such requests are typically minor revisions unless the missing details may reveal methodological problems.

Using an LLM as a copyeditor becomes problematic when authors do not or cannot take responsibility for the resulting text. If a reviewer finds clear evidence that LLM-generated text was not carefully reviewed (e.g., text like “As a Large Language Model, I...”), this may warrant rejection. If a reviewer suspects an author cannot take responsibility for AI-generated text, they should raise the concern with their editor or program chair. Due process requires that authors not be accused of misconduct without clear evidence.

If undisclosed LLM use is suspected, the reviewer should similarly consult their editor or program chair. When the evidence is conclusive, the key question is the degree to which undisclosed use affects the study’s contribution, ranging from negligible (e.g., word choice in a single sentence) to severe (e.g., generating ostensibly empirical data or statistical analyses).

See Also

Report Model Version, Configuration, and Customizations Changed

2026.04

Report Model Version, Configuration, and Customizations (G2)

tl;dr: Researchers MUST report the exact LLM model or tool version, configuration, and experiment date in the PAPER. When using quantized models, researchers SHOULD report the quantization level and method. For fine-tuned models, they MUST describe the fine-tuning goal, dataset, and procedure in the PAPER. Researchers SHOULD include default parameters, explain model choices, compare base- and fine-tuned model using suitable metrics and benchmarks, and share fine-tuning data and weights as SUPPLEMENTARY MATERIAL (or alternatively justify in the PAPER why they cannot share them).

Rationale

LLMs and LLM-based tools are frequently updated, and configuration parameters such as temperature or seed values affect content generation. This guideline focuses on documenting the model-specific aspects of empirical studies involving LLMs, concentrating on the models themselves, their version, configuration parameters, and customizations (e.g., fine-tuning). While the System and Prompt Design G3: System and Prompt Design  section addresses system-level integration and the authored artifacts (including prompts) that shape each call, the information outlined here is essential for reproducibility whenever an LLM is involved.

Recommendations

Researchers MUST document in the PAPER which model or tool version they used in their study, along with the date when the experiments were carried out and the configured parameters that affect output generation. Since default values might change over time, researchers SHOULD report all configuration values, even if they used the defaults. Checksums and fingerprints SHOULD be reported since they identify specific versions and configurations. Depending on the study context, other properties such as the context window size (number of tokens) SHOULD be reported. When using quantized models, researchers SHOULD report the quantization level (e.g., 4-bit, 8-bit) and method (e.g., GPTQ or AWQ), as different quantization approaches produce different outputs, affecting both output quality and reproducibility. Researchers SHOULD motivate in the PAPER why they selected certain models, versions, and configurations. Reasons may be monetary, technical, or methodological (e.g., planned comparison to previous work). Depending on the specific study context, additional information regarding the experiment or tool architecture SHOULD be reported.

A common customization approach for existing LLMs is fine-tuning. If a model was fine-tuned, researchers MUST describe the fine-tuning goal (e.g., improving the performance for a specific task), the fine-tuning procedure (e.g., full fine-tuning vs. Low-Rank Adaptation (LoRA), selected hyperparameters, loss function, learning rate, batch size, etc.), and the fine-tuning dataset (e.g., data sources, the preprocessing pipeline, dataset size) in the PAPER. Researchers SHOULD either share the fine-tuning dataset as part of the SUPPLEMENTARY MATERIAL or explain in the PAPER why the data cannot be shared (e.g., because it contains confidential or personal data that could not be anonymized). The same applies to the fine-tuned model weights. Suitable benchmarks and metrics SHOULD be used to compare the base model with the fine-tuned model.

In summary, our recommendation is to report:

  1. Model/tool name and version (MUST in PAPER );
  2. All relevant configured parameters that affect output generation (MUST in PAPER );
  3. Default values of all available parameters (SHOULD );
  4. Checksum/fingerprint of used model version and configuration (SHOULD );
  5. Additional properties such as context window size (SHOULD );
  6. Model quantization level and method, if applicable (SHOULD ).

For fine-tuned models, additional recommendations apply:

  1. Fine-tuning goal (MUST in PAPER );
  2. Fine-tuning dataset creation and characterization (MUST in PAPER );
  3. Fine-tuning parameters and procedure (MUST in PAPER );
  4. Fine-tuning dataset and fine-tuned model weights (SHOULD );
  5. Validation metrics and benchmarks (SHOULD ).

Commercial models (e.g., GPT-5) or LLM-based tools (e.g., ChatGPT) might not give researchers access to all required information. For these tools, researchers SHOULD report what is available and openly acknowledge limitations that hinder reproducibility.

Example(s)

Based on the documentation that OpenAI and Azure provide (OpenAI 2025; Microsoft 2025), researchers might, for example, report:

“We integrated a gpt-4 model in version 0125-Preview via the Azure OpenAI Service, and configured it with a temperature of 0.7, top_p set to 0.8, a maximum token length of 512, and the seed value 23487. We ran our experiment on 10th January 2025. The system fingerprint was fp_6b68a8204b.

Kang, Yoon, and Yoo (2023) provide a similar statement in their paper on exploring LLM-based bug reproduction (Kang, Yoon, and Yoo 2023):

“We access OpenAI Codex via its closed beta API, using the code-davinci-002 model. For Codex, we set the temperature to 0.7, and the maximum number of tokens to 256.”

Our guidelines additionally recommend reporting a checksum/fingerprint and exact dates; otherwise, this example is close to our recommendations.

Dhar, Vaidhyanathan, and Varma (2024) (Dhar, Vaidhyanathan, and Varma 2024) assessed whether LLMs can generate architectural design decisions, detailing the system architecture and the LLM’s role within it. They provide information on the fine-tuning approach and datasets, including the source of architectural decision records, preprocessing methods, and data selection criteria.

For self-hosted models, the SUPPLEMENTARY MATERIAL can become a true replication package. For example, for models provisioned using ollama, one can report the specific tag and checksum, e.g., “llama3.3, tag 70b-instruct-q8_0, checksum d5b5e1b84868.” Given suitable hardware, running the model is then as easy as executing the following command:
ollama run llama3.3:70b-instruct-q8_0

Unchanged text

Benefits

Reporting this information is a prerequisite for the verification, reproduction, and replication of LLM-based studies. While LLMs are inherently non-deterministic, this cannot excuse dismissing reproducibility. Although exact reproducibility is hard to achieve, reporting the information outlined in this guideline helps researchers come as close as possible to that standard.

Challenges

Different model providers and modes of operating the models allow for varying degrees of information. For example, OpenAI provides a model version and a system fingerprint describing the backend configuration, which can also influence the output. However, the fingerprint is intended only to detect changes in the model or its configuration; one cannot go back to a certain fingerprint. As a beta feature, OpenAI lets users set a seed parameter to receive “(mostly) consistent output” (OpenAI 2023). However, the seed value does not allow for full reproducibility and the fingerprint changes frequently. Although, as motivated above, open models significantly simplify re-running experiments, they also come with challenges in terms of reproducibility, as generated outputs can be inconsistent despite setting the temperature to 0 and using a seed value (see GitHub issue for Llama3). Setting the temperature to 0 configures greedy decoding (always selecting the most probable next token), which minimizes output variability but can degrade quality by producing repetitive text and missing higher-quality responses (Holtzman et al. 2020).

Even with a temperature of 0, full determinism is rarely guaranteed: floating-point arithmetic on GPUs causes slight numerical differences that cascade into divergent token selections (Yuan et al. 2025), Sparse Mixture-of-Experts routing amplifies this effect (Chann 2023), silent backend changes in commercial APIs produce different outputs over time (L. Chen, Zaharia, and Zou 2024), and even self-hosted open models with identical settings do not always yield consistent outputs (Angermeir et al. 2025; Song et al. 2025). Researchers SHOULD NOT treat a temperature of 0 as a guarantee of reproducibility, but as one measure among several, including fixed seed values (OpenAI 2023), system fingerprints, and archiving of raw outputs. When a temperature of 0 is chosen primarily for reproducibility, this motivation SHOULD be stated explicitly, along with an acknowledgment of its potential impact on output quality.

Study Types

This guideline MUST be followed for all study types for which the researcher has access to (parts of) the model’s configuration. They MUST always report the configuration that is visible to them, acknowledging the reproducibility challenges of commercial tools and models that are offered as-a-service. Depending on the specific study type, researchers SHOULD provide additional information on the system and prompt design (see System and Prompt Design G3: System and Prompt Design  ), session traces (see Session Traces G4: Session Traces  ), and specific limitations and mitigations (see Limitations and Mitigations G8: Limitations and Mitigations  ).

For example, when Studying LLM Usage S5: Studying LLM Usage  by focusing on commercial tools such as ChatGPT or GitHub Copilot, researchers MUST be as specific as possible in describing their study setup. The configured model name, version, and the date when the experiment was conducted MUST always be reported. See System and Prompt Design G3: System and Prompt Design  for prompt reporting and Session Traces G4: Session Traces  for interaction logs.

For LLMs as Annotators S1: LLMs as Annotators , LLMs as Judges S2: LLMs as Judges , and LLMs for Synthesis S3: LLMs for Synthesis , researchers MUST report the model configuration used for the respective annotation, judging, or synthesis tasks, including temperature and other sampling parameters that affect output variability. For LLMs as Subjects S4: LLMs as Subjects , researchers MUST report any persona-related configuration settings and parameters that shape the simulated behavior. For LLMs for Tools S6: LLMs for Tools , researchers MUST report the configuration for each model integrated in the tool’s architecture, including any model-specific parameter choices. For Benchmarking LLMs S7: Benchmarking LLMs , researchers MUST report the configuration for all benchmarked models to enable fair cross-model comparisons.

Unchanged text

Advice for Reviewers

Missing version, configuration, or parameter information is typically a minor revision request. Before concluding that information is absent, reviewers should check appendices and supplementary materials, as details are sometimes reported there rather than in the main text. Rejection over missing details is rarely warranted unless the omissions obscure deeper methodological problems.

See Also

2026.05

Report Model Version, Configuration, and Customizations

Summary: Researchers must report the exact LLM model or tool version, configuration, and experiment date in the paper. When using quantized models, researchers should report the quantization level and method. For fine-tuned models, they must describe the fine-tuning goal, dataset, and procedure in the paper. Researchers should include default parameters, explain model choices, compare base- and fine-tuned model using suitable metrics and benchmarks, and share fine-tuning data and weights as supplementary material (or alternatively justify in the paper why they cannot share them).

Rationale

LLMs and LLM-based tools are frequently updated, and configuration parameters such as temperature or seed values affect content generation. This guideline focuses on documenting the model-specific aspects of empirical studies involving LLMs, concentrating on the models themselves, their version, configuration parameters, and customizations (e.g., fine-tuning). While the System and Prompt Design section addresses system-level integration and the authored artifacts (including prompts) that the model uses on each call, the information outlined here is essential for reproducibility whenever an LLM is involved.

Recommendations

Researchers must document in the paper which model or tool version they used in their study, along with the date when the experiments were carried out and the configured parameters that affect output generation. Since default values might change over time, researchers should report all configuration values, even if they used the defaults. Checksums and fingerprints should be reported since they identify specific versions and configurations. Depending on the study context, other properties such as the context window size (number of tokens) should be reported. When using quantized models, researchers should report the quantization level (e.g., 4-bit, 8-bit) and method (e.g., GPTQ or AWQ), as different quantization approaches produce different outputs, affecting both output quality and reproducibility. Researchers should motivate in the paper why they selected certain models, versions, and configurations. Reasons may be monetary, technical, or methodological (e.g., planned comparison to previous work). Depending on the specific study context, additional information regarding the experiment or tool architecture should be reported.

A common customization approach for existing LLMs is fine-tuning. If a model was fine-tuned, researchers must describe the fine-tuning goal (e.g., improving the performance for a specific task), the fine-tuning procedure (e.g., full fine-tuning versus Low-Rank Adaptation (LoRA), selected hyperparameters, loss function, learning rate, and batch size), and the fine-tuning dataset (e.g., data sources, the preprocessing pipeline, dataset size) in the paper. Researchers should either share the fine-tuning dataset as part of the supplementary material or explain in the paper why the data cannot be shared (e.g., because it contains confidential or personal data that could not be anonymized). The same applies to the fine-tuned model weights. Suitable benchmarks and metrics should be used to compare the base model with the fine-tuned model.

In summary, researchers must report in the paper at minimum (1) the exact model or tool name and version, (2) all configured parameters that affect output generation, (3) the experiment date, and, for fine-tuned models, (4) the fine-tuning goal, dataset characterization, and parameters and procedure. Beyond these requirements, researchers should additionally report default parameter values, checksums or fingerprints, model properties relevant to the study (e.g., context-window size), and quantization level and method where applicable; for fine-tuned models, they should share the dataset and model weights as supplementary material (unless legal or privacy constraints prevent disclosure) and report validation metrics and benchmarks.

Commercial models (e.g., GPT-5) or LLM-based tools (e.g., ChatGPT) might not give researchers access to all required information. For these tools, researchers should report what is available and openly acknowledge limitations that hinder reproducibility.

Examples

Based on the documentation that OpenAI and Azure provide (OpenAI 2025; Microsoft 2025), researchers might, for example, report:

“We integrated a gpt-4 model in version 0125-Preview via the Azure OpenAI Service, and configured it with a temperature of 0.7, top_p set to 0.8, a maximum token length of 512, and the seed value 23487. We ran our experiment on 10th January 2025. The system fingerprint was fp_6b68a8204b.

Kang, Yoon, and Yoo (2023) provide a similar statement in their paper on exploring LLM-based bug reproduction (Kang, Yoon, and Yoo 2023):

“We access OpenAI Codex via its closed beta API, using the code-davinci-002 model. For Codex, we set the temperature to 0.7, and the maximum number of tokens to 256.”

Our guidelines additionally recommend reporting a checksum/fingerprint and exact dates; otherwise, this example is close to our recommendations.

Dhar, Vaidhyanathan, and Varma (2024) (Dhar, Vaidhyanathan, and Varma 2024) assessed whether LLMs can generate architectural design decisions, detailing the system architecture and the LLM’s role within it. They provide information on the fine-tuning approach and datasets, including the source of architectural decision records, preprocessing methods, and data selection criteria.

For self-hosted models, the supplementary material can become a true replication package. For example, for models provisioned using ollama, one can report the specific tag and checksum, e.g., “llama3.3, tag 70b-instruct-q8_0, checksum d5b5e1b84868.” Given suitable hardware, running the model is then as easy as executing the following command:
ollama run llama3.3:70b-instruct-q8_0

Unchanged text

Benefits

Reporting this information is a prerequisite for the verification, reproduction, and replication of LLM-based studies. While LLMs are inherently non-deterministic, this cannot excuse dismissing reproducibility. Although exact reproducibility is hard to achieve, reporting the information outlined in this guideline helps researchers come as close as possible to that standard.

Challenges

Different model providers and modes of operating the models allow for varying degrees of information. For example, OpenAI provides a model version and a system fingerprint describing the backend configuration, which can also influence the output. However, the fingerprint is intended only to detect changes in the model or its configuration; one cannot go back to a certain fingerprint. As a beta feature, OpenAI lets users set a seed parameter to receive “(mostly) consistent output” (OpenAI 2023). However, the seed value does not allow for full reproducibility and the fingerprint changes frequently. Although, as motivated above, open models substantially simplify re-running experiments, they also come with challenges in terms of reproducibility, as generated outputs can be inconsistent despite setting the temperature to 0 and using a seed value (see GitHub issue for Llama3). Setting the temperature to 0 configures greedy decoding (always selecting the most probable next token), which minimizes output variability but can degrade quality by producing repetitive text and missing higher-quality responses (Holtzman et al. 2020).

Even with a temperature of 0, full determinism is rarely guaranteed: floating-point arithmetic on GPUs causes slight numerical differences that cascade into divergent token selections (Yuan et al. 2025), Sparse Mixture-of-Experts routing amplifies this effect (Chann 2023), silent backend changes in commercial APIs produce different outputs over time (L. Chen, Zaharia, and Zou 2024), and even self-hosted open models with identical settings do not always yield consistent outputs (Angermeir et al. 2025; Song et al. 2025). Researchers should not treat a temperature of 0 as a guarantee of reproducibility, but as one measure among several, including fixed seed values (OpenAI 2023), system fingerprints, and archiving of raw outputs. When a temperature of 0 is chosen primarily for reproducibility, this motivation should be stated explicitly, along with an acknowledgment of its potential impact on output quality.

Study Types

This guideline must be followed for all study types for which the researcher has access to (parts of) the model’s configuration. They must always report the configuration that is visible to them, acknowledging the reproducibility challenges of commercial tools and models that are offered as-a-service. Depending on the specific study type, researchers should provide additional information on the system and prompt design (see System and Prompt Design ), session traces (see Session Traces ), and specific limitations and mitigations (see Limitations and Mitigations ).

For example, when Studying LLM Usage by focusing on commercial tools such as ChatGPT or GitHub Copilot, researchers must be as specific as possible in describing their study setup. The configured model name, version, and the date when the experiment was conducted must always be reported. See System and Prompt Design for prompt reporting and Session Traces for interaction logs.

For LLMs as Annotators, LLMs as Judges, and LLMs for Synthesis, researchers must report the model configuration used for the respective annotation, judging, or synthesis tasks, including temperature and other sampling parameters that affect output variability. For LLMs as Subjects, researchers must report any persona-related configuration settings and parameters that configure the simulated behavior. For LLMs for Tools, researchers must report the configuration for each model integrated in the tool’s architecture, including any model-specific parameter choices. For Benchmarking LLMs, researchers must report the configuration for all benchmarked models to enable fair cross-model comparisons.

Unchanged text

Advice for Reviewers

Missing version, configuration, or parameter information is typically a minor revision request. Before concluding that information is absent, reviewers should check appendices and supplementary materials, as details are sometimes reported there rather than in the main text. Rejection over missing details is rarely warranted unless the omissions obscure deeper methodological problems.

See Also

Report System and Prompt Design Changed

2026.04

Report System and Prompt Design (G3)

tl;dr: Researchers MUST describe in the PAPER the full architecture of LLM-based tools, from standalone uses to agentic systems, including the LLM’s role and its interactions with other components. Hosting, infrastructure, and latency MUST be reported. Researchers MUST publish all prompts as SUPPLEMENTARY MATERIAL, including prompt templates with representative instances and the dynamic generation process where applicable; sensitive content MUST be anonymized. If full prompt disclosure is not feasible, summaries or examples SHOULD be provided. The prompting strategy and prompt reuse across models and configurations MUST be specified, and how the prompts were developed SHOULD be described. Researchers MUST describe context-file mechanisms used and summarize in the PAPER which tools and skills were exposed; full file contents and the complete tool catalog (schemas, definitions, Model Context Protocol (MCP) servers) SHOULD be published as SUPPLEMENTARY MATERIAL. For agentic systems, researchers MUST specify the agents’ roles, reasoning frameworks, communication flows, and agent behavior traceability across LLM output, tool calls, and user interactions. For retrieval-augmented generation (RAG) and similar methods, researchers MUST describe how external data was retrieved, stored, and integrated. Where legally possible, the implementation SHOULD be open-sourced; non-disclosed proprietary components MUST be acknowledged as reproducibility limitations.

Rationale

LLM-based studies rest on artifacts that researchers design, author, or configure before any model is invoked: software layers that pre-process data, prepare prompts, filter user requests, or post-process responses, and the context those layers feed into each call (Galster et al. 2026). Context includes prompt templates, context files, tool and skill schemas, and retrieval mechanisms that bring external data into the call. For example, ChatGPT and GitHub Copilot use the same underlying models, but their outputs differ substantially because Copilot automatically adds project context. Researchers can also build tools using models directly via APIs.

Prompts are central to any LLM-based study (Sclar et al. 2024). A prompt is a concrete input to an LLM that guides its output (Schulhoff et al. 2024). Depending on the task, a prompt may include instructions (e.g., “classify the following bug report”), background information, input data, and output format specifications (e.g., “respond as JSON with fields ‘category’ and ‘justification’”), with outputs ranging from unstructured text to structured formats such as JSON (DAIR.AI 2024). A prompt template is a parameterized structure containing static elements (e.g., instructions, output format specifications) and placeholders for dynamic content (e.g., source code under analysis) that are filled in at runtime to construct concrete prompts (Schulhoff et al. 2024). In automated studies, researchers typically design prompt templates from which individual prompts are then instantiated. Prompts substantially influence a model’s output, so how they are formatted and integrated into an LLM-based study is essential for transparency, verifiability, and reproducibility. Sclar et al. (2024) (Sclar et al. 2024) question the methodological validity of comparing models with “an arbitrarily chosen, fixed prompt format”, because their research has shown that the performance of different prompt formats only weakly correlates between different models.

This section addresses everything researchers put in place before runs begin, complementing Version and Configuration G2: Version and Configuration , which focuses on model-specific details, and Session Traces G4: Session Traces , which covers what happens at runtime.

These guidelines do not apply when LLMs are used solely for language polishing, paraphrasing, translation, tone or style adaptation, or layout edits (see Scope ).

Recommendations

Researchers MUST clearly describe the tool architecture and what exactly the LLM (or ensemble of LLMs) contributes to the tool or method presented in a research paper, including any dependencies on proprietary tools that affect reproducibility. Researchers SHOULD explain design decisions and which retrieval mechanisms were implemented (keyword search, semantic similarity matching, rule-based extraction, etc.). Researchers SHOULD describe how the models were hosted and accessed; for time-sensitive measurements, the hosting choice (e.g., self-hosting on local hardware, an aggregator such as OpenRouter, or a vendor API such as the OpenAI API) can substantially impact results, so researchers MUST clarify whether local infrastructure or cloud services were used, including detailed infrastructure specifications and latency considerations. Where legally possible (e.g., not restricted by industry-partner agreements), researchers SHOULD release the source code of their implementation under an open-source license.

Standalone LLMs require limited architectural documentation, while study setups that integrate LLMs with other components, configure agents, or depend on particular context files or tool schemas require more detailed reporting. In the topic-specific paragraphs that follow, the PAPER MUST contain a high-level description of any reported component, with full details provided as SUPPLEMENTARY MATERIAL.

Prompt Reporting:

Researchers MUST report all prompts used in an empirical study, including instructions, background information, input data, and output indicators. The complete set MUST be made publicly available as SUPPLEMENTARY MATERIAL; when the paper cannot include all prompts in full, it SHOULD provide summaries and representative examples. The only exception to this disclosure requirement is when privacy, anonymity, or confidentiality concerns (e.g., when working with industry partners) prevent complete disclosure; in such cases, researchers SHOULD provide summaries and representative examples instead. For prompts containing sensitive information, researchers MUST anonymize personal identifiers, replace proprietary code with placeholders, and clearly highlight modified sections. When prompt templates are used, researchers MUST report them alongside representative instances, specifying which parts are static and which are dynamically filled. When prompts are generated dynamically (e.g., through preprocessing or retrieval-augmented generation (RAG)), the generation process MUST be thoroughly documented, including any automated algorithms or rules. Researchers SHOULD specify the exact formatting of prompts, including how code snippets were enclosed (e.g., triple backticks), whether whitespace was preserved, and how other artifacts such as error messages and stack traces were presented. For studies involving human participants who create or modify prompts, researchers SHOULD describe how these prompts were collected and analyzed.

Prompt Development:

Researchers SHOULD explain in the PAPER how they developed the prompts and why they decided to follow certain prompting strategies. If prompts from the early phases of a research project are unavailable, researchers SHOULD at least summarize the prompt evolution.

Prompt development is often iterative, involving collaboration between human researchers and AI tools. Researchers SHOULD report any instances in which LLMs were used to suggest prompt refinements and how these suggestions were incorporated. A prompt changelog can track prompt evolution, including key revisions and reasons for changes (e.g., v1.0: initial prompt; v2.0: incorporated examples of ideal responses). Because prompt effectiveness varies between models and model versions, researchers MUST make clear which prompts were used for which models in which versions and with which configuration.

Prompting Strategy and Input Handling:

Researchers MUST specify whether zero-shot, one-shot, or few-shot prompting was used. For few-shot prompts, the examples provided to the model SHOULD be clearly outlined, along with the rationale for selecting them. If multiple versions of a prompt were tested, researchers SHOULD describe how these variations were evaluated and how the final design was chosen.

When dealing with extensive or complex prompt context, researchers SHOULD describe the strategies they used to handle input length constraints (e.g., truncating, summarizing, or splitting prompts into multiple parts). Token optimization measures, such as simplifying code formatting or removing unnecessary comments, SHOULD also be documented if applied.

Context Files and Agent Configuration:

Agentic coding tools expose a range of configuration and extension mechanisms including context files, skills, subagents, hooks, settings, rules, and Model Context Protocol (MCP) servers (Galster et al. 2026). Context files such as AGENTS.md or CLAUDE.md are version-controlled Markdown files containing persistent, project-specific instructions automatically included in the context on each call (Mohsenimofidi et al. 2026). Researchers MUST describe which context-file mechanisms were used and SHOULD publish the full contents as SUPPLEMENTARY MATERIAL, because they shape agent behavior similarly to system prompts. Because context files are version-controlled, their evolution across a study is recoverable from the project’s git history and from the runtime traces reported under Session Traces G4: Session Traces .

Tool Catalog and Skill Definitions:

Agentic systems expose a set of callable tools, skills, and sub-agents to the model on each call, along with their schemas: parameter names, types, and natural-language descriptions as the model sees them. Tools vary in granularity, from one operation per tool to generic execution interfaces. Claude Code, for example, does not register individual command-line utilities but defines a single Bash tool (PowerShell on Windows) through which the agent runs arbitrary shell commands such as grep or ls (Anthropic 2026). The catalog, the wording of descriptions, and the order in which tools are presented all influence which tool the model selects, so the exact serialized form matters for reproducibility. Researchers MUST summarize in the PAPER which tools and skills were exposed to the model, and SHOULD include the complete list (names with one-line purposes), tool schemas, skill definitions, sub-agent definitions, and the identities of any connected Model Context Protocol (MCP) servers as SUPPLEMENTARY MATERIAL. Where sub-agents are used, researchers SHOULD also describe the delegation pattern: which sub-agent handles which kind of task, and how control returns to the orchestrator.

Agentic Systems and Pipelines:

If an LLM is used as a standalone system, for example by sending prompts directly to a GPT-5 model via the OpenAI API without pre-processing the prompts or post-processing the responses, a brief explanation is usually sufficient. For complex systems with pre-processing, retrieval mechanisms, or autonomous agents, aspects to consider include how the LLM interacts with other components such as databases, external APIs, and frameworks. If the LLM is part of an agent-based system that autonomously plans or executes tasks, researchers MUST describe its exact architecture, including the agents’ roles (e.g., planner, executor, coordinator), whether it is a single-agent or multi-agent system, how it interacts with external tools and users, and the reasoning framework used (e.g., chain-of-thought, self-reflection, multi-turn dialogue). For agent-based systems that use external tools (e.g., Claude Code and its subagents), researchers MUST describe agent behavior traceability by delineating: (1) LLM Input/Output, the internal deliberation, planning, and interpretation performed by the model; (2) External tool calls, the agent’s explicit invocations of APIs, databases, file systems, or other external services (e.g., via the Model Context Protocol (MCP)); (3) User or system interactions, human-in-the-loop feedback, environment responses, or multi-agent communication. This separation lets readers attribute task success to the LLM, the external tools, or their interaction; runtime trace-reporting for these components is covered in Session Traces G4: Session Traces .

RAG and Ensembles:

If retrieval-augmented generation (RAG) or related methods were used (e.g., rule-based retrieval, structured query generation, or hybrid approaches), researchers MUST describe how external data was retrieved, stored, and integrated into the LLM’s responses, including the type of storage (e.g., vector databases, relational databases, knowledge graphs) and how retrieved information was selected. Stored data used for context augmentation SHOULD be reported, including data preprocessing, versioning, and update frequency; if not confidential, an anonymized snapshot SHOULD be made available as SUPPLEMENTARY MATERIAL. For ensemble models, in addition to following the Version and Configuration G2: Version and Configuration  guideline for each model, researchers SHOULD describe the architecture connecting them: the routing logic that determines which model handles which input, model interactions, and the output combination strategy (e.g., majority voting, weighted averaging, sequential processing).

Example(s)

Schäfer et al. (2024) (Schäfer et al. 2024) evaluated LLMs for automated unit test generation, providing a detailed description of the system architecture including code parsing, prompt formulation, LLM interaction, and test suite integration. They also detail the datasets used, including sources, selection criteria, and preprocessing steps.

A second example is Yan et al. (2024)’s IVIE tool (Yan et al. 2024), which integrates LLMs into the VS Code interface. The authors document the tool architecture, detailing the IDE integration, context extraction from code editors, and the formatting pipeline for LLM-generated explanations.

Liang et al. (2024)’s paper (Liang et al. 2024) is a good example of comprehensive prompt reporting. The authors make the exact prompts available in their SUPPLEMENTARY MATERIAL on Figshare, including details such as code blocks enclosed in triple backticks. The PAPER explains the rationale behind the prompt design and the data output format, and it includes an overview figure and two concrete examples, keeping the main text concise while remaining reproducible.

Unchanged text

Benefits

Documenting the architecture and supplemental data of LLM-based systems strengthens reproducibility and transparency (Lu et al. 2024), enabling experiment replication, result validation, and cross-study comparison.

Detailed prompt documentation strengthens verifiability, reproducibility, and comparability of LLM-based studies, letting other researchers replicate studies, refine prompts, and evaluate how different content types and formatting choices influence LLM behavior. Reporting the tool catalog and context files enables other researchers to reconstruct the exact context the model saw, which is often the dominant factor in agent behavior.

Challenges

Documenting LLM-based architectures involves challenges such as proprietary APIs and dependencies that restrict disclosure, managing large-scale retrieval databases, and ensuring efficient query execution. Researchers must also balance transparency with data privacy concerns, adapt to the evolving nature of LLM integrations, and handle the complexity of multi-agent interactions and decision-making logic, all of which can impact reproducibility and system clarity.

Prompts themselves are challenging to document because they often combine multiple components such as code, error messages, and explanatory text, and privacy or confidentiality concerns can hinder sharing.

Not all systems allow reporting of complete (system) prompts, context files, and tool schemas. For commercial tools, researchers MUST report all available information and acknowledge unknown aspects as limitations. Disclosure practices vary across vendors: some publish their system prompts, others keep them proprietary. Where prompts are published, they also change between releases (Willison 2026), so researchers SHOULD record the tool version and date of use even when the prompt content itself is unavailable. Understanding suggestions of commercial tools such as GitHub Copilot might require recreating the exact state of the codebase at the time the suggestion was made, which is a challenging context to report. One solution is to use version control to capture the exact state of the codebase when a recommendation was made, keeping track of the files that were automatically added as context. We also recommend exploring open-source tools such as OpenCode (OpenCode Contributors 2025), which expose more of the configuration that shapes agent behavior.

Study Types

This guideline MUST be followed for all studies that involve tools with system-level components beyond bare LLMs, from lightweight wrappers that pre-process user input or post-process model outputs, to systems employing retrieval-augmented methods or complex agent-based architectures. It also MUST be followed by all studies that use concrete prompts or prompt templates.

For LLMs for Tools S6: LLMs for Tools , this guideline is of primary importance: researchers MUST describe the tool’s full architecture, explain how prompts were generated and structured within the tool, report the context files, tool catalog, and skill definitions used, and document how the role of each LLM fits into the overall system behavior. For Benchmarking LLMs S7: Benchmarking LLMs , researchers SHOULD describe the evaluation harness and infrastructure if it goes beyond bare model API calls (e.g., custom sandboxing, orchestration layers, or post-processing pipelines); see (Anthropic 2025) for a practitioner account of evaluation harness design for agent-based systems. Researchers using pre-defined prompts (e.g., HumanEval, SWE-Bench) MUST specify the benchmark version and any modifications made to the prompts or evaluation setup. If prompt tuning, retrieval-augmented generation (RAG), or other methods were used to adapt prompts, researchers MUST disclose and justify those changes, and SHOULD make the relevant code publicly available. For Studying LLM Usage S5: Studying LLM Usage , researchers SHOULD describe the tool architecture of the studied tool to the extent that it is accessible, as architectural details may influence observed usage patterns. For controlled experiments under Studying LLM Usage S5: Studying LLM Usage , exact prompts MUST be reported for all conditions. For LLMs as Annotators S1: LLMs as Annotators , researchers MUST document any predefined coding guides or instructions included in prompts, as these influence how the model labels artifacts. For LLMs as Judges S2: LLMs as Judges , researchers MUST report the evaluation criteria, scales, and examples embedded in the prompt to ensure consistent interpretation. For LLMs for Synthesis S3: LLMs for Synthesis  tasks (e.g., summarization, aggregation), researchers MUST document the sequence of prompts used to generate and refine outputs, including follow-ups and clarification queries. For LLMs as Subjects S4: LLMs as Subjects  (e.g., simulating human participants), researchers MUST report any role-playing instructions, constraints, or personas used to guide LLM behavior. For LLMs as Annotators S1: LLMs as Annotators , LLMs as Judges S2: LLMs as Judges , LLMs for Synthesis S3: LLMs for Synthesis , and LLMs as Subjects S4: LLMs as Subjects , if the research setup involves a custom pipeline (e.g., retrieval-augmented generation for annotation or chained prompts for synthesis), the architecture SHOULD also be reported.

Unchanged text

Advice for Reviewers

As with other guidelines, missing architectural or prompt information is typically a minor revision request unless so much is missing that methodological rigor cannot be assessed. Reviewers may ask authors to move key details from supplements into the paper body to ensure the main text is self-contained.

Regarding proprietary or confidential components, three principles apply: (1) empirically evaluating an opaque artifact is a valid scientific contribution; (2) the less available and inspectable the artifact, the weaker the contribution; (3) scientific instruments including tools, metrics, scales, and experimental materials must be fully disclosed, even when the object under study cannot be.

A challenging aspect of prompt reporting is the description of prompt development, which is inherently iterative and creative. Reviewers should not expect justification of each word choice or post-hoc rationalized accounts of prompt generation. Instead, reviewers should focus on whether (1) a new research team could use (not reproduce) exactly the same prompts in exactly the same way, and (2) potential biases or validity issues are transparent.

See Also

2026.05

Report System and Prompt Design

Summary: Researchers must describe in the paper the full architecture of LLM-based tools, from standalone uses to agentic systems, including the LLM’s role and its interactions with other components. Hosting, infrastructure, and latency must be reported. Researchers must publish all prompts as supplementary material, including prompt templates with representative instances and the dynamic generation process where applicable; sensitive content must be anonymized. If full prompt disclosure is not feasible, summaries or examples should be provided. The prompting strategy and prompt reuse across models and configurations must be specified, and how the prompts were developed should be described. Researchers must describe context-file mechanisms used and summarize in the paper which tools and skills were exposed; full file contents and the complete tool catalog (schemas, definitions, Model Context Protocol (MCP) servers) should be published as supplementary material. For agentic systems, researchers must specify the agents’ roles, reasoning frameworks, communication flows, and agent behavior traceability across LLM output, tool calls, and user interactions. For retrieval-augmented generation (RAG) and similar methods, researchers must describe how external data was retrieved, stored, and integrated. Where legally possible, the implementation should be open-sourced; non-disclosed proprietary components must be acknowledged as reproducibility limitations.

Rationale

LLM-based studies rest on artifacts that researchers design, author, or configure before any model is invoked: software layers that pre-process data, prepare prompts, filter user requests, or post-process responses, and the context those layers feed into each call (Galster et al. 2026). Context includes prompt templates, context files, tool and skill schemas, and retrieval mechanisms that bring external data into the call. For example, ChatGPT and GitHub Copilot use the same underlying models, but their outputs differ substantially because Copilot automatically adds project context. Researchers can also build tools using models directly via APIs.

Prompts are central to any LLM-based study (Sclar et al. 2024). A prompt is a concrete input to an LLM that guides its output (Schulhoff et al. 2024). Depending on the task, a prompt may include instructions (e.g., “classify the following bug report”), background information, input data, and output format specifications (e.g., “respond as JSON with fields ‘category’ and ‘justification’”), with outputs ranging from unstructured text to structured formats such as JSON (DAIR.AI 2024). A prompt template is a parameterized structure containing static elements (e.g., instructions, output format specifications) and placeholders for variable content (e.g., source code under analysis) that are filled in at runtime to construct concrete prompts (Schulhoff et al. 2024). In automated studies, researchers typically design prompt templates from which individual prompts are then instantiated. Prompts substantially influence a model’s output, so how they are formatted and integrated into an LLM-based study is essential for transparency, verifiability, and reproducibility. Sclar et al. (2024) (Sclar et al. 2024) question the methodological validity of comparing models with “an arbitrarily chosen, fixed prompt format”, because their research has shown that the performance of different prompt formats only weakly correlates between different models.

This guideline addresses everything researchers put in place before runs begin, complementing Version and Configuration (model-specific details) and Session Traces (runtime behavior). It does not apply when LLMs are used solely for language polishing, paraphrasing, translation, tone or style adaptation, or layout edits (see Scope ).

Recommendations

Researchers must clearly describe the tool architecture and what exactly the LLM (or ensemble of LLMs) contributes to the tool or method presented in a research paper, including any dependencies on proprietary tools that affect reproducibility. Researchers should explain design decisions and which retrieval mechanisms were implemented (e.g., keyword search, semantic similarity matching, rule-based extraction). Researchers should describe how the models were hosted and accessed; for time-sensitive measurements, the hosting choice (e.g., self-hosting on local hardware, an aggregator such as OpenRouter, or a vendor API such as the OpenAI API) can substantially impact results, so researchers must clarify whether local infrastructure or cloud services were used, including detailed infrastructure specifications and latency considerations. Where legally possible (e.g., not restricted by industry-partner agreements), researchers should release the source code of their implementation under an open-source license.

Standalone LLMs require limited architectural documentation, while study setups that integrate LLMs with other components, configure agents, or depend on particular context files or tool schemas require more detailed reporting. In the topic-specific paragraphs that follow, the paper must contain a high-level description of any reported component, with full details provided as supplementary material.

Prompt Reporting:

Researchers must report all prompts used in an empirical study, including instructions, background information, input data, and output indicators. The complete set must be made publicly available as supplementary material; when the paper cannot include all prompts in full, it should provide summaries and representative examples. The only exception to this disclosure requirement is when privacy, anonymity, or confidentiality concerns (e.g., when working with industry partners) prevent complete disclosure; in such cases, researchers should provide summaries and representative examples instead. For prompts containing sensitive information, researchers must anonymize personal identifiers, replace proprietary code with placeholders, and clearly highlight modified sections. When prompt templates are used, researchers must report them alongside representative instances, specifying which parts are static and which are dynamically filled. When prompts are generated dynamically (e.g., through preprocessing or retrieval-augmented generation (RAG)), the generation process must be thoroughly documented, including any automated algorithms or rules. Researchers should specify the exact formatting of prompts, including how code snippets were enclosed (e.g., triple backticks), whether whitespace was preserved, and how other artifacts such as error messages and stack traces were presented. For studies involving human participants who create or modify prompts, researchers should describe how these prompts were collected and analyzed.

Prompt Development:

Researchers should explain in the paper how they developed the prompts and why they decided to follow certain prompting strategies. If prompts from the early phases of a research project are unavailable, researchers should at least summarize the prompt evolution.

Prompt development is often iterative, involving collaboration between human researchers and AI tools. Researchers should report any instances in which LLMs were used to suggest prompt refinements and how these suggestions were incorporated. A prompt changelog can track prompt evolution, including revisions and reasons for changes (e.g., v1.0: initial prompt; v2.0: incorporated examples of ideal responses). Because prompt effectiveness varies between models and model versions, researchers must make clear which prompts were used for which models in which versions and with which configuration.

Prompting Strategy and Input Handling:

Researchers must specify whether zero-shot, one-shot, or few-shot prompting was used. For few-shot prompts, the examples provided to the model should be clearly outlined, along with the rationale for selecting them. If multiple versions of a prompt were tested, researchers should describe how these variations were evaluated and how the final design was chosen.

When dealing with extensive or complex prompt context, researchers should describe the strategies they used to handle input length constraints (e.g., truncating, summarizing, or splitting prompts into multiple parts). Token optimization measures, such as simplifying code formatting or removing unnecessary comments, should also be documented if applied.

Context Files and Agent Configuration:

Agentic coding tools expose a range of configuration and extension mechanisms including context files, skills, subagents, hooks, settings, rules, and Model Context Protocol (MCP) servers (Galster et al. 2026). Context files such as AGENTS.md or CLAUDE.md are version-controlled Markdown files containing persistent, project-specific instructions automatically included in the context on each call (Mohsenimofidi et al. 2026). Researchers must describe which context-file mechanisms were used and should publish the full contents as supplementary material, because they steer agent behavior similarly to system prompts. Because context files are version-controlled, their evolution across a study is recoverable from the project’s git history and from the runtime traces reported under Session Traces.

Tool Catalog and Skill Definitions:

Agentic systems expose a set of callable tools, skills, and sub-agents to the model on each call, along with their schemas: parameter names, types, and natural-language descriptions as the model sees them. Tools vary in granularity, from one operation per tool to generic execution interfaces. Claude Code, for example, does not register individual command-line utilities but defines a single Bash tool (PowerShell on Windows) through which the agent runs arbitrary shell commands such as grep or ls (Anthropic 2026). The catalog, the wording of descriptions, and the order in which tools are presented all influence which tool the model selects, so the exact serialized form matters for reproducibility. Researchers must summarize in the paper which tools and skills were exposed to the model, and should include the complete list (names with one-line purposes), tool schemas, skill definitions, sub-agent definitions, and the identities of any connected Model Context Protocol (MCP) servers as supplementary material. Where sub-agents are used, researchers should also describe the delegation pattern: which sub-agent handles which kind of task, and how control returns to the orchestrator.

Pipelines and Agentic Systems:

If an LLM is used as a standalone system, for example by sending prompts directly to a GPT-5 model via the OpenAI API without pre-processing the prompts or post-processing the responses, a brief explanation is usually sufficient. For complex systems with pre-processing, retrieval mechanisms, or autonomous agents, aspects to consider include how the LLM interacts with other components such as databases, external APIs, and frameworks. If the LLM is part of an agent-based system that autonomously plans or executes tasks, researchers must describe its exact architecture, including the agents’ roles (e.g., planner, executor, coordinator), whether it is a single-agent or multi-agent system, how it interacts with external tools and users, and the reasoning framework used (e.g., chain-of-thought, self-reflection, multi-turn dialogue). For agent-based systems that use external tools (e.g., Claude Code and its subagents), researchers must describe agent behavior traceability by delineating: (1) LLM Input/Output, the internal deliberation, planning, and interpretation performed by the model; (2) External tool calls, the agent’s explicit invocations of APIs, databases, file systems, or other external services (e.g., via the Model Context Protocol (MCP)); (3) User or system interactions, human-in-the-loop feedback, environment responses, or multi-agent communication. This separation lets readers attribute task success to the LLM, the external tools, or their interaction; runtime trace-reporting for these components is covered in Session Traces.

RAG and Ensembles:

If retrieval-augmented generation (RAG) or related methods were used (e.g., rule-based retrieval, structured query generation, or hybrid approaches), researchers must describe how external data was retrieved, stored, and integrated into the LLM’s responses, including the type of storage (e.g., vector databases, relational databases, knowledge graphs) and how retrieved information was selected. Stored data used for context augmentation should be reported, including data preprocessing, versioning, and update frequency; if not confidential, an anonymized snapshot should be made available as supplementary material. For ensemble models, in addition to following the Version and Configuration guideline for each model, researchers should describe the architecture connecting them: the routing logic that determines which model handles which input, model interactions, and the output combination strategy (e.g., majority voting, weighted averaging, sequential processing).

Examples

Schäfer et al. (2024) (Schäfer et al. 2024) evaluated LLMs for automated unit test generation, providing a detailed description of the system architecture including code parsing, prompt formulation, LLM interaction, and test suite integration. They also detail the datasets used, including sources, selection criteria, and preprocessing steps.

A second example is Yan et al. (2024)’s IVIE tool (Yan et al. 2024), which integrates LLMs into the VS Code interface. The authors document the tool architecture, detailing the IDE integration, context extraction from code editors, and the formatting pipeline for LLM-generated explanations.

Liang et al. (2024)’s paper (Liang et al. 2024) is a good example of comprehensive prompt reporting. The authors make the exact prompts available in their supplementary material on Figshare, including details such as code blocks enclosed in triple backticks. The paper explains the rationale behind the prompt design and the data output format, and it includes an overview figure and two concrete examples, keeping the main text concise while remaining reproducible.

Unchanged text

Benefits

Documenting the architecture and supplemental data of LLM-based systems strengthens reproducibility and transparency (Lu et al. 2024), enabling experiment replication, result validation, and cross-study comparison. Detailed prompt documentation strengthens verifiability, reproducibility, and comparability of LLM-based studies, letting other researchers replicate studies, refine prompts, and evaluate how different content types and formatting choices influence LLM behavior. Reporting the tool catalog and context files enables other researchers to reconstruct the exact context the model saw, which is often the dominant factor in agent behavior.

Challenges

Documenting LLM-based architectures involves challenges such as proprietary APIs and dependencies that restrict disclosure, managing large-scale retrieval databases, and ensuring efficient query execution. Researchers must also balance transparency with data privacy concerns, adapt to the evolving nature of LLM integrations, and handle the complexity of multi-agent interactions and decision-making logic, all of which can impact reproducibility and system clarity. Prompts themselves are challenging to document because they often combine multiple components such as code, error messages, and explanatory text, and privacy or confidentiality concerns can hinder sharing.

Not all systems allow reporting of complete (system) prompts, context files, and tool schemas. For commercial tools, researchers must report all available information and acknowledge unknown aspects as limitations. Disclosure practices vary across vendors: some publish their system prompts, others keep them proprietary. Where prompts are published, they also change between releases (Willison 2026), so researchers should record the tool version and date of use even when the prompt content itself is unavailable. Understanding suggestions of commercial tools such as GitHub Copilot might require recreating the exact state of the codebase at the time the suggestion was made, which is a challenging context to report. One solution is to use version control to capture the exact state of the codebase when a recommendation was made, keeping track of the files that were automatically added as context. We also recommend exploring open-source tools such as OpenCode (OpenCode Contributors 2025), which expose more of the configuration that controls agent behavior.

Study Types

This guideline must be followed for all studies that involve tools with system-level components beyond bare LLMs, from lightweight wrappers that pre-process user input or post-process model outputs, to systems employing retrieval-augmented methods or complex agent-based architectures. It also must be followed by all studies that use concrete prompts or prompt templates.

For LLMs for Tools, this guideline is of primary importance: researchers must describe the tool’s full architecture, explain how prompts were generated and structured within the tool, report the context files, tool catalog, and skill definitions used, and document how the role of each LLM fits into the overall system behavior. For Benchmarking LLMs, researchers should describe the evaluation harness and infrastructure if it goes beyond bare model API calls (e.g., custom sandboxing, orchestration layers, or post-processing pipelines); see (Anthropic 2025) for a practitioner account of evaluation harness design for agent-based systems. Researchers using pre-defined prompts (e.g., HumanEval, SWE-Bench) must specify the benchmark version and any modifications made to the prompts or evaluation setup. If prompt tuning, retrieval-augmented generation (RAG), or other methods were used to adapt prompts, researchers must disclose and justify those changes, and should make the relevant code publicly available. For Studying LLM Usage, researchers should describe the tool architecture of the studied tool to the extent that it is accessible, as architectural details may influence observed usage patterns. For controlled experiments under Studying LLM Usage, exact prompts must be reported for all conditions. For LLMs as Annotators, researchers must document any predefined coding guides or instructions included in prompts, as these influence how the model labels artifacts. For LLMs as Judges, researchers must report the evaluation criteria, scales, and examples embedded in the prompt to ensure consistent interpretation. For LLMs for Synthesis tasks (e.g., summarization, aggregation), researchers must document the sequence of prompts used to generate and refine outputs, including follow-ups and clarification queries. For LLMs as Subjects (e.g., simulating human participants), researchers must report any role-playing instructions, constraints, or personas used to guide LLM behavior. For LLMs as Annotators, LLMs as Judges, LLMs for Synthesis, and LLMs as Subjects, if the research setup involves a custom pipeline (e.g., retrieval-augmented generation for annotation or chained prompts for synthesis), the architecture should also be reported.

Unchanged text

Advice for Reviewers

As with other guidelines, missing architectural or prompt information is typically a minor revision request unless so much is missing that methodological rigor cannot be assessed. Reviewers may ask authors to move key details from supplements into the paper body to ensure the main text is self-contained.

Regarding proprietary or confidential components, three principles apply: (1) empirically evaluating an opaque artifact is a valid scientific contribution; (2) the less available and inspectable the artifact, the weaker the contribution; (3) scientific instruments including tools, metrics, scales, and experimental materials must be fully disclosed, even when the object under study cannot be.

A challenging aspect of prompt reporting is the description of prompt development, which is inherently iterative and creative. Reviewers should not expect justification of each word choice or post-hoc rationalized accounts of prompt generation. Instead, reviewers should focus on whether (1) a new research team could use (not reproduce) exactly the same prompts in exactly the same way, and (2) potential biases or validity issues are transparent.

See Also

Report Session Traces Changed

2026.04

Report Session Traces (G4)

tl;dr: To address model non-determinism and ensure reproducibility, especially when targeting SaaS-based commercial tools, researchers SHOULD include full interaction logs (prompts and responses) as SUPPLEMENTARY MATERIAL if privacy and confidentiality can be ensured. For agentic systems, interaction logs SHOULD be extended to cover exchanges between humans and the agent and between external tools and the agent, including human-in-the-loop feedback and approval decisions. Researchers SHOULD report the complete runtime trace as SUPPLEMENTARY MATERIAL, covering both external tool invocations (tool name, arguments, result, ordering) and which configured artifacts (skills, context files, sub-agents reported under System and Prompt Design G3: System and Prompt Design  ) were activated, so that readers can attribute task outcomes to the model, the tools, or their interaction pattern. Developed plans SHOULD be reported as SUPPLEMENTARY MATERIAL if available. When full trace disclosure is not feasible, representative or anonymized examples SHOULD be provided, and unobservable aspects of commercial-tool runs MUST be acknowledged as reproducibility limitations.

Rationale

A session is any bounded stretch of activity during which an LLM is invoked. Sessions cover one-shot prompts, batch runs, multi-turn conversations, and agentic runs that span many tool calls. A session trace records everything that crosses the LLM boundary or is produced around the model during that stretch: prompts received, responses returned, tool calls the model made together with their arguments and results, plans the model developed, and which of the statically configured artifacts (skills, context files, sub-agents, tools) were actually picked up at runtime.

Two kinds of session trace are important to capture. An interaction log captures what humans or other software tools sent to the LLM and what the LLM returned. A runtime trace captures what the LLM did at runtime: calls to external tools (APIs, file systems, databases, MCP servers, sub-agents) and activations of configured artifacts (context files, skills). Reporting both kinds is needed for an agentic run, because neither tells the full story on its own.

Even with the exact same prompts, decoding strategies, and parameters, LLMs can behave non-deterministically. Non-determinism can arise from probabilistic sampling and, even with greedy decoding (temperature = 0), from batching, input preprocessing, and floating-point arithmetic on GPUs (Chann 2023). For other researchers to verify the conclusions drawn from LLM interactions, the traces of what happened at runtime are often as important as the design of the system itself. This matters particularly for studies targeting commercial software-as-a-service (SaaS) solutions such as ChatGPT, and for agentic runs where behavior depends on how the model chose among many possible tool calls.

The rationale is similar to reporting interview transcripts in qualitative research. Just as a human participant might give different answers to the same question asked two months apart, the responses from tools such as ChatGPT can also vary over time, and the trace of an agent’s decisions on any given run is often not reproducible at a later date.

This section addresses runtime reporting and complements System and Prompt Design G3: System and Prompt Design , which covers the static artifacts that shape what the model sees.

Recommendations

Interaction Logs:

Researchers SHOULD report the full interaction logs (prompts sent to the LLM and responses returned) as part of their SUPPLEMENTARY MATERIAL. For agentic systems, interaction logs generalize to cover exchanges between humans and the agent as well as between external tools and the agent, including human-in-the-loop feedback, approval or rejection decisions, and iterative refinements. These SHOULD also be reported as SUPPLEMENTARY MATERIAL so that readers can reconstruct the sequence of exchanges and assess human oversight decisions. For traces containing sensitive information, researchers MUST anonymize personal identifiers, replace proprietary code with placeholders, and clearly highlight modified sections.

Runtime Traces:

When an LLM calls out to external tools (subagents, MCP servers, file systems, APIs, databases) or activates configured artifacts (context files, skills, sub-agents reported under System and Prompt Design G3: System and Prompt Design  ), this runtime activity forms a runtime trace distinct from the interaction log. Researchers SHOULD report the complete runtime trace as SUPPLEMENTARY MATERIAL, including for each entry the tool or artifact name, arguments (if any), result, and ordering relative to surrounding interaction-log entries. This lets readers attribute task success to the model, the external tools, or their interaction pattern, and distinguish artifacts that were configured from those that actually influenced a given run.

Agentic Plans:

For agentic systems that autonomously plan and execute tasks, the developed plans SHOULD be reported as SUPPLEMENTARY MATERIAL if available. In Claude Code, for example, a plan is a short Markdown document the user can open and edit during a session, listing the proposed steps and the files or commands the agent intends to touch. Other frameworks such as LangGraph keep plans inside the agent’s internal execution state.

All reported traces MUST be made publicly available as SUPPLEMENTARY MATERIAL, subject to privacy and confidentiality constraints. When full trace logging is not feasible, researchers SHOULD provide representative examples or anonymized traces.

Example(s)

An example of reporting full interaction logs is the study by Ronanki, Berger, and Horkoff (2023) (Ronanki, Berger, and Horkoff 2023), for which the authors reported the full answers of ChatGPT and uploaded them to Zenodo.

Benefits

Unlike human participant conversations, which often cannot be reported because of confidentiality, LLM interaction logs can be shared. This enables reproduction studies, tracking of response changes over time or across model versions, and secondary research on LLM consistency for specific SE tasks.

For agentic systems, reporting runtime traces alongside interaction logs lets readers attribute task outcomes to the right component: the model, the tool, or the orchestration pattern. Usage traces complement the static configuration reported under System and Prompt Design G3: System and Prompt Design  by showing which of the configured artifacts actually influenced a given run.

Challenges

Not all systems allow reporting of complete interaction logs with ease, and this hinders transparency and verifiability. For commercial tools, researchers MUST report all available information and acknowledge unknown aspects as limitations. Tool-call traces for commercial SaaS agents are often opaque: the user sees the final response but not the sequence of internal tool calls. When this is the case, authors should document what was and was not observable. For agents running locally or via open-source tools, these traces are usually more accessible and SHOULD be reported whenever available. Agent frameworks differ in whether and how they log agent-to-agent communication, so reporting practices vary across studies.

Study Types

Runtime trace reporting requirements depend heavily on the study type and on the accessibility of the underlying system.

For Studying LLM Usage S5: Studying LLM Usage , especially observational studies targeting commercial tools, researchers MUST report the full interaction logs except when transcripts might identify anonymous participants or reveal personal or confidential information. If complete interaction logs cannot be shared (e.g., because they contain confidential information), the prompts and responses MUST at least be summarized and described in the PAPER. For LLMs for Tools S6: LLMs for Tools  that use agentic execution, researchers SHOULD report runtime traces for the runs used to evaluate the tool. For Benchmarking LLMs S7: Benchmarking LLMs  that use agent-based harnesses, researchers SHOULD report runtime traces for representative runs, letting readers understand how task success depends on the agent’s decision sequence rather than the model’s raw output alone. For LLMs as Annotators S1: LLMs as Annotators , LLMs as Judges S2: LLMs as Judges , LLMs for Synthesis S3: LLMs for Synthesis , and LLMs as Subjects S4: LLMs as Subjects , when the research setup involves multi-turn interaction or agentic orchestration, researchers SHOULD report the corresponding interaction logs and, where applicable, runtime traces.

Unchanged text

Advice for Reviewers

As with other guidelines, missing trace information is typically a minor revision request unless so much is missing that methodological rigor cannot be assessed. Reviewers should recognize that complete trace reporting is easier for local and open-source setups than for commercial SaaS agents. When reviewing commercial-tool studies, reviewers should focus on whether authors have reported everything the system exposed and have been explicit about what remains unobservable.

See Also

2026.05

Report Session Traces

Summary: To address model non-determinism and ensure reproducibility, especially when targeting SaaS-based commercial tools, researchers should include full interaction logs (prompts and responses) as supplementary material if privacy and confidentiality can be ensured. For agentic systems, interaction logs should be extended to cover exchanges between humans and the agent and between external tools and the agent, including human-in-the-loop feedback and approval decisions. Researchers should report the complete runtime trace as supplementary material, covering both external tool invocations (tool name, arguments, result, ordering) and which configured artifacts (skills, context files, sub-agents reported under System and Prompt Design ) were activated, so that readers can attribute task outcomes to the model, the tools, or their interaction pattern. Developed plans should be reported as supplementary material if available. When full trace disclosure is not feasible, representative or anonymized examples should be provided, and unobservable aspects of commercial-tool runs must be acknowledged as reproducibility limitations.

Rationale

A session is any bounded stretch of activity during which an LLM is invoked. Sessions cover one-shot prompts, batch runs, multi-turn conversations, and agentic runs that span many tool calls. A session trace records everything that crosses the LLM boundary or is produced around the model during that stretch: prompts received, responses returned, tool calls the model made together with their arguments and results, plans the model developed, and which of the statically configured artifacts (e.g., skills, context files, sub-agents, tools) were actually picked up at runtime.

Two kinds of session trace are important to capture. An interaction log captures what humans or other software tools sent to the LLM and what the LLM returned. A runtime trace captures what the LLM did at runtime: calls to external tools (e.g., APIs, file systems, databases, MCP servers, sub-agents) and activations of configured artifacts (e.g., context files, skills). Reporting both kinds is needed for an agentic run, because neither tells the full story on its own.

Even with the exact same prompts, decoding strategies, and parameters, LLMs can behave non-deterministically. Non-determinism can arise from probabilistic sampling and, even with greedy decoding (temperature = 0), from batching, input preprocessing, and floating-point arithmetic on GPUs (Chann 2023). For other researchers to verify the conclusions drawn from LLM interactions, the traces of what happened at runtime are often as important as the design of the system itself. This matters particularly for studies targeting commercial software-as-a-service (SaaS) solutions such as ChatGPT, and for agentic runs where behavior depends on how the model chose among many possible tool calls.

The rationale is similar to reporting interview transcripts in qualitative research. Just as a human participant might give different answers to the same question asked two months apart, the responses from tools such as ChatGPT can also vary over time, and the trace of an agent’s decisions on any given run is often not reproducible at a later date. This guideline addresses runtime reporting and complements System and Prompt Design, which covers the static artifacts that determine what the model sees.

Recommendations

Interaction Logs:

Researchers should report the full interaction logs (prompts sent to the LLM and responses returned) as part of their supplementary material. For agentic systems, interaction logs generalize to cover exchanges between humans and the agent as well as between external tools and the agent, including human-in-the-loop feedback, approval or rejection decisions, and iterative refinements. These should also be reported as supplementary material so that readers can reconstruct the sequence of exchanges and assess human oversight decisions. For traces containing sensitive information, researchers must anonymize personal identifiers, replace proprietary code with placeholders, and clearly highlight modified sections.

Runtime Traces:

When an LLM calls out to external tools (e.g., APIs, file systems, databases, MCP servers, sub-agents) or activates configured artifacts (e.g., context files, skills, sub-agents reported under System and Prompt Design ), this runtime activity forms a runtime trace distinct from the interaction log. Researchers should report the complete runtime trace as supplementary material, including for each entry the tool or artifact name, arguments (if any), result, and ordering relative to surrounding interaction-log entries. This lets readers attribute task success to the model, the external tools, or their interaction pattern, and distinguish artifacts that were configured from those that actually influenced a given run. Researchers should record runtime traces in an open format compatible with the OpenTelemetry Protocol (OTLP), such as the OpenTelemetry GenAI semantic conventions (OpenTelemetry Authors 2026) or OpenInference (Arize AI 2026), and report the version used, rather than a study-specific schema.

Agentic Plans:

For agentic systems that autonomously plan and execute tasks, the developed plans should be reported as supplementary material if available. In Claude Code, for example, a plan is a short Markdown document the user can open and edit during a session, listing the proposed steps and the files or commands the agent intends to touch. Other frameworks such as LangGraph keep plans inside the agent’s internal execution state.

All reported traces must be made publicly available as supplementary material, subject to privacy and confidentiality constraints. When full trace logging is not feasible, researchers should provide representative examples or anonymized traces.

Examples

An example of reporting full interaction logs is the study by Ronanki, Berger, and Horkoff (2023) (Ronanki, Berger, and Horkoff 2023), for which the authors reported the full answers of ChatGPT and uploaded them to Zenodo. For agentic systems, Bouzenia and Pradel (2025) (Bouzenia and Pradel 2025) unified the runtime trajectories of three SE agents (RepairAgent, AutoCodeRover, OpenHands) into a common format and released the resulting 120 trajectories with 2,822 LLM interactions as a public dataset.

Benefits

Unlike human participant conversations, which often cannot be reported because of confidentiality, LLM interaction logs can be shared. This enables reproduction studies, tracking of response changes over time or across model versions, and secondary research on LLM consistency for specific SE tasks.

For agentic systems, reporting runtime traces alongside interaction logs lets readers attribute task outcomes to the right component: the model, the tool, or the orchestration pattern. Usage traces complement the static configuration reported under System and Prompt Design by showing which of the configured artifacts actually influenced a given run.

Challenges

Not all systems allow reporting of complete interaction logs with ease, and this hinders transparency and verifiability. For commercial tools, researchers must report all available information and acknowledge unknown aspects as limitations. Tool-call traces for commercial SaaS agents are often opaque: the user sees the final response but not the sequence of internal tool calls. When this is the case, authors should document what was and was not observable. For agents running locally or via open-source tools, these traces are usually more accessible and should be reported whenever available. Agent frameworks differ in whether and how they log agent-to-agent communication, so reporting practices vary across studies.

Study Types

Runtime trace reporting requirements depend heavily on the study type and on the accessibility of the underlying system. The general expectation is that interaction logs are reported as supplementary material whenever feasible; runtime traces are additionally expected when agentic execution or evaluation harnesses go beyond direct API calls.

For Studying LLM Usage, especially observational studies targeting commercial tools, researchers must report the full interaction logs except when transcripts might identify anonymous participants or reveal personal or confidential information. If complete interaction logs cannot be shared (e.g., because they contain confidential information), the prompts and responses must at least be summarized and described in the paper. For LLMs for Tools that use agentic execution, researchers should report runtime traces for the runs used to evaluate the tool. For Benchmarking LLMs that use agent-based harnesses, researchers should report runtime traces for representative runs, letting readers understand how task success depends on the agent’s decision sequence rather than the model’s raw output alone. For LLMs as Annotators, LLMs as Judges, LLMs for Synthesis, and LLMs as Subjects, when the research setup involves multi-turn interaction or agentic orchestration, researchers should report the corresponding interaction logs and, where applicable, runtime traces.

Unchanged text

Advice for Reviewers

As with other guidelines, missing trace information is typically a minor revision request unless so much is missing that methodological rigor cannot be assessed. Reviewers should recognize that complete trace reporting is easier for local and open-source setups than for commercial SaaS agents. When reviewing commercial-tool studies, reviewers should focus on whether authors have reported everything the system exposed and have been explicit about what remains unobservable.

See Also

Use Suitable Baselines, Benchmarks, and Metrics Changed

2026.04

Use Suitable Baselines, Benchmarks, and Metrics (G5)

tl;dr: Researchers MUST justify all benchmark and metric choices in the PAPER and SHOULD summarize benchmark structure, task types, and limitations. Where possible, traditional (non-LLM) baselines SHOULD be used for comparison. Researchers MUST explain in the PAPER why the selected metrics are suitable for the specific study; prior adoption in related work alone does not constitute sufficient justification. They SHOULD report established metrics to make study results comparable, but can report additional metrics that they consider appropriate. Due to the inherent non-determinism of LLMs, experiments SHOULD be repeated; the result distribution SHOULD then be reported using descriptive statistics. If comparing models or tools, researchers MUST use appropriate inferential statistics rather than relying solely on summary statistics. Researchers SHOULD justify the number of experiment repetitions, for example through a power analysis or by monitoring convergence of descriptive statistics.

Unchanged text

Rationale

Meaningful evaluation requires well-understood, valid measurement instruments. Without justified benchmarks and metrics, claims about LLM performance lack the rigor needed for scientific comparison and cumulative progress. A benchmark is a “standard tool for the competitive evaluation and comparison of competing systems or components according to specific characteristics, such as performance, dependability, or security” (Kistowski et al. 2015). “A metric is a method, algorithm, or procedure for assigning one or more numbers to a phenomenon” (Ralph et al. 2024). A baseline is a reference point, enabling comparison of LLMs against traditional algorithms with lower computational costs.

Recommendations

Selecting benchmarks and metrics requires understanding the benchmark tasks, what exactly is being measured, and how it relates to the (often latent) variables researchers actually care about. If one or more metrics or benchmarks are used, researchers:

  • MUST briefly justify (in the PAPER ) why selected benchmarks and metrics are suitable for the given task or study;

  • MUST discuss the reliability and validity (especially construct validity) of selected metrics and benchmarks;

  • SHOULD summarize the structure and tasks of the selected benchmark(s), including the programming language(s) and descriptive statistics such as the number of contained tasks and test cases;

  • SHOULD discuss the limitations of the selected benchmark(s) (e.g. widely used benchmarks such as HumanEval (M. Chen et al. 2021) and MBPP (Austin et al. 2021) Python functions assess a very specific part of software development, which is not representative of the full breadth of SE work (Chandra 2025)).

  • SHOULD include an example of a task and the corresponding test case(s) to illustrate the structure of the benchmark.

If multiple benchmarks exist for the same task, researchers SHOULD compare performance between benchmarks. When selecting only a subset of all available benchmarks, researchers SHOULD use the most specific benchmarks given the context.

Furthermore, researchers SHOULD check whether a less resource-intensive approach (e.g., for static analysis tasks or program repair) can serve as a baseline. If so, the LLM or LLM-based tool SHOULD be compared with such baselines using suitable metrics. Even if LLM-based tools outperform baselines, researchers SHOULD discuss whether the resources consumed justify the (often marginal) improvements (Menzies 2025).

To compare traditional and LLM-based approaches or different LLM-based tools, researchers SHOULD report established metrics whenever possible, as this allows secondary research. They can report additional metrics that they consider appropriate. We briefly discuss common metrics in the Example(s) subsection below. As mentioned, researchers MUST motivate why they chose a certain metric or variant thereof for their particular study. Prior adoption alone does not constitute sufficient justification; researchers MUST NOT justify metric choices solely by citing their use in prior work.

Due to LLM non-determinism, researchers SHOULD repeat experiments and report descriptive statistics of model or tool performance (e.g., arithmetic mean, median, confidence intervals, standard deviations) (Agarwal et al. 2021; Bjarnason, Silva, and Monperrus 2026). If comparing models or tools, researchers MUST use appropriate inferential statistics (e.g., hypothesis tests such as the Mann-Whitney U test, McNemar’s test for binary outcomes (Kübler et al. 2026), or bootstrap-based comparisons, along with effect sizes) rather than relying solely on differences in means or other summary statistics.

The number of required repetitions depends on factors such as the study type, the variability of the task, and the desired precision of estimates. As with sample sizes for human validation ( Human Validation G7: Human Validation  ), researchers SHOULD justify their chosen number of repetitions, for example, through a power analysis (Cohen 1992; Bjarnason, Silva, and Monperrus 2026) or by monitoring the convergence of descriptive statistics across incremental runs (Blackwell, Barry, and Cohn 2024). A pilot study can help estimate the expected variability and inform this decision.

From a measurement perspective, researchers SHOULD reflect on the theories, values, and measurement models on which the benchmarks and metrics they have selected for their study are based. For example, the phenomena labeled “bugs” in a large open dataset of software bugs are according to a certain theory of what constitutes a bug, and from the values and perspectives of the people who labeled the dataset. Reflecting on the context in which these labels were assigned and discussing whether and how the labels generalize to a new study context is crucial.

Example(s)

Common Metrics:

Two common metrics used for generation tasks are BLEU-N and pass@k. BLEU-N (Papineni et al. 2002) was originally developed for evaluating machine translation quality by measuring n-gram precision between a candidate and reference text, ranging from \(0\) (dissimilar) to \(1\) (similar). It has been widely adopted in SE for code generation tasks, though its validity in this context is debatable: n-gram overlap does not capture functional correctness, and syntactically different code can be semantically equivalent (see Challenges below). Code-specific variations such as CodeBLEU (Ren et al. 2020) and CrystalBLEU (Eghbali and Pradel 2022) attempt to address these limitations by introducing additional heuristics such as AST matching.

The metric pass@k reports the likelihood that a model correctly completes a code snippet at least once within \(k\) attempts. Originally introduced as success rate at B by Kulal et al. (2019) (Kulal et al. 2019) and popularized by M. Chen et al. (2021) (M. Chen et al. 2021), it ranges from \(0\) (no correct solution in \(k\) tries) to \(1\) (at least one correct solution), with correctness typically defined by passing test cases.

The pass@k metric is defined as: \[\text{pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}\,, \text{where:}\]

  • \(n\) is the total number of generated samples per prompt,

  • \(c\) is the number of correct samples among \(n\), and

  • \(k\) is the number of samples drawn.

The choice of \(k\) depends on the downstream task: pass@1 is critical for single-suggestion scenarios like code completion, while higher \(k\) values (e.g., \(2\), \(5\), \(10\)) assess multi-attempt capability (Rozière et al. 2023; Guo et al. 2024; Hui et al. 2024; R. Li et al. 2023). However, pass@k is not a universal metric suitable for all generation tasks. It requires a binary notion of correctness, making the metric appropriate for code synthesis evaluated via unit tests, but unsuitable for open-ended generation tasks such as comment generation, where multiple valid outputs exist.

A complementary perspective from industry practice is passk (also written pass^k), which measures the probability that all \(k\) trials succeed rather than at least one (Anthropic 2025). While pass@k increases with \(k\), passk decreases, making it useful for assessing reliability in deployment scenarios where consistent success matters (e.g., customer-facing agents).

If a study evaluates an LLM-based tool for supporting humans, a relevant metric is the acceptance rate, meaning the ratio of all accepted artifacts (e.g., test cases, code snippets) in relation to all artifacts that were generated and presented to the user. Another way of evaluating LLM-based tools is calculating inter-model agreement, which reveals how much a tool’s performance depends on specific models and versions. Metrics used to measure inter-model agreements include general agreement (percentage), Cohen’s kappa, and Krippendorff’s \(\alpha\) (see Human Validation G7: Human Validation  for recommended thresholds and best practices for measuring agreement).

Common problem types for LLM-based studies are classification, recommendation, and generation, each requiring different metrics (Hou et al. 2024). Hu et al. (2025) (Hu et al. 2025) categorized 191 LLM benchmarks by SE task, providing a valuable reference. For an overview of code foundation models, agents, and their evaluation, see Y. Yang et al. (2025) (Y. Yang et al. 2025). From a practitioner perspective, Anthropic (2025) (Anthropic 2025) categorize evaluation approaches for LLM-based agents into code-based graders (e.g., test suite execution, static analysis), model-based graders (e.g., rubric-scored LLM judgments), and human graders—a taxonomy that may help researchers systematically design evaluation strategies for agent-based tools. Common metrics include BLEU, pass@k, Accuracy@k, and Exact Match for generation; Mean Reciprocal Rank for recommendation; and Precision, Recall, F1-score, and Accuracy for classification.

Benchmark Examples:

Benchmarks used for code generation include HumanEval (available on GitHub) (M. Chen et al. 2021), MBPP (available on Hugging Face) (Austin et al. 2021), ClassEval (available on GitHub) (Du et al. 2023), LiveCodeBench (available on GitHub) (Jain et al. 2024), and SWE-bench (available on GitHub) (Jimenez et al. 2024). An example of a code translation benchmark is TransCoder (Lachaux et al. 2020) (available on GitHub).

Most benchmarks focus on generation tasks, but benchmarks for classification and recommendation also exist. For classification, DiverseVul (available on GitHub) (Y. Chen et al. 2023) provides vulnerable and non-vulnerable functions evaluated using standard classification metrics. For recommendation, CodeSearchNet (available on GitHub) (Husain et al. 2019) contains code-documentation pairs evaluated using Mean Reciprocal Rank.

Benefits

Benchmarks and metrics improve reproducibility and comparability across studies, leading to faster improvements in the cumulative body of knowledge. It is simply easier to assess a new system against one or a few benchmarks than hundreds of competing systems, and when most systems are assessed against the same benchmarks and metrics, we can see which approaches work best (at least, on those benchmarks). This comparison enables progress tracking; for example, when researchers iteratively improve a new LLM-based tool and test it against benchmarks after significant changes. For practitioners, leaderboards (published benchmark results of models) support selecting models for downstream tasks. Meanwhile, baselines help us understand just how much LLMs improve performance over non-LLM-enabled alternatives.

Challenges

Computer scientists have long used oversimplified, unvalidated metrics (Ralph and Tempero 2018) (e.g., lines of code, CPU time) as proxies for complex, multidimensional latent variables (e.g., system size, environmental sustainability). Unlike fields such as psychology where measurement theory has a longer tradition (Borsboom 2005), many AI metrics and benchmarks lack both theoretical underpinnings and empirical validation of their construct, measurement, and ecological validity.

For example, pass@k is intended to measure a model’s functional correctness, that is, its ability to generate code that produces the expected output for a given specification. However, functional correctness is only one dimension of code quality. pass@k does not capture maintainability, readability, security, or efficiency of the generated code, all of which are critical for downstream use. Furthermore, “correctness” is defined entirely by test suites, whose coverage is itself unvalidated: a solution that passes all provided tests may still be incorrect for untested inputs. Whether a test suite adequately operationalizes correctness for a given task is a subjective judgment that is rarely examined.

Similarly, HumanEval is intended to measure a model’s ability to synthesize short Python functions from docstring specifications. This operationalizes a narrow slice of “code generation ability”: it covers neither multi-file tasks, nor debugging, nor the use of existing codebases—tasks that dominate real-world software engineering (Chandra 2025; Y. Yang et al. 2025). Notably, neither pass@k nor HumanEval has undergone rigorous empirical validation of its construct validity; their widespread adoption rests on face validity and convenience rather than on evidence that they reliably measure what researchers intend them to measure (Cao et al. 2025).

Benchmarks and metrics also have generalizability problems. Prominent LLM benchmarks such as HumanEval and MBPP use Python, so researchers can optimize for Python’s idiosyncrasies, creating illusory improvements that do not generalize to other languages. Similarly, the metric BLEU-N is a syntactic metric, so code can score highly without being executable. The metric Exact Match, meanwhile, does not account for functional equivalence of syntactically different code. Both BLEU-N and Exact Match are influenced by code formatting, which confounds their intended use.

Execution-based metrics such as pass@k directly evaluate correctness by running test cases, but they require a setup with an execution environment. When researchers observe unexpected values for certain metrics, the specific results should be investigated in more detail to uncover potential problems.

Finally, benchmark data contamination—where the benchmark itself is part of the training data—may lead to artificially high performance if the model remembers the solution from the training data rather than actually solving the task based on the input data (C. Xu et al. 2024). Therefore, for proprietary LLMs that do not release their training data, researchers SHOULD consider using human validation, curating new data, or refactoring existing data (C. Xu et al. 2024).

To mitigate contamination, researchers can create new benchmark datasets by collecting data after a specified cutoff date; researchers MUST disclose data sources and collection dates for each release. However, this temporal approach requires continuous updates as new models may include the benchmark in future training data. Alternatively, keeping the benchmark private prevents inclusion in training sets, but requires trust in the benchmark creator and a system to execute it without leaking data. Researchers can also evaluate how contaminated their benchmark is (Choi et al. 2025).

Study Types

This guideline MUST be followed for all study types that automatically evaluate the performance of LLMs or LLM-based tools. The design of a benchmark and the selection of appropriate metrics are highly dependent on the specific study type and research goal. Recommending specific metrics for specific study types is beyond the scope of these guidelines, but Hu et al. (2025) provide a good overview of existing metrics for evaluating LLMs (Hu et al. 2025).

For Benchmarking LLMs S7: Benchmarking LLMs , this guideline is of primary importance: researchers MUST use established benchmarks or rigorously justify the creation of new ones, and MUST report standard metrics to enable cross-study comparison. For LLMs as Annotators S1: LLMs as Annotators , the research goal might be to assess which model comes close to a ground truth dataset created by human annotators. Especially for open annotation tasks, selecting suitable metrics to compare LLM-generated and human-generated labels is important. In general, annotation tasks can vary significantly. Are multiple labels allowed for the same sequence? Are the available labels predefined, or should the LLM generate a set of labels independently? Due to this task dependence, researchers MUST justify their metric choice, explaining what aspects of the task it captures together with known limitations. For LLMs for Tools S6: LLMs for Tools , researchers SHOULD benchmark the tool against suitable baselines using appropriate metrics that capture the tool’s intended contribution. For LLMs as Judges S2: LLMs as Judges , researchers SHOULD report inter-rater agreement metrics and validity measures to demonstrate the reliability and quality of LLM judgments. For LLMs for Synthesis S3: LLMs for Synthesis , researchers SHOULD specify metrics for comparing synthesized outputs (e.g., coverage, faithfulness) and justify their appropriateness for the synthesis task. For Studying LLM Usage S5: Studying LLM Usage , researchers SHOULD justify the measurement instruments and metrics used for studying LLM usage patterns, including any survey scales or behavioral measures. For LLMs as Subjects S4: LLMs as Subjects , researchers SHOULD compare simulated and real human responses using appropriate metrics to assess simulation fidelity. If researchers assess a well-established task such as code generation, they SHOULD report standard metrics such as pass@k and compare the performance between models. If non-standard metrics are used, researchers MUST state their reasoning.

Advice for Reviewers

Reviewers should expect manuscripts to:

  1. clearly identify the constructs or variables the study aims to measure (e.g., LLM performance, quality of generated code), including independent, dependent, and control variables;
  2. present their measurement model, i.e., which metrics, benchmarks, or baselines are used and how they relate to the target constructs;
  3. justify the selection of any metrics, benchmarks, and baselines used;
  4. discuss in detail the assumptions, reliability, and validity—especially construct validity—of each benchmark and metric;
  5. articulate any limitations regarding construct and measurement validity.

As with other guidelines, missing information about baselines or metrics is typically a revision request. However, vague descriptions that conflate broad concerns (e.g., effectiveness, quality) with specific counting methods should be questioned. Ubiquity of a benchmark or metric does not imply validity or appropriateness for a given context. Manuscripts should convey a solid understanding of construct and measurement validity by explaining and justifying their measurement models.

See Also

2026.05

Use Suitable Baselines, Benchmarks, and Metrics

Summary: Researchers must justify all benchmark and metric choices in the paper and should summarize benchmark structure, task types, and limitations. They should operationally define the phenomenon being measured, justify the sampling strategy used to select problems for inclusion in the benchmark, isolate the target capability from confounders where possible, and report an error analysis of the observed failures. Where possible, traditional (non-LLM) baselines should be used for comparison. Researchers must explain in the paper why the selected metrics are suitable for the specific study; prior adoption in related work alone does not constitute sufficient justification. They should report established metrics to make study results comparable, but can report additional metrics that they consider appropriate. Due to the inherent non-determinism of LLMs, experiments should be repeated; the result distribution should then be reported using descriptive statistics. If comparing models or tools, researchers must use appropriate inferential statistics rather than relying solely on summary statistics. Researchers should justify the number of experiment repetitions, for example through a power analysis or by monitoring convergence of descriptive statistics.

Unchanged text

Rationale

Meaningful evaluation requires well-understood, valid measurement instruments. Without justified benchmarks and metrics, claims about LLM performance lack the rigor needed for scientific comparison and cumulative progress. A benchmark is a “standard tool for the competitive evaluation and comparison of competing systems or components according to specific characteristics, such as performance, dependability, or security” (Kistowski et al. 2015). “A metric is a method, algorithm, or procedure for assigning one or more numbers to a phenomenon” (Ralph et al. 2024). A baseline is a reference point, enabling comparison of LLMs against traditional algorithms with lower computational costs.

Recommendations

Benchmark and metric selection requires understanding the benchmark tasks, what exactly is being measured, and how it relates to the (often latent) variables researchers actually care about. When one or more metrics or benchmarks are used, researchers must briefly justify in the paper why each selected benchmark and metric is suitable for the given task or study, and must discuss the reliability and validity (especially construct validity) of those choices. Beyond these requirements, researchers should:

  • provide an operational definition of the phenomenon the benchmark is intended to measure (e.g., functional correctness, code maintainability, vulnerability detection), including its scope and any sub-components (Bean et al. 2025);

  • summarize the structure and tasks of the selected benchmark(s), including the programming language(s) and descriptive statistics such as the number of contained tasks and test cases;

  • describe and justify the sampling strategy used to select problems for inclusion in the benchmark (e.g., function-completion problems in HumanEval, GitHub issues in SWE-bench, vulnerable functions in DiverseVul); if non-probability sampling (e.g., convenience) is used, discuss its implications for the generalizability of conclusions (Baltes and Ralph 2022; Bean et al. 2025);

  • discuss the limitations of the selected benchmark(s) (e.g., widely used benchmarks such as HumanEval (M. Chen et al. 2021) and MBPP (Austin et al. 2021) only test short Python functions, which is not representative of the full breadth of SE work (Chandra 2025));

  • include an example of a task and the corresponding test case(s) to illustrate the structure of the benchmark.

If multiple benchmarks exist for the same task, researchers should compare both performance and design choices (e.g., which tasks are included, how outputs are scored, what aspect of the phenomenon is covered) across benchmarks (Bean et al. 2025). When selecting only a subset of all available benchmarks, researchers should use the most specific benchmarks given the context. When adapting an existing benchmark, researchers should document what was changed and why, and should report performance on both the original and the adapted version where feasible (Bean et al. 2025).

Benchmark scores often confound the target capability with unrelated capabilities the task happens to require. For code translation benchmarks, prompt formatting alone can shift performance by up to 40% (Jia He et al. 2024; Cao et al. 2026), conflating prompt format sensitivity with translation capability. More broadly, output format compliance (e.g., specific JSON schemas or unit-test conventions) and general instruction-following capability are routinely bundled into the score (Bean et al. 2025). Researchers should identify which capabilities a benchmark conflates, isolate the target capability where possible (e.g., by reporting per-subtask breakdowns or by adopting benchmarks designed to test the target capability in isolation), and acknowledge remaining confounders as threats to construct validity.

After running a benchmark, researchers should perform an error analysis by categorizing the failures observed and reporting the relative frequency of each category. If failures concentrate on inputs that demand capabilities other than the target capability (e.g., reading across many files rather than fixing the bug the benchmark targets), this is a construct-validity threat and should be reported alongside the primary scores (Bean et al. 2025).

In addition to disclosing data sources and collection dates (see Challenges below), researchers creating or releasing a new benchmark should adopt concrete contamination-prevention mechanisms: maintaining a held-out subset of items for ongoing, uncontaminated evaluation; embedding canary strings, that is, unique markers that downstream tools can later search for in model outputs to detect inclusion in training data; and investigating whether the benchmark’s source materials may already appear in common LLM training corpora (Bean et al. 2025; Cao et al. 2026). Researchers using existing benchmarks should additionally discuss contamination as a study limitation (Limitations and Mitigations ).

Furthermore, researchers should check whether a less resource-intensive approach (e.g., for static analysis tasks or program repair) can serve as a baseline. If so, the LLM or LLM-based tool should be compared with such baselines using suitable metrics. Even if LLM-based tools outperform baselines, researchers should discuss whether the resources consumed justify the (often marginal) improvements (Menzies 2025).

To compare traditional and LLM-based approaches or different LLM-based tools, researchers should report established metrics whenever possible, as this allows secondary research. They can report additional metrics that they consider appropriate. We briefly discuss common metrics in the Examples subsection below. As mentioned, researchers must motivate why they chose a certain metric or variant thereof for their particular study. Prior adoption alone does not constitute sufficient justification; researchers must not justify metric choices solely by citing their use in prior work.

Due to LLM non-determinism, researchers should repeat experiments and report descriptive statistics of model or tool performance (e.g., arithmetic mean, median, confidence intervals, standard deviations) (Agarwal et al. 2021; Bjarnason, Silva, and Monperrus 2026). If comparing models or tools, researchers must use appropriate inferential statistics (e.g., hypothesis tests such as the Mann-Whitney U test, McNemar’s test for binary outcomes (Kübler et al. 2026), or bootstrap-based comparisons, along with effect sizes) rather than relying solely on differences in means or other summary statistics. When scoring uses ratings that vary across raters or runs (e.g., human raters, LLM-as-judge), researchers should report the distribution of ratings per item rather than only aggregated point estimates or exact-match agreement, since aggregation can mask systematic disagreement (Bean et al. 2025).

The number of required repetitions depends on factors such as the study type, the variability of the task, and the desired precision of estimates. As with sample sizes for human validation (Human Validation ), researchers should justify their chosen number of repetitions, for example, through a power analysis (Cohen 1992; Bjarnason, Silva, and Monperrus 2026) or by monitoring the convergence of descriptive statistics across incremental runs (Blackwell, Barry, and Cohn 2024). A pilot study can help estimate the expected variability and inform this decision.

From a measurement perspective, researchers should reflect on the theories, values, and measurement models on which the benchmarks and metrics they have selected for their study are based. For example, labeling phenomena as “bugs” in a large open dataset reflects a certain theory of what constitutes a bug, as well as the values and perspectives of the people who labeled the dataset. Reflecting on the context in which these labels were assigned and discussing whether and how the labels generalize to a new study context is crucial.

Researchers building or releasing new SE benchmarks should consult operational checklists. Cao et al. (2026) (Cao et al. 2026) provide HOW2BENCH, a 55-item checklist covering benchmark design, construction, evaluation, analysis, and release. Bean et al. (2025) (Bean et al. 2025) provide a complementary domain-agnostic checklist organized around the construct-validity recommendations summarized above.

Examples

Common Metrics:

Two common metrics used for generation tasks are BLEU-N and pass@k. BLEU-N (Papineni et al. 2002) was originally developed for evaluating machine translation quality by measuring n-gram precision between a candidate and reference text, ranging from 0 (dissimilar) to 1 (similar). It has been widely adopted in SE for code generation tasks, though its validity in this context is debatable: n-gram overlap does not capture functional correctness, and syntactically different code can be semantically equivalent (see Challenges below). Code-specific variations such as CodeBLEU (Ren et al. 2020) and CrystalBLEU (Eghbali and Pradel 2022) attempt to address these limitations by introducing additional heuristics such as AST matching.

The metric pass@k reports the likelihood that a model correctly completes a code snippet at least once within \(k\) attempts. Originally introduced by Kulal et al. (2019) (Kulal et al. 2019) as the success rate within a budget of \(B\) program attempts (the role now played by \(k\) in pass@k) and popularized by M. Chen et al. (2021) (M. Chen et al. 2021), it ranges from 0 (no correct solution in \(k\) tries) to 1 (at least one correct solution), with correctness typically defined by passing test cases.

The pass@k metric is defined as:

\[\text{pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}\,, \text{where:}\]

  • \(n\) is the total number of generated samples per prompt,

  • \(c\) is the number of correct samples among \(n\), and

  • \(k\) is the number of samples drawn.

The choice of \(k\) depends on the downstream task: pass@1 is critical for single-suggestion scenarios such as code completion, while higher \(k\) values (e.g., 2, 5, 10) assess multi-attempt capability and are commonly reported in technical reports of recent code LLMs (e.g., Code Llama (Rozière et al. 2023), DeepSeek-Coder (Guo et al. 2024), Qwen2.5-Coder (Hui et al. 2024), StarCoder (R. Li et al. 2023)). However, pass@k is not a universal metric suitable for all generation tasks. It requires a binary notion of correctness, making the metric appropriate for code synthesis evaluated via unit tests, but unsuitable for open-ended generation tasks such as comment generation, where multiple valid outputs exist.

A complementary perspective from industry practice is passk (also written pass^k), which measures the probability that all \(k\) trials succeed rather than at least one (Anthropic 2025). While pass@k increases with \(k\), passk decreases, making it useful for assessing reliability in deployment scenarios where consistent success matters (e.g., customer-facing agents).

If a study evaluates an LLM-based tool for supporting humans, a relevant metric is the acceptance rate, meaning the ratio of all accepted artifacts (e.g., test cases, code snippets) in relation to all artifacts that were generated and presented to the user. Another way of evaluating LLM-based tools is calculating inter-model agreement, which reveals how much a tool’s performance depends on specific models and versions. Metrics used to measure inter-model agreements include general agreement (percentage), Cohen’s \(\kappa\), and Krippendorff’s \(\alpha\) (see Human Validation for recommended thresholds and best practices for measuring agreement).

Common problem types for LLM-based studies are classification, recommendation, and generation, each requiring different metrics (Hou et al. 2024). Hu et al. (2025) (Hu et al. 2025) categorized 191 LLM benchmarks by SE task, providing a valuable reference. For an overview of code foundation models, agents, and their evaluation, see Y. Yang et al. (2025) (Y. Yang et al. 2025). From a practitioner perspective, Anthropic (2025) (Anthropic 2025) categorize evaluation approaches for LLM-based agents into code-based graders (e.g., test suite execution, static analysis), model-based graders (e.g., rubric-scored LLM judgments), and human graders. This taxonomy may help researchers systematically design evaluation strategies for agent-based tools. Common metrics include BLEU, pass@k, Accuracy@k, and Exact Match for generation; Mean Reciprocal Rank for recommendation; and Precision, Recall, F1-score, and Accuracy for classification.

Benchmark Examples:

Benchmarks used for code generation include HumanEval (available on GitHub) (M. Chen et al. 2021), MBPP (available on Hugging Face) (Austin et al. 2021), ClassEval (available on GitHub) (Du et al. 2024), LiveCodeBench (available on GitHub) (Jain et al. 2024), and SWE-bench (available on GitHub) (Jimenez et al. 2024). An example of a code translation benchmark is TransCoder (Rozière et al. 2020) (available on GitHub).

Most benchmarks focus on generation tasks, but benchmarks for classification and recommendation also exist. For classification, DiverseVul (available on GitHub) (Y. Chen et al. 2023) provides vulnerable and non-vulnerable functions evaluated using standard classification metrics. For recommendation, CodeSearchNet (available on GitHub) (Husain et al. 2019) contains code-documentation pairs evaluated using Mean Reciprocal Rank.

Benefits

Benchmarks and metrics improve reproducibility and comparability across studies, leading to faster improvements in the cumulative body of knowledge. It is simply easier to assess a new system against one or a few benchmarks than hundreds of competing systems, and when most systems are assessed against the same benchmarks and metrics, we can see which approaches work best (at least, on those benchmarks). This comparison enables progress tracking; for example, when researchers iteratively improve a new LLM-based tool and test it against benchmarks after substantial changes. For practitioners, leaderboards (published benchmark results of models) support selecting models for downstream tasks. Meanwhile, baselines help us understand just how much LLMs improve performance over non-LLM-enabled alternatives.

Challenges

Computer scientists commonly use simple metrics (e.g., lines of code, CPU time) as proxies for complex, multidimensional latent variables (e.g., system size, environmental sustainability), without empirically validating that the metrics capture the intended construct (Ralph and Tempero 2018). Unlike fields such as psychology where measurement theory has a longer tradition (Borsboom 2005), many AI metrics and benchmarks lack both theoretical underpinnings and empirical validation of their construct, measurement, and ecological validity.

These validity issues are widespread, not isolated. In a survey of 572 code benchmarks released between 2014 and 2025, Cao et al. (2026) (Cao et al. 2026) found that 84.2% did not consider test-suite coverage when constructing test cases, 64.0% reported single-pass evaluations without controlling for randomness, and 82.5% did not address data contamination. Bean et al. (2025) (Bean et al. 2025) reported comparable patterns in a parallel review of 445 LLM benchmarks from ML and NLP venues. As a result, model performance on these benchmarks can reflect benchmark artifacts rather than the capability the benchmark claims to test.

For example, pass@k is intended to measure a model’s functional correctness, that is, its capability to generate code that produces the expected output for a given specification. However, functional correctness is only one dimension of code quality. pass@k does not capture maintainability, readability, security, or efficiency of the generated code, all of which are critical for downstream use. Furthermore, “correctness” is defined entirely by test suites, whose coverage is itself unvalidated: a solution that passes all provided tests may still be incorrect for untested inputs. Whether a test suite adequately operationalizes correctness for a given task is a subjective judgment that is rarely examined.

Similarly, HumanEval is intended to measure a model’s capability to synthesize short Python functions from docstring specifications. This operationalizes a narrow slice of “code generation capability”: it covers neither multi-file tasks, nor debugging, nor the use of existing codebases. These tasks dominate real-world software engineering (Chandra 2025; Y. Yang et al. 2025). Notably, neither pass@k nor HumanEval has undergone rigorous empirical validation of its construct validity; their widespread adoption rests on face validity and convenience rather than on evidence that they reliably measure what researchers intend them to measure (Cao et al. 2026). HumanEval also contains implementation, documentation, and test-case bugs in its original release, which directly affect score interpretation (Liu et al. 2023); comparable issues have been documented in other widely-used code benchmarks (Cao et al. 2026).

Benchmarks and metrics also have generalizability problems. Prominent LLM benchmarks such as HumanEval and MBPP use Python, so researchers can optimize for Python’s idiosyncrasies, producing gains that do not generalize to other languages. Similarly, the metric BLEU-N is a syntactic metric, so code can score highly without being executable. The metric Exact Match, meanwhile, does not account for functional equivalence of syntactically different code. Both BLEU-N and Exact Match are influenced by code formatting, which confounds their intended use.

Execution-based metrics such as pass@k directly evaluate correctness by running test cases, but they require an execution environment. When metric values look surprising, examine the specific test cases driving them: common causes include outliers, bugs in test suites or scoring code, and items that exercise capabilities other than the one being measured.

Finally, benchmark data contamination, where the benchmark itself is part of the training data, may lead to artificially high performance if the model remembers the solution from the training data rather than deriving it from the input (C. Xu et al. 2024). Therefore, for proprietary LLMs that do not release their training data, researchers should consider using human validation, curating new data, or refactoring existing data (C. Xu et al. 2024).

To mitigate contamination, researchers can create new benchmark datasets by collecting data after a specified cutoff date; researchers must disclose data sources and collection dates for each release. However, this temporal approach requires continuous updates as new models may include the benchmark in future training data. Alternatively, keeping the benchmark private prevents inclusion in training sets, but requires trust in the benchmark creator and a system to execute it without leaking data. Researchers can also evaluate how contaminated their benchmark is (Choi et al. 2025).

Study Types

This guideline must be followed for all study types that automatically evaluate the performance of LLMs or LLM-based tools. The design of a benchmark and the selection of appropriate metrics are highly dependent on the specific study type and research goal. Recommending specific metrics for specific study types is beyond the scope of these guidelines, but Hu et al. (2025) provide a good overview of existing metrics for evaluating LLMs (Hu et al. 2025).

For Benchmarking LLMs, this guideline is of primary importance: researchers must use established benchmarks or rigorously justify the creation of new ones, must report standard metrics to enable cross-study comparison, and should report an error analysis alongside the primary scores so that readers can assess whether reported gains reflect the target capability or other capabilities the task happens to require. For LLMs as Annotators, the research goal might be to assess which model comes close to a ground truth dataset created by human annotators. Especially for open annotation tasks, selecting suitable metrics to compare LLM-generated and human-generated labels is important. In general, annotation tasks can vary widely. Are multiple labels allowed for the same sequence? Are the available labels predefined, or should the LLM generate a set of labels independently? Due to this task dependence, researchers must justify their metric choice, explaining what aspects of the task it captures together with known limitations. For LLMs for Tools, researchers should benchmark the tool against suitable baselines using appropriate metrics that capture the tool’s intended contribution. For LLMs as Judges, researchers should report inter-rater agreement metrics and validity measures to demonstrate the reliability and quality of LLM judgments. For LLMs for Synthesis, researchers should specify metrics for comparing synthesized outputs (e.g., coverage, faithfulness) and justify their appropriateness for the synthesis task. For Studying LLM Usage, researchers should justify the measurement instruments and metrics used for studying LLM usage patterns, including any survey scales or behavioral measures. For LLMs as Subjects, researchers should compare simulated and real human responses using appropriate metrics to assess simulation fidelity. If researchers assess a well-established task such as code generation, they should report standard metrics such as pass@k and compare the performance between models. If non-standard metrics are used, researchers must state their reasoning.

Advice for Reviewers

Reviewers should expect manuscripts to: (1) clearly identify the constructs or variables the study aims to measure (e.g., LLM performance, quality of generated code), including independent, dependent, and control variables; (2) present their measurement model, i.e., which metrics, benchmarks, or baselines are used and how they relate to the target constructs; (3) justify the selection of metrics, benchmarks, and baselines; (4) discuss in detail the assumptions, reliability, and validity (especially construct validity) of each benchmark and metric; (5) articulate any limitations regarding construct and measurement validity.

As with other guidelines, missing information about baselines or metrics is typically a revision request. However, vague descriptions that conflate broad concerns (e.g., effectiveness, quality) with specific counting methods should be questioned. Ubiquity of a benchmark or metric does not imply validity or appropriateness for a given context. Manuscripts should convey a solid understanding of construct and measurement validity by explaining and justifying their measurement models.

See Also

Use an Open LLM as a Baseline Changed

2026.04

Use an Open LLM as a Baseline (G6)

tl;dr: Researchers SHOULD include an open LLM as a baseline when using commercial models and report inter-model agreement. By open LLM, we mean a model that everyone with the required hardware can deploy and operate. Such models are usually “open weight.” A full replication package with step-by-step instructions SHOULD be provided as part of the SUPPLEMENTARY MATERIAL.

Unchanged text

Rationale

Reproducibility depends on access to the model under study. When research relies exclusively on proprietary models, other researchers cannot independently verify or build upon the findings. Including an open LLM as a baseline ensures that at least part of the study can be fully replicated.

Recommendations

Empirical studies using LLMs in SE, especially those that target commercial tools or models, SHOULD incorporate an open LLM as a baseline and report established metrics for inter-model agreement. We acknowledge that including an open LLM baseline might not always be possible, for example, if the study involves human participants, and letting them work on the tasks using two different models might not be feasible. Using an open model as a baseline is also not necessary if the use of the LLM is tangential to the study goal.

Open models allow other researchers to verify research results and build upon them, even without access to commercial models. A comparison of commercial and open models also allows researchers to contextualize model performance. Researchers SHOULD provide a complete replication package as part of their SUPPLEMENTARY MATERIAL, including clear step-by-step instructions on how to verify and reproduce the results reported in the paper.

Open LLMs are available on platforms such as Hugging Face and can be hosted locally using frameworks such as Ollama or LM Studio, or on cloud services such as Together AI, AWS, Azure, and Google Cloud.

The term “open” can have different meanings in the context of LLMs. Widder, Whittaker, and West (2024) discuss three types of openness (transparency, reusability, and extensibility) and what openness can and cannot provide (Widder, Whittaker, and West 2024). The Open Source Initiative (OSI) (Open Source Initiative (OSI) 2025) defines open-source AI as having access to everything needed to understand, modify, share, retrain, and recreate the model.

Researchers can also explore open-source tools such as Continue, Cline, and opencode as alternatives to commercial tools like GitHub Copilot and Claude Code, enabling instrumentation, architectural transparency, and detailed telemetry collection.

Example(s)

An increasing number of studies have adopted open LLMs as baseline models. For example, C. Wang et al. (2024) evaluated seven advanced LLMs, six of which were open-source, testing 145 API mappings drawn from eight popular Python libraries across 28,125 completion prompts aimed at detecting deprecated API usage in code completion (C. Wang et al. 2024). Moumoula et al. (2024) compared four LLMs on a cross-language code clone detection task (Moumoula et al. 2024). Three evaluated models were open-source. Gonçalves et al. (2025) fine-tuned the open LLM LLaMA 3.2 on a refined version of the DiverseVul dataset to benchmark vulnerability detection performance (Gonçalves et al. 2025). CodeBERT, a bimodal transformer pre-trained by Microsoft Research, has been widely used as an open baseline (G. Yang et al. 2023; Xia, Shao, and Deng 2024; Sonnekalb et al. 2022; Cai et al. 2024), with model weights, source code, and data-processing scripts published on GitHub (Microsoft 2023).

Unchanged text

Benefits

A true open LLM baseline improves reproducibility by exposing model architectures, parameter settings, and ideally training data, enabling independent verification of results. Moreover, by adopting an open-source baseline, researchers can directly compare novel methods against established performance metrics without the variability introduced by proprietary systems. These models allow detailed inspection of data processing pipelines and decision-making routines, helping identify biases and model limitations. Furthermore, unlike closed-source alternatives, which can be withdrawn or altered without notice, open LLMs ensure long-term accessibility and stability, preserving critical resources for future studies. Finally, the licensing requirements associated with open-source implementations lower financial barriers, making advanced language models attainable for research groups operating under constrained budgets.

Challenges

Open-source LLMs face several notable challenges. First, they often lag behind the most advanced proprietary “frontier” models in common benchmarks, making it difficult for researchers to demonstrate clear improvements when evaluating new methods using open LLMs alone. Additionally, deploying and experimenting with these models typically requires substantial hardware resources, in particular high-performance GPUs, which may be beyond reach for many academic groups. The notion of “openness” remains in flux: many models release only trained weights without training data or methodological details (“open weight” openness) (Gibney 2024), which is why we reference the OSI definition in our recommendations (Open Source Initiative (OSI) 2025). Finally, unlike APIs provided by proprietary vendors (e.g., the OpenAI API), installing, configuring, and fine-tuning open-source models can be technically demanding. Documentation is often sparse or fragmented, placing a high barrier to entry for researchers without specialized engineering support.

Study Types

This guideline applies primarily to study types in which the researcher controls which LLM is used. In formal benchmarking studies and controlled experiments (see Benchmarking LLMs S7: Benchmarking LLMs  ), an open LLM MUST be one of the models under evaluation. When evaluating LLMs for Tools S6: LLMs for Tools , researchers SHOULD use an open LLM as a baseline whenever it is technically feasible; if integration proves too complex, they SHOULD report the initial benchmarking results of open models. For LLMs as Annotators S1: LLMs as Annotators  and LLMs as Judges S2: LLMs as Judges , researchers SHOULD compare annotation or judgment quality from open vs. commercial models to assess the extent to which results depend on a specific proprietary model. For LLMs for Synthesis S3: LLMs for Synthesis , researchers SHOULD compare synthesis results from open and commercial models to evaluate robustness of the findings. For Studying LLM Usage S5: Studying LLM Usage , using an open LLM as a baseline is often not feasible when the study observes participants using specific commercial tools; in such cases, investigators SHOULD explicitly acknowledge its absence and discuss how this limitation might affect their conclusions. For LLMs as Subjects S4: LLMs as Subjects , using an open LLM as a baseline may similarly be impractical if the study design requires the capabilities of a specific model; researchers SHOULD acknowledge this limitation when applicable.

Unchanged text

Advice for Reviewers

Reviewers should distinguish between LLM use that is central to the research (e.g., building LLM-driven SE tools, using LLMs for synthesis) and use that is tangential (e.g., generating recruiting materials). An open LLM baseline is expected only when LLM use is central; otherwise, its absence need not be justified. When an open baseline is expected, reviewers should look for either the use of an open LLM or a convincing argument for why it is impractical. If authors claim openness, some justification for that characterization is appropriate. Where practical, reviewers should examine whether the replication package contains sufficiently detailed instructions. Reviewers should not penalize studies for performance differences between open and proprietary models, as this is beyond the authors’ control.

See Also

2026.05

Use an Open LLM as a Baseline

Summary: Researchers should include an open LLM as a baseline when using commercial models and report inter-model agreement. By open LLM, we mean a model that everyone with the required hardware can deploy and operate. Such models are usually “open weight.” A full replication package with step-by-step instructions should be provided as part of the supplementary material.

Unchanged text

Rationale

Reproducibility depends on access to the model under study. When research relies exclusively on proprietary models, other researchers cannot independently verify or build upon the findings. Including an open LLM as a baseline ensures that at least part of the study can be fully replicated.

Recommendations

Empirical studies using LLMs in SE, especially those that target commercial tools or models, should incorporate an open LLM as a baseline and report established metrics for inter-model agreement. We acknowledge that including an open LLM baseline might not always be possible, for example, if the study involves human participants, and letting them work on the tasks using two different models might not be feasible. Using an open model as a baseline is also not necessary if the use of the LLM is tangential to the study goal.

Open models allow other researchers to verify research results and build upon them, even without access to commercial models. A comparison of commercial and open models also allows researchers to contextualize model performance. Researchers should provide a complete replication package as part of their supplementary material, including clear step-by-step instructions on how to verify and reproduce the results reported in the paper.

Open LLMs are available on platforms such as Hugging Face and can be hosted locally using frameworks such as Ollama or LM Studio, or on cloud services such as Together AI, AWS, Azure, and Google Cloud. Researchers can also explore open-source tools such as Continue, Cline, and opencode (OpenCode Contributors 2025), which publish their full agent code, system prompts, and tool catalog. Vendor-hosted services such as GitHub Copilot and Claude Code make parts of their tooling public (e.g., editor extensions, hook examples), but their deployed agent loops, system prompts, and tool catalogs typically remain proprietary.

The term “open” can have different meanings in the context of LLMs. Widder, Whittaker, and West (2024) discuss three types of openness (i.e., transparency, reusability, and extensibility) and what openness can and cannot provide (Widder, Whittaker, and West 2024). The Open Source Initiative (OSI) (Open Source Initiative (OSI) 2025) defines open-source AI as having access to everything needed to understand, modify, share, retrain, and recreate the model.

Examples

An increasing number of studies have adopted open LLMs as baseline models. For example, C. Wang et al. (2024) evaluated seven advanced LLMs, six of which were open-source, testing 145 API mappings drawn from eight popular Python libraries across 28,125 completion prompts aimed at detecting deprecated API usage in code completion (C. Wang et al. 2024). Moumoula et al. (2024) compared four LLMs on a cross-language code clone detection task (Moumoula et al. 2024). Three evaluated models were open-source. Gonçalves et al. (2025) fine-tuned the open LLM LLaMA 3.2 on a refined version of the DiverseVul dataset to benchmark vulnerability detection performance (Gonçalves et al. 2025). CodeBERT, a bimodal transformer pre-trained by Microsoft Research, is published with model weights, source code, and data-processing scripts on GitHub (Microsoft 2023). It has been used as an open baseline across diverse SE tasks, including exploit code generation (G. Yang et al. 2023), vulnerability detection (Xia, Shao, and Deng 2024), code clone detection (Sonnekalb et al. 2022), and programming assistance for exception handling (Cai et al. 2024).

Unchanged text

Benefits

A true open LLM baseline improves reproducibility by exposing model architectures, parameter settings, and ideally training data, enabling independent verification of results. Moreover, by adopting an open-source baseline, researchers can directly compare novel methods against established performance metrics without the variability introduced by proprietary systems. These models allow detailed inspection of data processing pipelines and decision-making routines, helping identify biases and model limitations. Furthermore, unlike closed-source alternatives, which can be withdrawn or altered without notice, open LLMs ensure long-term accessibility and stability, preserving critical resources for future studies. Finally, the licensing requirements associated with open-source implementations lower financial barriers, making advanced language models attainable for research groups operating under constrained budgets.

Challenges

Open-source LLMs face several notable challenges. First, they often lag behind the most advanced proprietary “frontier” models in common benchmarks, making it difficult for researchers to demonstrate clear improvements when evaluating new methods using open LLMs alone. Additionally, deploying and experimenting with these models typically requires substantial hardware resources, in particular high-performance GPUs, which may be beyond reach for many academic groups. The notion of “openness” remains in flux: many models release only trained weights without training data or methodological details (“open weight” openness) (Gibney 2024), which is why we reference the OSI definition in our recommendations (Open Source Initiative (OSI) 2025). Finally, unlike APIs provided by proprietary vendors (e.g., the OpenAI API), installing, configuring, and fine-tuning open-source models can be technically demanding. Documentation is often sparse or fragmented, placing a high barrier to entry for researchers without specialized engineering support.

Study Types

This guideline applies primarily to study types in which the researcher controls which LLM is used. In formal benchmarking studies (see Benchmarking LLMs ), an open LLM must be one of the models under evaluation; without one, the reported scores cannot be re-run by others, defeating the cumulative comparability that distinguishes a benchmark from a one-off measurement. For controlled experiments, an open LLM should be one of the models under test; if the experimental design requires capabilities that only a specific commercial model exhibits, researchers should acknowledge this as a limitation. When evaluating LLMs for Tools, researchers should use an open LLM as a baseline whenever it is technically feasible; if integration proves too complex, they should report the initial benchmarking results of open models. For LLMs as Annotators and LLMs as Judges, researchers should compare annotation or judgment quality from open vs. commercial models to assess the extent to which results depend on a specific proprietary model. For LLMs for Synthesis, researchers should compare synthesis results from open and commercial models to evaluate robustness of the findings. For Studying LLM Usage, using an open LLM as a baseline is often not feasible when the study observes participants using specific commercial tools; in such cases, investigators should explicitly acknowledge its absence and discuss how this limitation might affect their conclusions. For LLMs as Subjects, using an open LLM as a baseline may similarly be impractical if the study design requires the capabilities of a specific model; researchers should acknowledge this limitation when applicable.

Unchanged text

Advice for Reviewers

Reviewers should distinguish between LLM use that is central to the research (e.g., building LLM-driven SE tools, using LLMs for synthesis) and use that is tangential (e.g., generating recruiting materials). An open LLM baseline is expected only when LLM use is central; otherwise, its absence need not be justified. When an open baseline is expected, reviewers should look for either the use of an open LLM or a convincing argument for why it is impractical. If authors claim openness, some justification for that characterization is appropriate. Where practical, reviewers should examine whether the replication package contains sufficiently detailed instructions. Reviewers should not penalize studies for performance differences between open and proprietary models, as this is beyond the authors’ control.

See Also

Use Human Validation for LLM Outputs Changed

2026.04

Use Human Validation for LLM Outputs (G7)

tl;dr: If assessing the quality of generated artifacts is important and no reference datasets or suitable comparison metrics exist, researchers SHOULD use human validation for LLM outputs. If they do, they MUST define the measured construct (e.g., usability, maintainability) and describe the measurement instrument in the PAPER. Researchers SHOULD consider human validation early in the study design (not as an afterthought) and build on established reference models for human-LLM comparison. When developing or adapting measurement instruments, researchers MUST share them. When aggregating LLM judgments, methods and rationale SHOULD be reported and inter-rater agreement SHOULD be assessed. Confounding factors SHOULD be controlled for, and power analysis SHOULD be conducted to ensure statistical robustness.

Rationale

Even with well-justified benchmarks and metrics ( Benchmarks and Metrics G5: Benchmarks and Metrics  ), automated measurement captures only the constructs the benchmark was designed to measure. Subjective constructs and constructs that no current benchmark covers require human assessment.

Recommendations

When LLMs automate research or software development tasks previously performed by humans, the LLM’s performance needs to be assessed. Where no benchmark adequately operationalizes the target construct, researchers SHOULD validate LLM outputs against human judgment.

Study Design Considerations:

Integrating human participants in the study design will require additional considerations, including a recruitment strategy, annotation guidelines, training sessions, or ethical approvals. Therefore, researchers SHOULD consider human validation early in the study design, not as an afterthought. Authors MUST clearly define in the PAPER the constructs that the human and LLM annotators evaluate (Ralph and Tempero 2018). When designing custom instruments to assess LLM output (e.g., questionnaires, scales), researchers MUST share their instruments.

Subjective Judgment and Agreement:

When the judgment is subjective (i.e., depends on the judge’s values or theories), the LLM output SHOULD be evaluated against judgments aggregated from multiple human judges, and researchers SHOULD clearly describe their aggregation method and reasoning. One systematic approach is to randomly order the objects requiring judgment and put them in groups small enough to judge in one to three hours. Two or three human experts review the first group together, simultaneously judging, discussing, and creating a set of decision rules documenting their reasoning. Then, the experts iterate among rounds of:

  1. independently rating one group of objects,

  2. calculating inter-rater agreement (IRA) or reliability (IRR),

  3. meeting to discuss disagreements and reach consensus,

  4. updating the decision rules so the disagreement will not recur.

The goal is to reach a target IRA or IRR (e.g., Krippendorff’s \(\alpha>0.8\)), indicating sufficient decision rules. Once this target is reached and sustained for two to three groups, it is permissible to continue with a single rater. Researchers SHOULD report measures of IRA or IRR, preferably broken down by round.

Confounding factors SHOULD be discussed and, where feasible, controlled for (e.g., by categorizing participants according to their level of experience or expertise). Where applicable, researchers may perform a power analysis (Cohen 1992; Dybå, Kampenes, and Sjøberg 2006) to estimate the required sample size, ensuring sufficient statistical power in their experimental design. However, we also note that to our knowledge, there is limited established guidance specific to determining sample sizes for LLM-human comparisons. In other fields, 100 comparisons are often made without further explanation (Tam et al. 2024).

Researchers SHOULD use established reference models to compare humans with LLMs. For example, Schneider, Fotrousi, and Wohlrab (2025) (Schneider, Fotrousi, and Wohlrab 2025) outline design considerations for studies comparing LLMs with humans. If studies aim to automate annotating software artifacts using an LLM, researchers SHOULD follow systematic approaches to decide whether and how human annotators can be replaced (e.g., showing that a jury of three LLMs exhibit model-to-model agreement comparable to trained human annotators, such as Krippendorff’s \(\alpha>0.8\)). Model-to-model agreement requires the same threshold as human-to-human agreement, since low thresholds may simply reflect shared model biases rather than annotation quality.

Agentic Tools:

Besides generating and modifying content, agentic software development tools such as Claude Code can autonomously call command-line tools or pull in additional information from MCP servers. For such tools, it makes sense to separate textual output (e.g., file changes) from output related to tool execution. When evaluating agentic tools, researchers SHOULD assess the feedback that users provided for proposed changes, report statistics on how frequently they accepted the content, and how they modified it. Agentic human-in-the-loop interaction is a built-in human validation, with larger degrees of freedom than traditional experiments that use LLMs directly.

Example(s)

T. Ahmed et al. (2025) (T. Ahmed et al. 2025) proposed a systematic method for deciding whether LLMs can replace human annotators on a given task, using model-to-model agreement as an initial screening criterion (with a threshold of \(\alpha>0.5\)) and model confidence for sample-level decisions. However, their threshold is well below the levels generally considered acceptable for inter-rater agreement. Krippendorff (2018) recommends discarding data with \(\alpha<0.667\), considers \(0.667 \leq \alpha < 0.8\) sufficient only for tentative conclusions, and requires \(\alpha \geq 0.8\) for reliable data (Krippendorff 2018). Researchers adopting similar screening approaches should apply thresholds consistent with these established standards. Notably, while three LLMs exhibited higher inter-model agreement than human annotators on some tasks, human-model agreement remained low on others. This illustrates that high model-model reliability does not guarantee alignment with human judgment, reinforcing the need for human validation before assuming that LLM annotations are valid. Moreover, a high model-model agreement could merely indicate that the models share systematic biases and hence reliably agree on the wrong answer.

Hymel and Johnson (2025) (Hymel and Johnson 2025) evaluated ChatGPT’s capability to generate requirements documents by comparing an LLM-generated and a human-generated document based on the same business use case. Domain experts reviewed both documents and attempted to distinguish their origin.

Benefits

Validating LLMs against human judgments builds confidence in the LLM’s accuracy and validity, increasing the trustworthiness of findings, especially when IRA or IRR metrics are reported (Khraisha et al. 2024). Furthermore, incorporating feedback from human judges may help to improve the LLM or LLM-based tool based on the reported experiences.

Challenges

Assessing a construct like LLM performance is challenging because ensuring that a construct is defined well and operationalized using an appropriate measurement model requires a deep understanding of (1) the construct, (2) construct validity in general, and (3) instrumentation (Sjøberg and Bergersen 2023; Ralph et al. 2024). Comparing an LLM to human judges is typically slower and more expensive than machine-generated measures. More fundamentally, neither human judgment nor machine-generated measures provides an objective ground truth against which LLM accuracy can be firmly determined.

Human judgments exhibit variability due to differences in experience, expertise, interpretations, and personal biases (McDonald, Schoenebeck, and Forte 2019). When diverse humans rate items reliably given clear decision rules, we assume that reliability implies validity, but it does not. Measuring reliability is much easier than measuring validity, and often the best researchers can do is argue conceptually for why their judges, decision rules, and constructs should produce valid ratings.

Researchers must weigh the cost and time need for human validation against the expected corresponding improvement in validity over machine-generated measures.

Study Types

This guideline applies to all study types, although the need for human validation varies. For LLMs as Annotators S1: LLMs as Annotators , researchers SHOULD validate LLM-generated annotations against human annotators to assess labeling quality and identify systematic biases. When using LLMs as Judges S2: LLMs as Judges , researchers SHOULD co-create initial rating criteria with humans and validate a sample of LLM judgments against human expert assessments. For LLMs for Synthesis S3: LLMs for Synthesis , researchers SHOULD employ human oversight to verify that qualitative interpretations and synthesized outputs faithfully represent the underlying data. For LLMs as Subjects S4: LLMs as Subjects , researchers SHOULD validate simulated responses against real human data to assess the fidelity of the simulation. For Studying LLM Usage S5: Studying LLM Usage , researchers SHOULD carefully reflect on the validity of their evaluation criteria and validate subjective assessments with human experts. For LLMs for Tools S6: LLMs for Tools  whose output is supposed to match human expectations, researchers SHOULD validate the LLM output against human judgment. For Benchmarking LLMs S7: Benchmarking LLMs , there is less need for human validation when using extensively validated and widely-used benchmarks, but researchers SHOULD employ human validation when creating or adapting new benchmarks.

Advice for Reviewers

Human validation may be the most challenging of our guidelines to assess because it often requires evaluating conceptual arguments. If LLM output is validated only by comparison with other LLMs, reviewers should look for quantitative empirical evidence that such comparison is reliable and valid. High inter-model agreement alone is insufficient, as reliability does not imply validity. Similarly, reviewers should expect evidence that any employed benchmarks are reliable and valid. Absent such evidence, human validation is warranted. A single human judge is appropriate only when judgments depend on widely accepted theories and involve limited value conflict (e.g., tagging method names containing abbreviations). For multiple judges, reviewers should expect IRA/IRR improvement techniques as described in the recommendations above (experienced raters, organized rounds, consensus meetings, updated decision rules). Low IRA or IRR (e.g., Krippendorff’s \(\alpha<0.8\)) without these techniques is a concern. Conversely, if authors have followed best practices and still obtained mediocre results (e.g., \(0.66<\alpha<0.8\)), this should be noted as a limitation. Beyond reliability, reviewers should expect authors to explain conceptually why their human judgments should be valid, considering construct definitions, decision rules, and judge expertise. As with other guidelines, missing information is typically a revision request. Absent judge instructions, instruments, decision rules, or construct definitions may prevent assessment of rigor and validity, while missing recruitment details are less critical. Clarification requests about construct definitions are routine and should not alone warrant rejection.

2026.05

Use Human Validation for LLM Outputs

Summary: If assessing the quality of generated artifacts is important and no reference datasets or suitable comparison metrics exist, researchers should use human validation for LLM outputs. If they do, they must define the measured construct (e.g., usability, maintainability) and describe the measurement instrument in the paper. Researchers should consider human validation early in the study design (not as an afterthought) and build on established reference models for human-LLM comparison. When developing or adapting measurement instruments, researchers must share them. When aggregating LLM judgments, methods and rationale should be reported and inter-rater agreement should be assessed. Confounding factors should be controlled for, and power analysis should be conducted to ensure statistical robustness. For value-laden or culturally contingent constructs, researchers should describe rater demographics beyond expertise and discuss potential demographic biases.

Rationale

Even with well-justified benchmarks and metrics (Benchmarks and Metrics ), automated measurement captures only the constructs the benchmark was designed to measure. Subjective constructs and constructs that no current benchmark covers require human assessment.

Recommendations

When LLMs automate research or software development tasks previously performed by humans, the LLM’s performance needs to be assessed. Where no benchmark adequately operationalizes the target construct, researchers should validate LLM outputs against human judgment.

Study Design Considerations:

Studies that include human participants need additional considerations, including a recruitment strategy, annotation guidelines, training sessions, or ethical approvals. Therefore, researchers should consider human validation early in the study design, not as an afterthought. Authors must clearly define in the paper the constructs that the human and LLM annotators evaluate (Ralph and Tempero 2018). When designing custom instruments to assess LLM output (e.g., questionnaires, scales), researchers must share their instruments.

Subjective Judgment and Agreement:

When the judgment is subjective (i.e., depends on the judge’s values or theories), the LLM output should be evaluated against judgments aggregated from multiple human judges, and researchers should clearly describe their aggregation method and reasoning. One systematic approach is to randomly order the objects requiring judgment and put them in groups small enough to judge in one to three hours. Two or three human experts review the first group together, simultaneously judging, discussing, and creating a set of decision rules documenting their reasoning. Then, the experts iterate among rounds of:

  1. independently rating one group of objects,

  2. calculating inter-rater agreement (IRA) or reliability (IRR),

  3. meeting to discuss disagreements and reach consensus,

  4. updating the decision rules so the disagreement will not recur.

The goal is to reach a target IRA or IRR (e.g., Krippendorff’s \(\alpha>0.8\)), indicating sufficient decision rules. Once this target is reached and sustained for two to three groups, it is permissible to continue with a single rater. Researchers should report measures of IRA or IRR, preferably broken down by round.

Confounding factors should be discussed and, where feasible, controlled for (e.g., by categorizing participants according to their level of experience or expertise). For value-laden or culturally contingent constructs (e.g., judging code-style appropriateness or comment helpfulness), researchers should describe rater demographics beyond expertise (e.g., geographic, linguistic, or professional background) and discuss potential demographic biases in rater recruitment and instructions (Bean et al. 2025). Where applicable, researchers may perform a power analysis (Cohen 1992; Dybå, Kampenes, and Sjøberg 2006) to estimate the required sample size, ensuring sufficient statistical power in their experimental design. However, we also note that to our knowledge, there is limited established guidance specific to determining sample sizes for LLM-human comparisons. In other fields, 100 comparisons are often made without further explanation (Tam et al. 2024).

Researchers should use established reference models to compare humans with LLMs. For example, Schneider, Fotrousi, and Wohlrab (2025) (Schneider, Fotrousi, and Wohlrab 2025) outline design considerations for studies comparing LLMs with humans. If studies aim to automate annotating software artifacts using an LLM, researchers should follow systematic approaches to decide whether and how human annotators can be replaced (e.g., showing that a jury of three LLMs exhibit model-to-model agreement comparable to trained human annotators, such as Krippendorff’s \(\alpha>0.8\)). Model-to-model agreement requires the same threshold as human-to-human agreement, since low thresholds may simply reflect shared model biases rather than annotation quality.

Agentic Tools:

Besides generating and modifying content, agentic software development tools such as Claude Code can autonomously call command-line tools or pull in additional information from MCP servers. For such tools, it makes sense to separate textual output (e.g., file changes) from output related to tool execution. When evaluating agentic tools, researchers should assess the feedback that users provided for proposed changes, report statistics on how frequently they accepted the content, and how they modified it. Agentic human-in-the-loop interaction is a built-in human validation, with larger degrees of freedom than traditional experiments that use LLMs directly.

Examples

T. Ahmed et al. (2025) (T. Ahmed et al. 2025) proposed a systematic method for deciding whether LLMs can replace human annotators on a given task, using model-to-model agreement as an initial screening criterion (with a threshold of \(\alpha>0.5\)) and model confidence for sample-level decisions. However, their threshold is well below the levels generally considered acceptable for inter-rater agreement. Krippendorff (2018) recommends discarding data with \(\alpha<0.667\), considers \(0.667 \leq \alpha < 0.8\) sufficient only for tentative conclusions, and requires \(\alpha \geq 0.8\) for reliable data (Krippendorff 2018). Researchers adopting similar screening approaches should apply thresholds consistent with these established standards. Notably, while three LLMs exhibited higher inter-model agreement than human annotators on some tasks, human-model agreement remained low on others. This illustrates that high model-model reliability does not guarantee alignment with human judgment, reinforcing the need for human validation before assuming that LLM annotations are valid. Moreover, a high model-model agreement could merely indicate that the models share systematic biases and hence reliably agree on the wrong answer.

Hymel and Johnson (2025) (Hymel and Johnson 2025) evaluated ChatGPT’s capability to generate requirements documents by comparing an LLM-generated and a human-generated document based on the same business use case. Domain experts reviewed both documents and attempted to distinguish their origin. For agentic tools, Takerngsaksiri et al. (2025) (Takerngsaksiri et al. 2025) report acceptance and modification rates from real users for HULA, an agentic plan-and-code system deployed in Atlassian JIRA.

Benefits

Validation against human judgments builds confidence in the LLM’s accuracy and validity, increasing the trustworthiness of findings, especially when IRA or IRR metrics are reported (Khraisha et al. 2024). Furthermore, incorporating feedback from human judges may help to improve the LLM or LLM-based tool based on the reported experiences.

Challenges

Assessment of a construct such as LLM performance is challenging because ensuring that a construct is defined well and operationalized using an appropriate measurement model requires a deep understanding of (1) the construct, (2) construct validity in general, and (3) instrumentation (Sjøberg and Bergersen 2023; Ralph et al. 2024). Comparing an LLM to human judges is typically slower and more expensive than machine-generated measures. More fundamentally, neither human judgment nor machine-generated measures provides an objective ground truth against which LLM accuracy can be firmly determined. Human validation is worth its added cost only when automated measures alone fail to capture the constructs the study evaluates.

Human judgments exhibit variability due to differences in experience, expertise, interpretations, and personal biases (McDonald, Schoenebeck, and Forte 2019). When diverse humans rate items reliably given clear decision rules, we assume that reliability implies validity, but it does not. Measuring reliability is much easier than measuring validity, and often the best researchers can do is argue conceptually for why their judges, decision rules, and constructs should produce valid ratings.

Study Types

This guideline applies to all study types, although the need for human validation varies. For LLMs as Annotators, researchers should validate LLM-generated annotations against human annotators to assess labeling quality and identify systematic biases. When using LLMs as Judges, researchers should co-create initial rating criteria with humans and validate a sample of LLM judgments against human expert assessments. For LLMs for Synthesis, researchers should employ human oversight to verify that qualitative interpretations and synthesized outputs faithfully represent the underlying data. For LLMs as Subjects, researchers should validate simulated responses against real human data to assess the fidelity of the simulation. For Studying LLM Usage, researchers should carefully reflect on the validity of their evaluation criteria and validate subjective assessments with human experts. For LLMs for Tools whose output is supposed to match human expectations, researchers should validate the LLM output against human judgment. For Benchmarking LLMs, there is less need for human validation when using extensively validated and widely-used benchmarks, but researchers should employ human validation when creating or adapting new benchmarks.

Unchanged text

Advice for Reviewers

Human validation may be the most challenging of our guidelines to assess because it often requires evaluating conceptual arguments. If LLM output is validated only by comparison with other LLMs, reviewers should look for quantitative empirical evidence that such comparison is reliable and valid. High inter-model agreement alone is insufficient, as reliability does not imply validity. Similarly, reviewers should expect evidence that any employed benchmarks are reliable and valid. Absent such evidence, human validation is warranted. A single human judge is appropriate only when judgments depend on widely accepted theories and involve limited value conflict (e.g., tagging method names containing abbreviations). For multiple judges, reviewers should expect IRA/IRR improvement techniques as described in the recommendations above (experienced raters, organized rounds, consensus meetings, updated decision rules). Low IRA or IRR (e.g., Krippendorff’s \(\alpha<0.8\)) without these techniques is a concern. Conversely, if authors have followed best practices and still obtained mediocre results (e.g., \(0.66<\alpha<0.8\)), this should be noted as a limitation. Beyond reliability, reviewers should expect authors to explain conceptually why their human judgments should be valid, considering construct definitions, decision rules, and judge expertise. As with other guidelines, missing information is typically a revision request. Absent judge instructions, instruments, decision rules, or construct definitions may prevent assessment of rigor and validity, while missing recruitment details are less critical. Clarification requests about construct definitions are routine and should not alone warrant rejection.

See Also

Report Limitations and Mitigations Changed

2026.04

Report Limitations and Mitigations (G8)

tl;dr: Researchers MUST transparently report study limitations, including the impact of non-determinism and generalizability constraints. The PAPER  MUST specify whether generalization across LLMs or across time was assessed, and discuss model and version differences. Authors MUST describe measurement constructs and methods, disclose any data leakage risks, and avoid leaking evaluation data into LLM improvement pipelines. For studies involving sensitive data, they MUST discuss data governance mechanisms. They SHOULD justify LLM usage in light of its resource demands. Mitigation strategies such as replication packages, human validation, longitudinal re-runs, triangulation, and sensitivity analysis SHOULD be employed and reported where applicable. Where full data sharing is not possible, a subset of the validation data SHOULD be included to enable partial replication.

Rationale

When using LLMs for empirical studies in SE, researchers face unique challenges and potential limitations that can influence the validity, reliability, and reproducibility of their findings (Sallou, Durieux, and Panichella 2024). Researchers must openly discuss these limitations and explain how their impact was mitigated. These limitations are relative to current LLM capabilities and tool architectures; speculating about future improvements is beyond the scope of a paper’s limitation section. Nevertheless, risk management and threat mitigation SHOULD be planned during study design, not as an afterthought.

Recommendations

Researchers MUST clearly present the limitations of their work without defensiveness or obfuscation. These limitations may concern diverse topics including generalizability, internal validity (e.g. data leakage), reliability (i.e. non-determinism), and reproducibility (e.g. resource requirements). When deterministic reproducibility is structurally unattainable (e.g., SaaS-based models with opaque versioning), researchers SHOULD adopt trustworthiness criteria from qualitative research to substantiate dependability and confirmability of findings. LLM-based studies may involve many kinds of generalization including the following:

  1. From tested LLMs to others; that is, do the performance characteristics of the LLM(s) studied generalize to LLM(s) not included in the study.

  2. From tested configurations to others; that is, how sensitive are the results to the specific configuration(s) of the LLM under test.

  3. From research to practice; for instance, just because researchers can get a certain level of code generation performance under ideal conditions does not mean software developers, given the same tools, will get similar results without having the researchers present to show them exactly what to do.

  4. From a sample of people (e.g. human validators; human participants) to a larger population of people.

  5. From one time period to another; i.e., does the same LLM, or other versions of the same LLM, produce similar results at a different point in time?

The following concerns present an overview that has to be tailored to the individual study context. This section does not repeat requirements from other recommendations.

Internal Validity:

The primary threats to internal validity are:

  • Data leakage and contamination, including inter-dataset duplication, potentially resulting in a training–evaluation overlaps, leading to overly optimistic evaluation results.

  • Unintended inclusion of evaluation data in model-improvement pipelines, contributing to data leakage. This is especially relevant for longitudinal studies including LLMs.

  • Incomplete architecture, prompt, or pipeline reporting, posing hidden confounding factors.

Inter-dataset duplication is prevalent in SE, particularly for code-related benchmarks. As transparency on training data is limited for LLMs, researchers MUST discuss potential data leakage effects and their impact on results.

Construct Validity:

The primary threats to construct validity are:

  • Metric–construct mismatch (e.g., BLEU/ROUGE vs. functional correctness). More traditional metrics might not capture all relevant SE-specific aspects.

  • Over-reliance on benchmark-specific metrics. Optimizing for a single benchmark might result in dataset-specific heuristics rather than the intended construct, overstating real-world utility.

  • Benchmark scope limitations. Benchmarks commonly ignore runtime behaviors, security implications, readability, testability, and maintainability, yielding results that may not transfer to realistic development settings.

If constructs are based on subjective interpretations, purely automated metrics are insufficient. Researchers MUST discuss how they ensured quality of subjective results, similarly to qualitative research.

External Validity:

The primary threats to external validity are:

  • Cross-model transfer limitations. Results obtained with one LLM (or family of LLMs) may not generalize to others due to differences in training data, architecture, and capabilities.

  • Tool-architecture specificity. Tools built around a specific LLM’s API or capabilities may not transfer to other models without significant re-engineering.

  • Limited domain coverage. Studies often focus on a narrow set of programming languages, task types, or application domains, limiting generalizability to other SE contexts.

  • Limited participant diversity. Study participants (e.g., human validators, developers) may not represent the broader population in terms of expertise, geographic location, or cultural background.

  • Cross-time instability (model evolution). Performance of proprietary models can change over time, leading to non-generalizable study outcomes (L. Chen, Zaharia, and Zou 2024; D. Li et al. 2024).

Generalizability is particularly critical for proprietary and non-deterministic systems whose behavior is subject to drift (i.e., silent changes in model output over time) (L. Chen, Zaharia, and Zou 2024). Researchers MUST discuss the limitations and mitigations of external validity.

Reliability & Reproducibility:

The primary threats to reliability and reproducibility are:

  • Non-deterministic outputs. Identical prompts and configurations can yield different outputs across runs due to factors such as floating-point arithmetic, batching, and stochastic decoding strategies.

  • Infrastructure dependence. Results may vary depending on the hardware, software stack, and hosting environment used, making exact replication challenging across different infrastructure setups.

  • Resource inequality preventing replication. LLM research is resource-intensive and hence remains predominantly in the domain of private companies or well-funded research institutions (Schwartz et al. 2020; N. Ahmed, Wahed, and Thompson 2023), excluding researchers from under-resourced institutions.

Researchers MUST discuss the measures taken to increase reliability and reproducibility. However, non-deterministic reproducibility is not inherently disqualifying. Qualitative traditions such as ethnography, grounded theory, and action research ensure trustworthiness through credibility, transferability, dependability, and confirmability (Guba 1981). The same applies to SaaS-based LLM research, where providers frequently deprecate model versions without guaranteeing stable behavior.

Ethical & Regulatory Boundaries:

The primary concerns for ethical and regulatory matters are:

  • Use of sensitive or proprietary data. Studies involving proprietary code, confidential business data, or personally identifiable information may face restrictions on data sharing that limit reproducibility.

  • Jurisdictional obligations. Data protection regulations (e.g., GDPR, CCPA) and institutional policies may impose constraints on data collection, processing, and sharing in LLM-based studies.

  • Implicit model bias. Especially for qualitative research LLM might “reinforce dominant paradigms and biases” and “identify, replicate and reinforce dominant language and patterns” (Jowsey et al. 2025).

Studies involving sensitive data MUST discuss data governance mechanisms tailored towards LLM environments, compliant with applicable juristical obligations. If applicable, researchers SHOULD discuss how model biases potentially impact the study outcomes and how those biases were evaluated.

Environmental & Sustainability Constraints:

  • Energy consumption. With growing model size, the environmental impact of experiments with LLMs increases. Considering the environmental impact in the study design is important given the substantial energy costs of LLM experiments (Strubell, Ganesh, and McCallum 2019).

  • Trade-off between repetition and sustainability. Repeating experiments increases reliability but also energy consumption, requiring trade-offs during study design.

Researchers SHOULD justify the LLM’s resource consumption against the benefits over traditional approaches.

Mitigation Strategies:

The following mitigation strategies can help address the threats described above, depending on the study scope and feasibility.

  • Replication Packages covering prompt and architecture specifications, model outputs, and representative examples for partial replicability, accompanied by an implementation using an open model for long-term stability.

  • Human Validation of subjective constructs, following quality criteria known from qualitative research.

  • Longitudinal Re-Runs to repeat experiments with LLMs over time, complemented by statistical analyses.

  • Methodological Trustworthiness Measures. Researchers SHOULD consider triangulation, reflexivity, audit trails, and peer debriefing as complementary measures when deterministic reproduction is structurally impossible.

  • Triangulation via multiple models (e.g., proprietary and open models), multiple independent datasets, and multiple complementary metrics.

  • Cost Accounting through reporting of input/output tokens, token and service costs, or hardware specifications.

  • Energy Preservation by selecting smaller or newer less resource-intensive models and employing techniques such as input/output token reduction, model pruning, quantization, or knowledge distillation (Mitu and Mitu 2024) where feasible. Carbon footprint estimation is desirable, but still difficult.

  • Ethical and Regulatory Considerations through data governance mechanisms, ethical reviews, and bias assessment procedures.

  • Sensitivity Analysis through variation of LLM configurations, prompts, architecture decisions, datasets, and if applicable human participant backgrounds.

Study Types

Researchers MUST follow this guideline for all study types. Transparently reporting limitations and mitigations is a universal requirement, but specific concerns vary by study type. For LLMs as Annotators S1: LLMs as Annotators , researchers MUST discuss potential biases in label assignment, label reliability limitations, and sensitivity of annotations to prompt wording and model choice. For LLMs as Judges S2: LLMs as Judges , researchers MUST address measurement validity concerns, known biases such as position bias or verbosity bias, and the extent to which LLM judgments align with human expert assessments. For LLMs for Synthesis S3: LLMs for Synthesis , researchers MUST discuss the risk of contextual misinterpretation, potential loss of nuance in summarized or aggregated outputs, and reflexivity limitations inherent in using an LLM for qualitative interpretation. For LLMs as Subjects S4: LLMs as Subjects , researchers MUST discuss the fundamental inability of LLMs to truly simulate human behavior, the risk of stereotype amplification, and the limited ecological validity of simulated responses. For Studying LLM Usage S5: Studying LLM Usage , researchers MUST discuss generalizability constraints across different tools and user populations, and acknowledge how observed usage patterns may not transfer to other contexts. For LLMs for Tools S6: LLMs for Tools , researchers MUST discuss replicability constraints arising from dependencies on commercial models, the impact of model updates on tool behavior, and limitations of the evaluation setup. For Benchmarking LLMs S7: Benchmarking LLMs , researchers MUST discuss potential data contamination, benchmark scope limitations, and the extent to which benchmark performance generalizes to real-world tasks.

Advice for Reviewers

Reviewers should verify that the limitation section is comprehensive and appropriate for the specific study type, checking that: (1) limitations address the specific threats relevant to the study type (e.g., label reliability for annotation studies, simulation fidelity for studies using LLMs as subjects); (2) mitigations are concrete and correspond to identified limitations rather than being generic statements; (3) the impact of LLM non-determinism on findings is discussed; (4) generalizability constraints—across models, configurations, time periods, and populations—are acknowledged. When important limitations are missing, reviewers should request they be added. The absence of a limitation section, or one that is formulaic or insufficiently specific, is a more serious concern than any individual missing limitation and may warrant a major revision.

See Also

Rationale and key recommendations for each guideline. ●  = MUST, ○  = SHOULD.
ID Rationale Core Recommendations
G1 Transparency enables informed assessment of scope and limitations. ● Declare which LLM, how it was used, and where in the research process.
G2 Reproducibility requires precise identification of the system used in a study. ● Report exact version, date, configuration, and fine-tuning details.
G3 Static artifacts, from system architecture to prompt templates, determine what the model sees on every call and must be documented in full. ● Describe full architecture, infrastructure, all prompts or prompt templates (with development strategy and reuse), context files, tool catalog, agent architecture, and retrieval mechanisms.
○ For studies of LLM usage (S5), describe the tool architecture only to the extent accessible.
G4 Runtime traces make LLM and agent behavior verifiable despite non-determinism and commercial-tool opacity. ● For studies of LLM usage (S5), share full interaction logs subject to privacy constraints.
○ Otherwise, share interaction logs, runtime traces, and plans where feasible.
G5 Meaningful evaluation requires reasoned valid measurement. ● Justify metric and benchmark choices; discuss their validity.
○ Repeat experiments and report descriptive statistics.
G6 Reproducibility depends on access to the model under study. ● For benchmarking studies (S7), include an open LLM among the models evaluated.
○ Otherwise, include an open LLM as a baseline; provide a replication package.
G7 Automated metrics alone cannot ensure validity of subjective constructs. ● Define the measured construct and share any custom measurement instruments.
○ Validate against human judgment with inter-rater reliability, where applicable.
G8 Honest acknowledgment of threats strengthens a study. ● Report threats by validity type.
○ Employ mitigations where possible.

2026.05

Report Limitations and Mitigations

Summary: Researchers must transparently report study limitations, including the impact of non-determinism and generalizability constraints. The paper  must specify whether generalization across LLMs or across time was assessed, and discuss model and version differences. Authors must describe measurement constructs and methods, disclose any data leakage risks, and avoid leaking evaluation data into LLM improvement pipelines. For studies involving sensitive data, they must discuss data governance mechanisms. They should justify LLM usage in light of its resource demands. Mitigation strategies such as replication packages, human validation, longitudinal re-runs, triangulation, and sensitivity analysis should be employed and reported where applicable. Where full data sharing is not possible, a subset of the validation data should be included to enable partial replication.

Rationale

When using LLMs for empirical studies in SE, researchers face unique challenges and potential limitations that can influence the validity, reliability, and reproducibility of their findings (Sallou, Durieux, and Panichella 2024). Researchers must openly discuss these limitations and explain how their impact was mitigated. These limitations are relative to current LLM capabilities and tool architectures; speculating about future improvements is beyond the scope of a paper’s limitation section. Nevertheless, risk management and threat mitigation should be planned during study design, not as an afterthought.

Recommendations

Researchers must clearly present the limitations of their work without defensiveness or obfuscation. These limitations may concern diverse topics including generalizability, internal validity (e.g. data leakage), reliability (i.e. non-determinism), and reproducibility (e.g. resource requirements). When deterministic reproducibility is structurally unattainable (e.g., SaaS-based models with opaque versioning), researchers should adopt trustworthiness criteria from qualitative research to substantiate dependability and confirmability of findings. LLM-based studies may involve many kinds of generalization including the following:

  1. From tested LLMs to others; i.e., whether the performance characteristics of the LLM(s) studied transfer to LLM(s) not included in the study.

  2. From tested configurations to others; i.e., how sensitive results are to the specific configuration(s) of the LLM under test.

  3. From research to practice; i.e., whether developers using the same tools obtain results comparable to those reported under researcher-supervised conditions.

  4. From a sampled population to a larger one; i.e., whether findings from a specific group of human validators or participants apply to a broader population.

  5. From one time period to another; i.e., whether the same LLM, or other versions of it, produce similar results at a different point in time.

The following concerns present an overview that has to be tailored to the individual study context. This section does not repeat requirements from other recommendations.

Internal Validity:

The primary threats to internal validity are:

  • Data leakage and contamination. Inter-dataset duplication can produce training-evaluation overlap, yielding overly optimistic results.

  • Evaluation data entering model-improvement pipelines. Evaluation samples can unintentionally feed retraining or fine-tuning, especially in longitudinal studies involving LLMs.

  • Incomplete architecture, prompt, or pipeline reporting. Undisclosed components introduce hidden confounders.

Inter-dataset duplication is prevalent in SE, particularly for code-related benchmarks. As transparency on training data is limited for LLMs, researchers must discuss potential data leakage effects and their impact on results.

Construct Validity:

The primary threats to construct validity are:

  • Metric-construct mismatch. Traditional metrics such as BLEU or ROUGE may miss SE-specific aspects such as functional correctness or behavioral equivalence.

  • Over-reliance on benchmark-specific metrics. Optimizing for a single benchmark may produce dataset-specific shortcuts that pass the benchmark without exhibiting the capability it was designed to test, overstating real-world utility.

  • Benchmark scope limitations. Benchmarks commonly ignore runtime behaviors, security implications, readability, testability, and maintainability, yielding results that may not transfer to realistic development settings.

If constructs are based on subjective interpretations, purely automated metrics are insufficient. Researchers must discuss how they ensured quality of subjective results, similarly to qualitative research.

External Validity:

The primary threats to external validity are:

  • Cross-model transfer limitations. Results obtained with one LLM or family of LLMs may not generalize to others due to differences in training data, architecture, and post-training procedures (e.g., fine-tuning and RLHF).

  • Tool-architecture specificity. Tools built around vendor-specific APIs or features (e.g., function calling, structured output, or context-window size) may not transfer to other models without substantial re-engineering.

  • Limited domain coverage. Studies often focus on a narrow set of programming languages, task types, or application domains, limiting generalizability to other SE contexts.

  • Limited participant diversity. Study participants such as human validators or developers may not represent the broader population in terms of expertise, geographic location, or cultural background.

  • Cross-time instability. Performance of proprietary models can change over time, leading to non-generalizable study outcomes (L. Chen, Zaharia, and Zou 2024; D. Li et al. 2024).

Generalizability is particularly critical for proprietary and non-deterministic systems whose behavior is subject to drift (i.e., silent changes in model output over time) (L. Chen, Zaharia, and Zou 2024). Researchers must discuss the limitations and mitigations of external validity.

Reliability & Reproducibility:

The primary threats to reliability and reproducibility are:

  • Non-deterministic outputs. Identical prompts and configurations can yield different outputs across runs due to factors such as floating-point arithmetic, batching, and stochastic decoding strategies.

  • Infrastructure dependence. Results may vary depending on the hardware, software stack, and hosting environment used; vendor-imposed quotas, throttling, or pricing changes can further prevent re-running experiments at the original scale, making exact replication challenging across different infrastructure setups.

  • Resource inequality. LLM research is resource-intensive and remains predominantly in the domain of private companies or well-funded research institutions (Schwartz et al. 2020; N. Ahmed, Wahed, and Thompson 2023), excluding researchers from under-resourced institutions.

Researchers must discuss the measures taken to increase reliability and reproducibility. However, non-deterministic reproducibility is not inherently disqualifying. Qualitative traditions such as ethnography, grounded theory, and action research ensure trustworthiness through credibility, transferability, dependability, and confirmability (Guba 1981). The same applies to SaaS-based LLM research, where providers frequently deprecate model versions without guaranteeing stable behavior.

Ethical & Regulatory Boundaries:

The primary concerns for ethical and regulatory matters are:

  • Use of sensitive or proprietary data. Studies involving proprietary code, confidential business data, or personally identifiable information may face restrictions on data sharing that limit reproducibility.

  • Jurisdictional obligations. Data protection regulations such as GDPR or CCPA and institutional policies may impose constraints on data collection, processing, and sharing in LLM-based studies.

  • Implicit model bias. Especially for qualitative research, LLMs might “reinforce dominant paradigms and biases” and “identify, replicate and reinforce dominant language and patterns” (Jowsey et al. 2025).

Studies involving sensitive data must discuss data governance mechanisms tailored towards LLM environments, compliant with applicable juristical obligations. If applicable, researchers should discuss how model biases potentially impact the study outcomes and how those biases were evaluated.

Environmental & Sustainability Constraints:

  • Energy consumption. With growing model size, the environmental impact of experiments with LLMs increases, and the substantial energy costs of LLM experiments warrant consideration in study design (Strubell, Ganesh, and McCallum 2019).

  • Trade-off between repetition and sustainability. Repeating experiments increases reliability but also energy consumption, requiring trade-offs during study design.

Researchers should justify the LLM’s resource consumption against the benefits over traditional approaches.

Mitigation Strategies:

The following mitigation strategies can help address the threats described above, depending on the study scope and feasibility.

  • Replication Packages. Cover prompt and architecture specifications, model outputs, and representative examples for partial replicability, accompanied by an implementation using an open model for long-term stability.

  • Human Validation. Validate subjective constructs following quality criteria known from qualitative research.

  • Longitudinal Re-Runs. Repeat experiments with LLMs over time, complemented by statistical analyses.

  • Methodological Trustworthiness Measures. Researchers should consider triangulation, reflexivity, audit trails, and peer debriefing as complementary measures when deterministic reproduction is structurally impossible.

  • Triangulation. Use multiple models (e.g., proprietary and open), multiple independent datasets, and multiple complementary metrics.

  • Cost Accounting. Report input and output tokens, token and service costs, or hardware specifications.

  • Energy Preservation. Select smaller or newer less resource-intensive models and apply techniques such as input/output token reduction, model pruning, quantization, or knowledge distillation (Mitu and Mitu 2024) where feasible. Carbon footprint estimation is desirable, but still difficult.

  • Ethical and Regulatory Considerations. Apply data governance mechanisms, ethical reviews, and bias assessment procedures.

  • Sensitivity Analysis. Vary LLM configurations, prompts, architecture decisions, datasets, and where applicable human participant backgrounds.

Examples

Sallou, Durieux, and Panichella (2024) (Sallou, Durieux, and Panichella 2024) catalog three categories of LLM-specific threats to validity (i.e., closed-source models, implicit data leakage, and reproducibility) and pair each with concrete mitigation strategies (e.g., versioned model archives, metamorphic test data, multiple replication runs with variability metrics, and detailed execution metadata). Du et al. (2024) (Du et al. 2024) pair each threat in their ClassEval evaluation with a concrete mitigation: manually constructing the benchmark with multiple annotators to limit data leakage, piloting prompts on held-out tasks to control for prompt sensitivity, and reporting greedy-decoding results to control for non-determinism.

Benefits

Transparent reporting of limitations and mitigations helps readers calibrate confidence in the findings, makes explicit which threats were addressed and which remain open, and documents design decisions that other authors can borrow or refine. Transparent threat discussion also keeps a paper’s claims proportionate to its evidence.

Challenges

Identifying limitations one is not already aware of is the hardest part of writing a threats section, particularly for methodological threats outside the team’s primary expertise. Publication and reviewing norms can pressure authors to downplay weaknesses, while page limits make exhaustive treatment impractical. Threat lists that recite generic LLM-research issues (e.g., model bias, non-determinism, or contamination) without showing how each one applies to specific design choices in this study leave reviewers unable to tell which risks actually applied.

Study Types

Researchers must follow this guideline for all study types. Transparently reporting limitations and mitigations is a universal requirement, but specific concerns vary by study type. For LLMs as Annotators, researchers must discuss potential biases in label assignment, label reliability limitations, and sensitivity of annotations to prompt wording and model choice. For LLMs as Judges, researchers must address measurement validity concerns, known biases such as position bias or verbosity bias, and the extent to which LLM judgments align with human expert assessments. For LLMs for Synthesis, researchers must discuss the risk of contextual misinterpretation, potential loss of nuance in summarized or aggregated outputs, and reflexivity limitations inherent in using an LLM for qualitative interpretation. For LLMs as Subjects, researchers must discuss the fundamental inability of LLMs to truly simulate human behavior, the risk of stereotype amplification, and the limited ecological validity of simulated responses. For Studying LLM Usage, researchers must discuss generalizability constraints across different tools and user populations, and acknowledge how observed usage patterns may not transfer to other contexts. For LLMs for Tools, researchers must discuss replicability constraints arising from dependencies on commercial models, the impact of model updates on tool behavior, and limitations of the evaluation setup. For Benchmarking LLMs, researchers must discuss potential data contamination, benchmark scope limitations, and the extent to which benchmark performance generalizes to real-world tasks.

Advice for Reviewers

Reviewers should verify that the limitation section is comprehensive and appropriate for the specific study type, checking that: (1) limitations address the specific threats relevant to the study type (e.g., label reliability for annotation studies, simulation fidelity for studies using LLMs as subjects); (2) mitigations are concrete and correspond to identified limitations rather than being generic statements; (3) the impact of LLM non-determinism on findings is discussed; (4) generalizability constraints, across models, configurations, time periods, and populations, are acknowledged. When important limitations are missing, reviewers should request they be added. The absence of a limitation section, or one that is formulaic or insufficiently specific, is a more serious concern than any individual missing limitation and may warrant a major revision.

See Also

Conclusion Changed

2026.04

Conclusion

LLM non-determinism, opaque training data, and rapidly evolving models threaten the reproducibility and replicability of empirical SE studies. This paper presents a taxonomy of seven study types that organizes how LLMs are used in SE research, together with eight guidelines for designing and reporting such studies. Each guideline distinguishes requirements (MUST ) from recommended practices (SHOULD ) and is contextualized by the study types it applies to. Our guidelines recommend that researchers:

  1. declare LLM usage and role;
  2. report model versions, configurations, and customizations;
  3. document the tool architecture beyond the model;
  4. disclose prompts, their development, and, where feasible, interaction logs;
  5. validate LLM outputs with humans;
  6. include an open LLM as a baseline;
  7. use suitable baselines, benchmarks, and metrics; and
  8. articulate limitations and mitigations, including costs, potential biases, and environmental impact.

An applicability matrix maps guidelines to study types, and a reporting checklist (Section 7) supports authors and reviewers. The study types and guidelines were derived collaboratively by 22 co-authors, following a process similar to that of guidelines in other disciplines (see, e.g., (Begg et al. 1996)).

Following the quality dimensions of the SIGSOFT Empirical Standards (Ralph et al. 2021), we consider our guidelines (1) comprehensive, as they cover seven study types that we identified and were derived from 22 co-authors’ collective expertise; (2) useful, as they provide actionable recommendations with an applicability matrix and a concrete reporting checklist (Section 7); (3) well-argued, as each guideline is motivated by a dedicated rationale and supported by references to the literature; and (4) integrated with previous work, as they build on and complement established empirical SE standards while addressing LLM-specific challenges. Adopting these practices will not eliminate LLM non-determinism, but it should make claims auditable and results easier to replicate and compare across studies. Because models and tools evolve, we maintain the guidelines as a living resource and invite the community to refine and extend them at llm-guidelines.org.

2026.05

Conclusion

LLM non-determinism, opaque training data, and rapidly evolving models threaten the reproducibility and replicability of empirical SE studies. This paper presents a taxonomy of seven study types that organizes how LLMs are used in SE research, together with eight guidelines for designing and reporting such studies. Each guideline distinguishes requirements (must ) from recommended practices (should ) and is contextualized by the study types it applies to. Our guidelines recommend that researchers:

  1. declare LLM usage and role;
  2. report model versions, configurations, and customizations;
  3. document the tool architecture beyond the model;
  4. disclose prompts, their development, and, where feasible, interaction logs;
  5. validate LLM outputs with humans;
  6. include an open LLM as a baseline;
  7. use suitable baselines, benchmarks, and metrics; and
  8. articulate limitations and mitigations, including costs, potential biases, and environmental impact.

An applicability matrix maps guidelines to study types, and a reporting checklist (Section 9) supports authors and reviewers. The study types and guidelines were derived collaboratively by 22 co-authors, following a process similar to that of guidelines in other disciplines (see, e.g., (Begg et al. 1996)).

Following the quality dimensions of the SIGSOFT Empirical Standards (Ralph et al. 2021), we consider our guidelines (1) comprehensive, as they cover seven study types that we identified and were derived from 22 co-authors’ collective expertise; (2) useful, as they provide actionable recommendations with an applicability matrix and a concrete reporting checklist (Section 9); (3) well-argued, as each guideline is motivated by a dedicated rationale and supported by references to the literature; and (4) integrated with previous work, as they build on and complement established empirical SE standards while addressing LLM-specific challenges. Adopting these practices will not eliminate LLM non-determinism, but it should make claims auditable and results easier to replicate and compare across studies. Because models and tools evolve, we maintain the guidelines as a living resource and invite the community to refine and extend them at llm-guidelines.org.

Applicability Matrix Added section

2026.04

Section absent in this release.

2026.05

Applicability Matrix

Declare Usage ●  ●  ●  ●  ●  ●  ● 
Model Version ●  ●  ●  ●  ●  ●  ● 
Design ●  ●  ●  ●  ○  ●  ● 
Traces ○  ○  ○  ○  ●  ○  ○ 
Benchmarks ●  ○  ○  ○  ○  ○  ● 
Open LLM ○  ○  ○  – – ○  ● 
Human Validation ○  ○  ○  ○  ○  ○  ○ 
Limitations ●  ●  ●  ●  ●  ●  ● 
Rationale and Recommendations Added section

2026.04

Section absent in this release.

2026.05

Rationale and Recommendations

Guideline Rationale Core Recommendations
Declare Usage Transparency enables informed assessment of scope and limitations. ● Declare which LLM, how it was used, and where in the research process.
Model Version Reproducibility requires precise identification of the system used in a study. ● Report exact version, date, configuration, and fine-tuning details.
○ Report defaults, checksums, and quantization; motivate model choice; acknowledge commercial-model reproducibility limits.
Design Static artifacts determine what the model sees on every call and must be documented in full. ● Describe architecture, infrastructure, prompts and templates (with development strategy), context files, tool catalog, agent architecture, and retrieval mechanisms.
○ For LLM usage, describe the tool architecture to the extent accessible.
Traces Runtime traces make LLM and agent behavior verifiable despite non-determinism and tool opacity. ● For studies of LLM usage, share full interaction logs subject to privacy constraints.
○ Otherwise, share interaction logs, runtime traces, and plans where feasible.
Benchmarks Meaningful evaluation requires reasoned valid measurement. ● Justify metric and benchmark choices; discuss their validity.
○ Define the phenomenon and sampling strategy; isolate confounders; analyze errors; repeat experiments and report distributions.
Open LLM Reproducibility depends on access to the model under study. ● For benchmarking studies, include an open LLM.
○ Otherwise, include an open LLM as a baseline; provide a replication package.
Human Validation Automated metrics alone cannot ensure validity of subjective constructs. ● Define the measured construct and share any custom measurement instruments.
○ Validate against human judgment with inter-rater reliability; describe rater demographics for value-laden constructs.
Limitations Honest acknowledgment of threats strengthens a study. ● Discuss threats to internal validity (data leakage), reliability (non-determinism), construct and external validity.
○ Employ mitigations where possible.
Reporting Checklist Changed

2026.04

Reporting Checklist

The following checklist, inspired by CONSORT (Schulz, Altman, and Moher 2010), summarizes actionable items from the guidelines based on the tl;dr sections. The checklist is organized along typical paper sections. Items marked ● are requirements (MUST ); items marked ○ are recommendations (SHOULD ). Each item references its source guideline (G1–G8). Items annotated with PAPER or SUPPLEMENTARY MATERIAL indicate where information should be reported; unmarked items may be reported in either.

Introduction

  • ● Disclose any use of LLMs in the empirical study, specifying which LLM, how, and where it was used (G1).

  • ○ Report the purpose of using LLMs, automated tasks, and expected benefits in the PAPER  (G1).

Research Design and Methods


Model Selection and Configuration

  • ● Report the exact LLM model or tool version, configuration, and experiment date in the PAPER  (G2).

  • ● For fine-tuned models, describe the fine-tuning goal, dataset, and procedure in the PAPER  (G2).

  • ○ Report default parameters and explain model and version choices (G2).

  • ○ For quantized models, report the quantization level (e.g., 4-bit, 8-bit) and method (e.g., GPTQ or AWQ) (G2).

  • ○ Compare base and fine-tuned models using suitable metrics and benchmarks; share fine-tuning data and weights as SUPPLEMENTARY MATERIAL (or justify in the PAPER why they cannot be shared) (G2).

System and Prompt Design

  • ● Describe the full architecture of LLM-based tools in the PAPER, including the role of the LLM, interactions with other components, and overall system behavior (G3).

  • ● Describe any context-file mechanisms used (e.g., AGENTS.md, CLAUDE.md) in the PAPER  (G3).

  • ● Summarize which tools and skills were exposed to the model in the PAPER  (G3).

  • ● Specify whether zero-shot, one-shot, or few-shot prompting was used in the PAPER  (G3).

  • ● If autonomous agents are used, specify agent roles, reasoning frameworks, and communication flows (G3).

  • ● Report hosting, hardware setup, and latency implications (G3).

  • ● Specify prompt reuse across models and configurations (G3).

  • ● For retrieval-augmented generation (RAG) or related methods, describe how external data was retrieved, stored, and integrated (G3).

  • ● For dynamically generated prompts, document the generation process thoroughly (G3).

  • ● Publish all prompts or, when using templates, prompt templates with representative instances, including their structure, content, formatting, and dynamic components, as SUPPLEMENTARY MATERIAL  (G3).

  • ○ For ensemble architectures, explain the coordination logic between models in the PAPER  (G3).

  • ○ Justify design decisions (G3).

  • ○ Where legally possible, release the source code of the implementation under an open-source license (G3).

  • ○ Report data preprocessing, versioning, and update frequency for stored data used for context augmentation (G3).

  • ○ Describe prompt development rationale and selection process (G3).

  • ○ Document input handling and token optimization strategies when prompts are long or complex (G3).

  • ○ For user-authored prompts, describe how they were collected and analyzed (G3).

  • ○ If full prompt disclosure is not feasible, provide summaries or examples (G3).

  • ○ Report prompt evolution and any LLM-suggested refinements (G3).

  • ○ Include context-file contents as SUPPLEMENTARY MATERIAL  (G3).

  • ○ Include the complete list of tools and skills (names with one-line purposes), tool schemas, skill definitions, sub-agent definitions, and connected MCP servers as SUPPLEMENTARY MATERIAL  (G3).

Session Traces

  • ○ Include full interaction logs (prompts and responses) as SUPPLEMENTARY MATERIAL if privacy and confidentiality can be ensured (G4).

  • ○ For agentic systems, include interaction logs covering exchanges between humans and the agent and between external tools and the agent (human-in-the-loop feedback, approvals, refinements) as SUPPLEMENTARY MATERIAL  (G4).

  • ○ For agentic systems, report the complete runtime trace as SUPPLEMENTARY MATERIAL, including for each entry the tool or artifact name, arguments, result, and ordering, and which configured artifacts (skills, context files, sub-agents) were activated (G4).

  • ○ For agentic systems, report developed plans as SUPPLEMENTARY MATERIAL if available (G4).

Human Validation

  • ● If using human validation, define the measured construct (e.g., usability, maintainability) and describe the measurement instrument in the PAPER  (G7).

  • ● When developing or adapting measurement instruments, share them (G7).

  • ○ Consider human validation early in the study design and build on established reference models for human-LLM comparison (G7).

  • ○ Validate LLM judgments against human judgment, report aggregation methods, and assess human-LLM agreement (G7).

  • ○ Discuss and, where feasible, control for confounding factors (G7).

Benchmarks and Metrics

  • ● Justify all benchmark and metric choices in the PAPER  (G5).

  • ● Explain in the PAPER why the selected metrics are suitable for the specific study (G5).

  • ○ Include an open LLM as a baseline when using commercial models and report inter-model agreement (G6).

  • ○ Summarize benchmark structure, task types, and limitations (G5).

  • ○ Justify the number of experiment repetitions, for example through a power analysis or by monitoring convergence of descriptive statistics (G5).

Reproducibility, Ethics, and Resources

  • ● For studies involving sensitive data, discuss data governance mechanisms compliant with applicable jurisdictional obligations (G8).

  • ○ Justify LLM usage in light of its resource demands (G8).

  • ○ Where full sharing of prompts, traces, or datasets is not feasible, share representative examples for partial replicability (G8).

  • ○ Provide a full replication package with step-by-step instructions as SUPPLEMENTARY MATERIAL  (G6).

Results

  • ● If comparing models or tools, use appropriate inferential statistics (e.g., hypothesis tests, effect sizes) rather than relying solely on summary statistics (G5).

  • ○ Repeat experiments due to the inherent non-determinism of LLMs and report the result distribution using descriptive statistics (G5).

  • ○ Use traditional (non-LLM) baselines for comparison where possible (G5).

  • ○ Report established metrics to make study results comparable; additional metrics may be reported where appropriate (G5).

Limitations and Threats to Validity

  • ● Describe measurement constructs and methods; disclose any data leakage risks and avoid leaking evaluation data into LLM improvement pipelines (G8).

  • ● Acknowledge non-disclosed confidential or proprietary components as reproducibility limitations (G3).

  • ● Transparently report study limitations, including the impact of non-determinism and generalizability constraints (G8).

  • ● Specify whether generalization across LLMs or across time was assessed, and discuss model and version differences (G8).

  • ○ Employ and report strategies to mitigate identified validity and reproducibility threats, such as replication packages, human validation, longitudinal re-runs, triangulation, and sensitivity analysis (G8).

2026.05

Reporting Checklist

The following checklist, inspired by CONSORT (Schulz, Altman, and Moher 2010), summarizes actionable items from the guidelines based on the summary sections. The checklist is organized along typical paper sections. Items marked ● are requirements (must ), and items marked ○ are recommendations (should ). Each item references its source guideline by short name. Items annotated with paper or supplementary material indicate where we expect the information to be reported. Unmarked items may be reported either in the paper or as supplementary material. Items prefixed with a bracketed tag apply only to studies with that characteristic (e.g., [fine-tuning], [agents]). Readers can hover each tag to read its description and use the filter panel to hide items that do not apply to their study. Beyond these characteristic-tagged items, each guideline’s Study Types subsection lists study-type-specific recommendations where applicable.

Introduction

  • ● Disclose any use of LLMs in the empirical study, specifying which LLM, how, and where it was used (Declare Usage).

  • ○ Report in the paper the purpose of using LLMs, the tasks they automate, and the expected benefits (Declare Usage).

Research Design and Methods


Model Selection and Configuration

  • ● Report in the paper the exact LLM model or tool version, the configuration, and the experiment date (Model Version).

  • ● [fine-tuning] For fine-tuned models, describe in the paper the fine-tuning goal, the dataset, and the procedure (Model Version).

  • ○ Report default parameters and explain model and version choices (Model Version).

  • ○ Report checksums and additional model properties where available; for commercial tools, openly acknowledge their reproducibility limits (Model Version).

  • ○ [quantization] For quantized models, report the quantization level (e.g., 4-bit, 8-bit) and method (e.g., GPTQ or AWQ) (Model Version).

  • ○ [fine-tuning] Compare base and fine-tuned models using suitable metrics and benchmarks; share fine-tuning data and weights as supplementary material (or justify in the paper why they cannot be shared) (Model Version).

  • ○ [commercial-models] Include an open LLM as a baseline when using commercial models and report inter-model agreement (Open LLM).

System and Prompt Design

  • ● Describe in the paper the full architecture of LLM-based tools, including the role of the LLM, interactions with other components, and overall system behavior (Design).

  • ● Specify in the paper whether zero-shot, one-shot, or few-shot prompting was used (Design).

  • ● Specify prompt reuse across models and configurations (Design).

  • ● Publish all prompts or, when using templates, prompt templates with representative instances, including their structure, content, formatting, and variable components, as supplementary material  (Design).

  • ● Report hosting, hardware setup, and latency implications (Design).

  • ● [dynamic-prompts] For dynamically generated prompts, document the generation process thoroughly (Design).

  • ● [context-files] Describe in the paper any context-file mechanisms used (e.g., AGENTS.md, CLAUDE.md) (Design).

  • ● [tool-use] Summarize in the paper which tools and skills were exposed to the model (Design).

  • ● [agents] If autonomous agents are used, specify agent roles, reasoning frameworks, and communication flows (Design).

  • ● [context-augmentation] For retrieval-augmented generation (RAG) or related methods, describe how external data was retrieved, stored, and integrated (Design).

  • ○ Justify design decisions (Design).

  • ○ Describe prompt development rationale and selection process (Design).

  • ○ Report prompt evolution and any LLM-suggested refinements (Design).

  • ○ Where legally possible, release the source code of the implementation under an open-source license (Design).

  • ○ [participant-prompts] For user-authored prompts, describe how they were collected and analyzed (Design).

  • ○ [long-prompts] Document input handling and token optimization strategies when prompts are long or complex (Design).

  • ○ [restricted-sharing] If full prompt disclosure is not feasible, provide summaries or examples (Design).

  • ○ [ensemble] For ensemble architectures, explain in the paper the coordination logic between models (Design).

  • ○ [context-augmentation] Report data preprocessing, versioning, and update frequency for stored data used for context augmentation (Design).

  • ○ [context-files] Include context-file contents as supplementary material  (Design).

  • ○ [tool-use] Include the complete list of tools and skills (names with one-line purposes), tool schemas, skill definitions, sub-agent definitions, and connected MCP servers as supplementary material  (Design).

Session Traces

  • ○ Include full interaction logs (prompts and responses) as supplementary material if privacy and confidentiality can be ensured (Traces).

  • ○ [agents] For agentic systems, include interaction logs covering exchanges between humans and the agent and between external tools and the agent (human-in-the-loop feedback, approvals, refinements) as supplementary material  (Traces).

  • ○ [agents] For agentic systems, report the complete runtime trace as supplementary material, including for each entry the tool or artifact name, arguments, result, and ordering, and which configured artifacts (skills, context files, sub-agents) were activated (Traces).

  • ○ [agents] For agentic systems, report developed plans as supplementary material if available (Traces).

Benchmarks and Metrics

  • ● Justify in the paper all benchmark and metric choices (Benchmarks).

  • ● Explain in the paper why the selected metrics are suitable for the specific study (Benchmarks).

  • ○ Provide an operational definition of the phenomenon the benchmark is intended to measure, including its scope and any sub-components (Benchmarks).

  • ○ Summarize benchmark structure, task types, and limitations (Benchmarks).

  • ○ Identify the capabilities a benchmark conflates with the target phenomenon, isolate the target where possible, and acknowledge remaining confounders as construct-validity threats (Benchmarks).

  • ○ Perform an error analysis: categorize the failures observed and report their relative frequency; report failures that cluster on confounding capabilities as construct-validity threats (Benchmarks).

  • ○ Describe and justify the sampling strategy used to select problems for inclusion in the benchmark (Benchmarks).

  • ○ Justify the number of experiment repetitions, for example through a power analysis or by monitoring convergence of descriptive statistics (Benchmarks).

  • ○ [non-probability-sampling] For non-probability sampling (e.g., convenience), discuss the implications for the generalizability of conclusions (Benchmarks).

  • ○ [new-benchmark] For new or released benchmarks, adopt contamination-prevention mechanisms: held-out subset, canary strings, and pre-exposure investigation against common training corpora (Benchmarks).

  • ○ [multi-rater-scoring] For ratings that vary across raters or runs (human raters, LLM-as-judge), report the distribution of ratings per item rather than only aggregated point estimates (Benchmarks).

Human Validation

  • ● [human-validation] If using human validation, define in the paper the measured construct (e.g., usability, maintainability) and describe the measurement instrument (Human Validation).

  • ● [human-validation] When developing or adapting measurement instruments, share them (Human Validation).

  • ○ Consider human validation early in the study design and build on established reference models for human-LLM comparison (Human Validation).

  • ○ [human-validation] Validate LLM judgments against human judgment, report aggregation methods, and assess human-LLM agreement (Human Validation).

  • ○ [human-validation] Discuss and, where feasible, control for confounding factors (Human Validation).

  • ○ [human-validation][subjective-constructs] For value-laden or culturally contingent constructs, describe rater demographics beyond expertise and discuss potential demographic biases (Human Validation).

Reproducibility, Ethics, and Resources

  • ● [restricted-sharing] For studies involving sensitive data, discuss data governance mechanisms compliant with applicable jurisdictional obligations (Limitations).

  • ○ Justify LLM usage in light of its resource demands (Limitations).

  • ○ Provide a full replication package as supplementary material, including step-by-step instructions for verifying and reproducing the results (Open LLM).

  • ○ [restricted-sharing] Where full sharing of prompts, traces, or datasets is not feasible, share representative examples for partial replicability (Limitations).

Results

  • ● [comparing-models] If comparing models or tools, use appropriate inferential statistics (e.g., hypothesis tests, effect sizes) rather than relying solely on summary statistics (Benchmarks).

  • ○ Repeat experiments due to the inherent non-determinism of LLMs and report the result distribution using descriptive statistics (Benchmarks).

  • ○ Use traditional (non-LLM) baselines for comparison where possible (Benchmarks).

  • ○ Report established metrics to make study results comparable; additional metrics may be reported where appropriate (Benchmarks).

Limitations and Threats to Validity

  • ● Describe measurement constructs and methods; disclose any data leakage risks and avoid leaking evaluation data into LLM improvement pipelines (Limitations).

  • ● Transparently report study limitations, including the impact of non-determinism and generalizability constraints (Limitations).

  • ● Specify whether generalization across LLMs or across time was assessed, and discuss model and version differences (Limitations).

  • ● [restricted-sharing] Acknowledge non-disclosed confidential or proprietary components as reproducibility limitations (Design).

  • ○ Employ and report strategies to mitigate identified validity and reproducibility threats, such as replication packages, human validation, longitudinal re-runs, triangulation, and sensitivity analysis (Limitations).

References Changed

2026.04

References

ACM Publications Board. 2020. “Artifact Review and Badging – Current.” https://www.acm.org/publications/policies/artifact-review-and-badging-current.
Unchanged text
Agarwal, Rishabh, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Marc G. Bellemare. 2021. “Deep Reinforcement Learning at the Edge of the Statistical Precipice.” In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, Virtual, edited by Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, 29304–20. https://proceedings.neurips.cc/paper/2021/hash/f514cec81cb148559cf475e7426eed5e-Abstract.html.
Ahmed, Nur, Muntasir Wahed, and Neil C. Thompson. 2023. “The Growing Influence of Industry in AI Research.” Science 379 (6635): 884–86. https://doi.org/10.1126/science.ade2420.
Ahmed, Toufique, Premkumar T. Devanbu, Christoph Treude, and Michael Pradel. 2025. “Can LLMs Replace Manual Annotation of Software Engineering Artifacts?” In 22nd IEEE/ACM International Conference on Mining Software Repositories, MSR@ICSE 2025, Ottawa, ON, Canada, April 28-29, 2025, 526–38. IEEE. https://doi.org/10.1109/MSR66628.2025.00086.
Ahuja, Sanchit, Varun Gumma, and Sunayana Sitaram. 2024. “Contamination Report for Multilingual Benchmarks.” CoRR abs/2410.16186. https://doi.org/10.48550/ARXIV.2410.16186.
Angermeir, Florian, Maximilian Amougou, Mark Kreitz, Andreas Bauer, Matthias Linhuber, Davide Fucci, Fabiola Moyón Constante, Daniel Méndez, and Tony Gorschek. 2025. “Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies.” CoRR abs/2510.25506. https://doi.org/10.48550/ARXIV.2510.25506.
Anthropic. 2025. “Demystifying Evals for AI Agents.” https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents.
———. 2026. “Claude Code Tools Reference.” https://code.claude.com/docs/en/tools-reference.
Argyle, Lisa P., Ethan C. Busby, Nancy Fulda, Joshua Gubler, Christopher Michael Rytting, and David Wingate. 2022. “Out of One, Many: Using Language Models to Simulate Human Samples.” CoRR abs/2209.06899. https://doi.org/10.48550/ARXIV.2209.06899.
Association for Computing Machinery. 2023. “ACM Policy on Authorship.” https://www.acm.org/publications/policies/new-acm-policy-on-authorship.
Austin, Jacob, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, et al. 2021. “Program Synthesis with Large Language Models.” CoRR abs/2108.07732. https://arxiv.org/abs/2108.07732.
Azanza, Maider, Juanan Pereira, Arantza Irastorza, and Aritz Galdos. 2024. “Can LLMs Facilitate Onboarding Software Developers? An Ongoing Industrial Case Study.” In 36th International Conference on Software Engineering Education and Training, CSEE&t 2024, 1–6. IEEE. https://doi.org/10.1109/CSEET62301.2024.10662989.
Bano, Muneera, Hashini Gunatilake, and Rashina Hoda. 2025. “What Does a Software Engineer Look Like? Exploring Societal Stereotypes in LLMs.” In 47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Society, ICSE-SEIS 2025, Ottawa, ON, Canada, April 27 - May 3, 2025, 173–84. IEEE. https://doi.org/10.1109/ICSE-SEIS66351.2025.00023.
Unchanged text
Bano, Muneera, Rashina Hoda, Didar Zowghi, and Christoph Treude. 2024. “Large Language Models for Qualitative Research in Software Engineering: Exploring Opportunities and Challenges.” Autom. Softw. Eng. 31 (1): 8. https://doi.org/10.1007/S10515-023-00407-8.
Bano, Muneera, Didar Zowghi, and Jon Whittle. 2023. “Exploring Qualitative Research Using LLMs.” CoRR abs/2306.13298. https://doi.org/10.48550/ARXIV.2306.13298.
Barros, Cauã Ferreira, Bruna Borges Azevedo, Valdemar Vicente Graciano Neto, Mohamad Kassab, Marcos Kalinowski, Hugo Alexandre Dantas do Nascimento, and Michelle C. G. S. P. Bandeira. 2025. “Large Language Model for Qualitative Research: A Systematic Mapping Study.” In IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering, WSESE@ICSE 2025, Ottawa, ON, Canada, May 3, 2025, 48–55. IEEE. https://doi.org/10.1109/WSESE66602.2025.00015.
Bavaresco, Anna, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, et al. 2024. “LLMs Instead of Human Judges? A Large Scale Empirical Study Across 20 NLP Evaluation Tasks.” CoRR abs/2406.18403. https://doi.org/10.48550/ARXIV.2406.18403.
Begg, C., M. Cho, S. Eastwood, R. Horton, D. Moher, I. Olkin, R. Pitkin, et al. 1996. “Improving the Quality of Reporting of Randomized Controlled Trials. The CONSORT statement.” JAMA 276 (8): 637–39. https://doi.org/10.1001/jama.276.8.637.
Bjarnason, Bjarni Haukur, André Silva, and Martin Monperrus. 2026. “On Randomness in Agentic Evals.” https://arxiv.org/abs/2602.07150.
Blackwell, Robert E., Jon Barry, and Anthony G. Cohn. 2024. “Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scores.” https://arxiv.org/abs/2410.03492.
Borsboom, Denny. 2005. Measuring the Mind: Conceptual Issues in Contemporary Psychometrics. Cambridge University Press. https://doi.org/10.1017/CBO9780511490026.
Cai, Yuchen, Aashish Yadavally, Abhishek Mishra, Genesis Montejo, and Tien N. Nguyen. 2024. “Programming Assistant for Exception Handling with CodeBERT.” In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024, 94:1–13. ACM. https://doi.org/10.1145/3597503.3639188.
Cao, Jialun, Yuk-Kit Chan, Zixuan Ling, Wenxuan Wang, Shuqing Li, Mingwei Liu, Ruixi Qiao, et al. 2025. “How Should We Build a Benchmark? Revisiting 274 Code-Related Benchmarks for LLMs.” CoRR abs/2501.10711. https://arxiv.org/abs/2501.10711.
Chandra, Satish. 2025. “Benchmarks for AI in Software Engineering (BLOG@CACM).” https://cacm.acm.org/blogcacm/benchmarks-for-ai-in-software-engineering/.
Unchanged text
Chann, Sherman. 2023. “Non-determinism in GPT-4 is caused by Sparse MoE.” https://152334h.github.io/blog/non-determinism-in-gpt-4/.
Chen, Lingjiao, Matei Zaharia, and James Zou. 2024. “How Is ChatGPT’s Behavior Changing over Time?” Harvard Data Science Review 6 (2). https://doi.org/10.1162/99608f92.5317da47.
Chen, Mark, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, et al. 2021. “Evaluating Large Language Models Trained on Code.” CoRR abs/2107.03374. https://arxiv.org/abs/2107.03374.
Chen, Yizheng, Zhoujie Ding, Lamya Alowain, Xinyun Chen, and David A. Wagner. 2023. “DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection.” In Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses, RAID 2023, Hong Kong, China, October 16-18, 2023, 654–68. ACM. https://doi.org/10.1145/3607199.3607242.
Choi, Hyeong Kyu, Maxim Khanov, Hongxin Wei, and Yixuan Li. 2025. “How Contaminated Is Your Benchmark? Quantifying Dataset Leakage in Large Language Models with Kernel Divergence.” CoRR abs/2502.00678. https://doi.org/10.48550/ARXIV.2502.00678.
Cohen, Jacob. 1992. “A Power Primer.” Psychological Bulletin 112 (1): 155–59. https://doi.org/10.1037/0033-2909.112.1.155.
DAIR.AI. 2024. “Elements of a Prompt.” https://www.promptingguide.ai/introduction/elements.
Dhar, Rudra, Karthik Vaidhyanathan, and Vasudeva Varma. 2024. “Can LLMs Generate Architectural Design Decisions? - an Exploratory Empirical Study.” In 21st IEEE International Conference on Software Architecture, ICSA 2024, Hyderabad, India, June 4-8, 2024, 79–89. IEEE. https://doi.org/10.1109/ICSA59870.2024.00016.
Du, Xueying, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2023. “ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-Level Code Generation.” CoRR abs/2308.01861. https://doi.org/10.48550/ARXIV.2308.01861.
Dybå, Tore, Vigdis By Kampenes, and Dag I. K. Sjøberg. 2006. “A Systematic Review of Statistical Power in Software Engineering Experiments.” Inf. Softw. Technol. 48 (8): 745–55. https://doi.org/10.1016/J.INFSOF.2005.08.009.
Eghbali, Aryaz, and Michael Pradel. 2022. “CrystalBLEU: Precisely and Efficiently Measuring the Similarity of Code.” In 37th IEEE/ACM International Conference on Automated Software Engineering, ASE 2022, 28:1–12. ACM. https://doi.org/10.1145/3551349.3556903.
El-Hajjami, Abdelkarim, and Camille Salinesi. 2025. “How Good Are Synthetic Requirements? Evaluating LLM-Generated Datasets for AI4RE.” CoRR abs/2506.21138. https://doi.org/10.48550/ARXIV.2506.21138.
Gallegos, Isabel O., Ryan A. Rossi, Joe Barrow, Md. Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen Ahmed. 2024. “Bias and Fairness in Large Language Models: A Survey.” Computational Linguistics 50: 1097–1179. https://doi.org/10.1162/coli_a_00524.
Gallifant, Jack, Majid Afshar, Saleem Ameen, Yindalon Aphinyanaphongs, Shan Chen, Giovanni Cacciamani, Dina Demner-Fushman, et al. 2025. “The TRIPOD-LLM Reporting Guideline for Studies Using Large Language Models.” Nature Medicine 31 (1): 60–69. https://doi.org/10.1038/s41591-024-03425-5.
Unchanged text
Galster, Matthias, Seyedmoein Mohsenimofidi, Jai Lal Lulla, Muhammad Auwal Abubakar, Christoph Treude, and Sebastian Baltes. 2026. “Configuring Agentic AI Coding Tools: An Exploratory Study.” In Proceedings of the 3rd ACM International Conference on AI-Powered Software (AIware 2026).
Gerosa, Marco Aurélio, Bianca Trinkenreich, Igor Steinmacher, and Anita Sarma. 2024. “Can AI Serve as a Substitute for Human Subjects in Software Engineering Research?” Autom. Softw. Eng. 31 (1): 13. https://doi.org/10.1007/S10515-023-00409-6.
Gibney, Elizabeth. 2024. “Not all ‘open source’ AI models are actually open.” Nature News. https://doi.org/10.1038/d41586-024-02012-5.
Gilardi, Fabrizio, Meysam Alizadeh, and Maël Kubli. 2023. “ChatGPT Outperforms Crowd Workers for Text-Annotation Tasks.” Proceedings of the National Academy of Sciences 120 (30): e2305016120. https://doi.org/10.1073/pnas.2305016120.
Gonçalves, José, Miguel Silva, Bernardo Cabral, Tiago Dias, Eva Maia, Isabel Praça, Ricardo Severino, and Luı́s Lino Ferreira. 2025. “Evaluating LLaMA 3.2 for Software Vulnerability Detection.” In Cybersecurity - 9th European Interdisciplinary Cybersecurity Conference, EICC 2025, Rennes, France, June 18-19, 2025, Proceedings, edited by Isabel Praça, Simona Bernardi, and Pedro R. M. Inácio, 2500:38–51. Communications in Computer and Information Science. Springer. https://doi.org/10.1007/978-3-031-94855-8\_3.
Graziotin, Daniel. 2024. “Acmsigsoft/Open-Science-Policies: V1.0.0.” Zenodo. https://doi.org/10.5281/zenodo.10796477.
Unchanged text
Guba, Egon G. 1981. “Criteria for Assessing the Trustworthiness of Naturalistic Inquiries.” ECTJ 29 (2): 75–91. https://doi.org/10.1007/BF02766777.
Guo, Daya, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, et al. 2024. “DeepSeek-Coder: When the Large Language Model Meets Programming - the Rise of Code Intelligence.” CoRR abs/2401.14196. https://doi.org/10.48550/ARXIV.2401.14196.
Harding, Jacqueline, William D’Alessandro, N. G. Laskowski, and Robert Long. 2024. “AI Language Models Cannot Replace Human Research Participants.” AI Soc. 39 (5): 2603–5. https://doi.org/10.1007/S00146-023-01725-X.
He, Junda, Jieke Shi, Terry Yue Zhuo, Christoph Treude, Jiamou Sun, Zhenchang Xing, Xiaoning Du, and David Lo. 2025. “LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead.” CoRR abs/2510.24367. https://doi.org/10.48550/ARXIV.2510.24367.
He, Zeyu, Chieh-Yang Huang, Chien-Kuang Cornelia Ding, Shaurya Rohatgi, and Ting-Hao Kenneth Huang. 2024. “If in a Crowdsourced Data Annotation Pipeline, a GPT-4.” In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI 2024, edited by Florian ’Floyd’Mueller, Penny Kyburz, Julie R. Williamson, Corina Sas, Max L. Wilson, Phoebe O. Toups Dugas, and Irina Shklovski, 1040:1–25. ACM. https://doi.org/10.1145/3613904.3642834.
Holtzman, Ari, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. “The Curious Case of Neural Text Degeneration.” In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. https://openreview.net/forum?id=rygGQyrFvH.
Hou, Xinyi, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. “Large Language Models for Software Engineering: A Systematic Literature Review.” ACM Trans. Softw. Eng. Methodol. 33 (8): 1–79. https://doi.org/10.1145/3695988.
Hu, Xing, Feifei Niu, Junkai Chen, Xin Zhou, Junwei Zhang, Junda He, Xin Xia, and David Lo. 2025. “Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Tasks.” CoRR abs/2505.08903. https://doi.org/10.48550/ARXIV.2505.08903.
Unchanged text
Huang, Fan, Haewoon Kwak, and Jisun An. 2023. “Is ChatGPT Better Than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech.” In Companion Proceedings of the ACM Web Conference 2023, WWW 2023, edited by Ying Ding, Jie Tang, Juan F. Sequeda, Lora Aroyo, Carlos Castillo, and Geert-Jan Houben, 294–97. ACM. https://doi.org/10.1145/3543873.3587368.
Huang, Jiangping, Bochen Yi, Weisong Sun, Bangrui Wan, Yang Xu, Yebo Feng, Wenguang Ye, and Qinjun Qin. 2024. “Enhancing Review Classification via LLM-Based Data Annotation and Multi-Perspective Feature Representation Learning.” SSRN Electronic Journal, 1–15. https://doi.org/10.2139/ssrn.5002351.
Hui, Binyuan, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, et al. 2024. “Qwen2.5-Coder Technical Report.” CoRR abs/2409.12186. https://doi.org/10.48550/ARXIV.2409.12186.
Husain, Hamel, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. “CodeSearchNet Challenge: Evaluating the State of Semantic Code Search.” CoRR abs/1909.09436. http://arxiv.org/abs/1909.09436.
Hymel, Cory, and Hiroe Johnson. 2025. “Analysis of LLMs Vs Human Experts in Requirements Engineering.” CoRR abs/2501.19297. https://doi.org/10.48550/ARXIV.2501.19297.
Jahic, Jasmin, and Ashkan Sami. 2024. “State of Practice: LLMs in Software Engineering and Software Architecture.” In 21st IEEE International Conference on Software Architecture, ICSA 2024 - Companion, Hyderabad, India, June 4-8, 2024, 311–18. IEEE. https://doi.org/10.1109/ICSA-C63560.2024.00059.
Jain, Naman, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. “LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code.” CoRR abs/2403.07974. https://doi.org/10.48550/ARXIV.2403.07974.
Jimenez, Carlos E., John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. “SWE-Bench: Can Language Models Resolve Real-World Github Issues?” In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum?id=VTF8yNQM66.
Jowsey, Tanisha, Virginia Braun, Victoria Clarke, Victoria Clarke, Deborah Lupton, and Michelle Fine. 2025. “We Reject the Use of Generative Artificial Intelligence for Reflexive Qualitative Research.” SSRN. https://doi.org/10.2139/ssrn.5676462.
Unchanged text
Kang, Sungmin, Juyeon Yoon, and Shin Yoo. 2023. “Large Language Models Are Few-Shot Testers: Exploring LLM-Based General Bug Reproduction.” In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, 2312–23. IEEE. https://doi.org/10.1109/ICSE48619.2023.00194.
Khojah, Ranim, Mazen Mohamad, Philipp Leitner, and Francisco Gomes de Oliveira Neto. 2024. “Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice.” Proc. ACM Softw. Eng. 1 (FSE): 1819–40. https://doi.org/10.1145/3660788.
Khraisha, Qusai, Sophie Put, Johanna Kappenberg, Azza Warraitch, and Kristin Hadfield. 2024. “Can Large Language Models Replace Humans in Systematic Reviews? Evaluating GPT-4’s Efficacy in Screening and Extracting Data from Peer-Reviewed and Grey Literature in Multiple Languages.” Research Synthesis Methods 15 (4): 616–26. https://doi.org/10.1002/jrsm.1715.
Kistowski, Jóakim v., Jeremy A. Arnold, Karl Huppler, Klaus-Dieter Lange, John L. Henning, and Paul Cao. 2015. “How to Build a Benchmark.” In Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering, 333–36. ICPE ’15. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/2668930.2688819.
Kong, Aobo, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, and Xin Zhou. 2023. “Better Zero-Shot Reasoning with Role-Play Prompting.” CoRR abs/2308.07702. https://doi.org/10.48550/ARXIV.2308.07702.
Krippendorff, Klaus. 2018. Content Analysis: An Introduction to Its Methodology. 4th ed. SAGE Publications.
Kübler, Jonas M., Kailash Budhathoki, Matthäus Kleindessner, Xiong Zhou, Junming Yin, Ashish Khetan, and George Karypis. 2026. “When LLMs Get Significantly Worse: A Statistical Approach to Detect Model Degradations.” In Proc. Of the International Conference on Learning Representations (ICLR).
Kulal, Sumith, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. 2019. “SPoC: Search-Based Pseudocode to Code.” CoRR abs/1906.04908. http://arxiv.org/abs/1906.04908.
Lachaux, Marie-Anne, Baptiste Rozière, Lowik Chanussot, and Guillaume Lample. 2020. “Unsupervised Translation of Programming Languages.” CoRR abs/2006.03511. https://arxiv.org/abs/2006.03511.
Li, David, Kartik Gupta, Mousumi Bhaduri, Paul Sathiadoss, Sahir Bhatnagar, and Jaron Chong. 2024. “Comparing GPT-3.5 and GPT-4 Accuracy and Drift in Radiology Diagnosis Please Cases.” Radiology 310 (1): e232411. https://doi.org/10.1148/radiol.232411.
Li, Raymond, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, et al. 2023. “StarCoder: May the Source Be with You!” CoRR abs/2305.06161. https://doi.org/10.48550/ARXIV.2305.06161.
Liang, Jenny T, Carmen Badea, Christian Bird, Robert DeLine, Denae Ford, Nicole Forsgren, and Thomas Zimmermann. 2024. “Can Gpt-4 Replicate Empirical Software Engineering Research?” Proceedings of the ACM on Software Engineering 1 (FSE): 1330–53.
Lu, Qinghua, Liming Zhu, Xiwei Xu, Zhenchang Xing, and Jon Whittle. 2024. “Toward Responsible AI in the Era of Generative AI: A Reference Architecture for Designing Foundation Model-Based Systems.” IEEE Softw. 41 (6): 91–100. https://doi.org/10.1109/MS.2024.3406333.
Unchanged text
Lubos, Sebastian, Alexander Felfernig, Thi Ngoc Trang Tran, Damian Garber, Merfat El Mansi, Seda Polat Erdeniz, and Viet-Man Le. 2024. “Leveraging LLMs for the Quality Assurance of Software Requirements.” In 32nd IEEE International Requirements Engineering Conference, RE 2024, Reykjavik, Iceland, June 24-28, 2024, edited by Grischa Liebel, Irit Hadar, and Paola Spoletini, 389–97. IEEE. https://doi.org/10.1109/RE59067.2024.00046.
Madampe, Kashumi, John Grundy, Rashina Hoda, and Humphrey O. Obie. 2024. “The Struggle Is Real! The Agony of Recruiting Participants for Empirical Software Engineering Studies.” In 2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), Liverpool, UK, September 2-6, 2024, 417–22. IEEE. https://doi.org/10.1109/VL/HCC60511.2024.00065.
McDonald, Nora, Sarita Schoenebeck, and Andrea Forte. 2019. “Reliability and Inter-Rater Reliability in Qualitative Research: Norms and Guidelines for CSCW and HCI Practice.” Proc. ACM Hum. Comput. Interact. 3 (CSCW): 72:1–23. https://doi.org/10.1145/3359174.
Menzies, Tim. 2025. “The Case for Compact AI.” Commun. ACM 68 (9): 6–7. https://doi.org/10.1145/3746057.
Microsoft. 2023. “CodeBERT on GitHub.” https://github.com/microsoft/CodeBERT.
Microsoft. 2025. “Azure OpenAI Service models.” https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models.
Mitu, Narcis Eduard, and George Teodor Mitu. 2024. “The Hidden Cost of AI: Carbon Footprint and Mitigation Strategies.” Revista de Stiinte Politice. Revue Des Sciences Politiques 84: 9–16. https://doi.org/10.2139/ssrn.5036344.
Mohamed, Suad, Abdullah Parvin, and Esteban Parra. 2024. “Chatting with AI: Deciphering Developer Conversations with ChatGPT.” In 21st IEEE/ACM International Conference on Mining Software Repositories, MSR 2024, Lisbon, Portugal, April 15-16, 2024, edited by Diomidis Spinellis, Alberto Bacchelli, and Eleni Constantinou, 187–91. ACM. https://doi.org/10.1145/3643991.3645078.
Mohsenimofidi, Seyedmoein, Matthias Galster, Christoph Treude, and Sebastian Baltes. 2026. “Context Engineering for AI Agents in Open-Source Software.” In Proceedings of the 23rd IEEE/ACM International Conference on Mining Software Repositories (MSR 2026). https://arxiv.org/abs/2510.21413.
Montes, Cristina Martinez, Robert Feldt, Cristina Miguel Martos, Sofia Ouhbi, Shweta Premanandan, and Daniel Graziotin. 2025. “Large Language Models in Thematic Analysis: Prompt Engineering, Evaluation, and Guidelines for Qualitative Software Engineering Research.” CoRR abs/2510.18456. https://doi.org/10.48550/ARXIV.2510.18456.
Morais Leça, Matheus de, Lucas Valença, Reydne Santos, and Ronnie de Souza Santos. 2025. “Applications and Implications of Large Language Models in Qualitative Analysis: A New Frontier for Empirical Software Engineering.” In IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering, WSESE@ICSE 2025, Ottawa, ON, Canada, May 3, 2025, 36–43. IEEE. https://doi.org/10.1109/WSESE66602.2025.00013.
Moumoula, Micheline Bénédicte, Abdoul Kader Kaboré, Jacques Klein, and Tegawendé F. Bissyandé. 2024. “Large Language Models for Cross-Language Code Clone Detection.” CoRR abs/2408.04430. https://doi.org/10.48550/ARXIV.2408.04430.
Open Source Initiative (OSI). 2025. “Open Source AI Definition 1.0.” https://opensource.org/ai/open-source-ai-definition.
OpenAI. 2023. “How to make your completions outputs consistent with the new seed parameter.” https://cookbook.openai.com/examples/reproducible_outputs_with_the_seed_parameter.
———. 2025. “OpenAI API Introduction.” https://platform.openai.com/docs/api-reference/chat/streaming.
OpenCode Contributors. 2025. “OpenCode: The Open Source AI Coding Agent.” https://opencode.ai/.
Ornelas, Tatiane, Allysson Allex Araújo, Júlia Araújo, Marina Araújo, Bianca Trinkenreich, and Marcos Kalinowski. 2025. “LLM-Assisted Thematic Analysis: Opportunities, Limitations, and Recommendations.” CoRR abs/2511.14528. https://doi.org/10.48550/ARXIV.2511.14528.
Pan, Qian, Zahra Ashktorab, Michael Desmond, Martin Santillan Cooper, James Johnson, Rahul Nair, Elizabeth Daly, and Werner Geyer. 2024. “Human-Centered Design Recommendations for LLM-as-a-Judge.” In ACL 2024 Workshop HuCLLM. arXiv. https://doi.org/10.48550/arXiv.2407.03479.
Pangakis, Nicholas, Samuel Wolken, and Neil Fasching. 2023. “Automated Annotation with Generative AI Requires Validation.” CoRR abs/2306.00176. https://doi.org/10.48550/ARXIV.2306.00176.
Panickssery, Arjun, Samuel Bowman, and Shi Feng. 2024. “LLM Evaluators Recognize and Favor Their Own Generations.” In Advances in Neural Information Processing Systems, edited by A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, 37:68772–802. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2024/file/7f1f0218e45f5414c79c0679633e47bc-Paper-Conference.pdf.
Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. “Bleu: A Method for Automatic Evaluation of Machine Translation.” In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, 311–18. ACL. https://doi.org/10.3115/1073083.1073135.
Pezeshkpour, Pouya, and Estevam Hruschka. 2024. “Large Language Models Sensitivity to the Order of Options in Multiple-Choice Questions.” In Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024, edited by Kevin Duh, Helena Gómez-Adorno, and Steven Bethard, NAACL 2024:2006–17. Findings of ACL. Association for Computational Linguistics. https://doi.org/10.18653/V1/2024.FINDINGS-NAACL.130.
Unchanged text
Rabbi, Md. Fazle, Arifa I. Champa, Minhaz Fahim Zibran, and Md. Rakibul Islam. 2024. “AI Writes, We Analyze: The ChatGPT Python Code Saga.” In 21st IEEE/ACM International Conference on Mining Software Repositories, MSR 2024, Lisbon, Portugal, April 15-16, 2024, edited by Diomidis Spinellis, Alberto Bacchelli, and Eleni Constantinou, 177–81. ACM. https://doi.org/10.1145/3643991.3645076.
Ralph, Paul, Nauman bin Ali, Sebastian Baltes, Domenico Bianculli, Jessica Diaz, Yvonne Dittrich, Neil Ernst, et al. 2021. “Empirical Standards for Software Engineering Research.” https://arxiv.org/abs/2010.03525.
Ralph, Paul, Miikka Kuutila, Hera Arif, and Bimpe Ayoola. 2024. “Teaching Software Metrology: The Science of Measurement for Software Engineering.” In Handbook on Teaching Empirical Software Engineering, edited by Daniel Méndez, Paris Avgeriou, Marcos Kalinowski, and Nauman Bin Ali, 101–54. Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-71769-7\_5.
Ralph, Paul, and Ewan D. Tempero. 2018. “Construct Validity in Software Engineering Research and Software Metrics.” In Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering, EASE2018, edited by Austen Rainer, Stephen G. MacDonell, and Jacky W. Keung, 13–23. ACM. https://doi.org/10.1145/3210459.3210461.
Rasheed, Zeeshan, Muhammad Waseem, Aakash Ahmad, Kai-Kristian Kemell, Xiaofeng Wang, Anh Nguyen-Duc, and Pekka Abrahamsson. 2024. “Can Large Language Models Serve as Data Analysts? A Multi-Agent Assisted Approach for Qualitative Data Analysis.” CoRR abs/2402.01386. https://doi.org/10.48550/ARXIV.2402.01386.
Reiss, Michael V. 2023. “Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark.” CoRR abs/2304.11085. https://doi.org/10.48550/ARXIV.2304.11085.
Ren, Shuo, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. “CodeBLEU: A Method for Automatic Evaluation of Code Synthesis.” CoRR abs/2009.10297. https://arxiv.org/abs/2009.10297.
Renze, Matthew. 2024. “The Effect of Sampling Temperature on Problem Solving in Large Language Models.” In Findings of the Association for Computational Linguistics: EMNLP 2024, 7346–56. https://doi.org/10.18653/v1/2024.findings-emnlp.432.
Richards, Jonan, and Mairieli Wessel. 2024. “What You Need Is What You Get: Theory of Mind for an LLM-Based Code Understanding Assistant.” In IEEE International Conference on Software Maintenance and Evolution, ICSME 2024, 666–71. IEEE. https://doi.org/10.1109/ICSME58944.2024.00070.
Ronanki, Krishna, Christian Berger, and Jennifer Horkoff. 2023. “Investigating ChatGPT’s Potential to Assist in Requirements Elicitation Processes.” In 2023 49th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), 354–61. IEEE.
Rozière, Baptiste, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, et al. 2023. “Code Llama: Open Foundation Models for Code.” CoRR abs/2308.12950. https://doi.org/10.48550/ARXIV.2308.12950.
Russo, Daniel. 2024. “Navigating the Complexity of Generative AI Adoption in Software Engineering.” ACM Transactions on Software Engineering and Methodology 33 (5): 1–50.
Sallou, June, Thomas Durieux, and Annibale Panichella. 2024. “Breaking the Silence: The Threats of Using LLMs in Software Engineering.” In Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results, NIER@ICSE 2024, Lisbon, Portugal, April 14-20, 2024, 102–6. ACM. https://doi.org/10.1145/3639476.3639764.
Unchanged text
Santos, Adrian, Sira Vegas, Markku Oivo, and Natalia Juristo. 2021. “A Procedure and Guidelines for Analyzing Groups of Software Engineering Replications.” IEEE Trans. Software Eng. 47 (9): 1742–63. https://doi.org/10.1109/TSE.2019.2935720.
Schäfer, Max, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024. “An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation.” IEEE Trans. Software Eng. 50 (1): 85–105. https://doi.org/10.1109/TSE.2023.3334955.
Schilke, Oliver, and Martin Reimann. 2025. “The Transparency Dilemma: How AI Disclosure Erodes Trust.” Organizational Behavior and Human Decision Processes 188: 104405. https://doi.org/10.1016/j.obhdp.2025.104405.
Schneider, Kurt, Farnaz Fotrousi, and Rebekka Wohlrab. 2025. “A Reference Model for Empirically Comparing LLMs with Humans.” In 47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Society, ICSE-SEIS 2025, Ottawa, ON, Canada, April 27 - May 3, 2025, 130–34. IEEE. https://doi.org/10.1109/ICSE-SEIS66351.2025.00018.
Schröder, Sarah, Thekla Morgenroth, Ulrike Kuhl, Valerie Vaquet, and Benjamin Paaßen. 2025. “Large Language Models Do Not Simulate Human Psychology.” CoRR abs/2508.06950. https://doi.org/10.48550/ARXIV.2508.06950.
Schroeder, Kayla, and Zach Wood-Doughty. 2024. “Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge.” CoRR abs/2412.12509. https://doi.org/10.48550/ARXIV.2412.12509.
Schulhoff, Sander, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, et al. 2024. “The Prompt Report: A Systematic Survey of Prompting Techniques.” CoRR abs/2406.06608. https://doi.org/10.48550/ARXIV.2406.06608.
Schulz, Kenneth F., Douglas G. Altman, and David Moher. 2010. “CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomised Trials.” BMJ 340: c332. https://doi.org/10.1136/bmj.c332.
Schwartz, Roy, Jesse Dodge, Noah A. Smith, and Oren Etzioni. 2020. “Green AI.” Communications of the ACM 63 (12): 54–63. https://doi.org/10.1145/3381831.
Sclar, Melanie, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. “Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I Learned to Start Worrying about Prompt Formatting.” In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum?id=RIu5lyNXjT.
Shull, Forrest, Janice Singer, and Dag I. K. Sjøberg, eds. 2008. Guide to Advanced Empirical Software Engineering. Springer. https://doi.org/10.1007/978-1-84800-044-5.
Silva, André, and Martin Monperrus. 2025. “RepairBench: Leaderboard of Frontier Models for Program Repair.” In IEEE/ACM International Workshop on Large Language Models for Code, LLM4Code@ICSE 2025, Ottawa, ON, Canada, May 3, 2025, 9–16. IEEE. https://doi.org/10.1109/LLM4CODE66737.2025.00006.
Sjøberg, Dag I. K., and Gunnar Rye Bergersen. 2023. “Construct Validity in Software Engineering.” IEEE Trans. Software Eng. 49 (3): 1374–96. https://doi.org/10.1109/TSE.2022.3176725.
Song, Yifan, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. 2025. “The Good, the Bad, and the Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism.” In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2025 - Volume 1: Long Papers, edited by Luis Chiruzzo, Alan Ritter, and Lu Wang, 4195–4206. Association for Computational Linguistics. https://doi.org/10.18653/V1/2025.NAACL-LONG.211.
Sonnekalb, Tim, Bernd Gruner, Clemens-Alexander Brust, and Patrick Mäder. 2022. “Generalizability of Code Clone Detection on CodeBERT.” In 37th IEEE/ACM International Conference on Automated Software Engineering, ASE 2022, 143:1–3. ACM. https://doi.org/10.1145/3551349.3561165.
Stack Overflow. 2025. “2025 Developer Survey.” https://survey.stackoverflow.co/2025/.
Strubell, Emma, Ananya Ganesh, and Andrew McCallum. 2019. “Energy and Policy Considerations for Deep Learning in NLP.” In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 3645–50. https://doi.org/10.18653/v1/P19-1355.
Sullivan, Gail M, and Joan Sargeant. 2011. “Qualities of Qualitative Research: Part I.” J Grad Med Educ 3 (4): 449–52.
Sumers, Theodore R., Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. 2024. “Cognitive Architectures for Language Agents.” Trans. Mach. Learn. Res. 2024. https://openreview.net/forum?id=1i6ZCvflQJ.
Takerngsaksiri, Wannita, Jirat Pasuksmit, Patanamon Thongtanunam, Chakkrit Tantithamthavorn, Ruixiong Zhang, Fan Jiang, Jing Li, Evan Cook, Kun Chen, and Ming Wu. 2025. “Human-in-the-Loop Software Development Agents.” In 47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice, SEIP@ICSE 2025, Ottawa, ON, Canada, April 27 - May 3, 2025, 342–52. IEEE. https://doi.org/10.1109/ICSE-SEIP66354.2025.00036.
Tam, Thomas Yu Chow, Sonish Sivarajkumar, Sumit Kapoor, Alisa V Stolyar, Katelyn Polanska, Karleigh R McCarthy, Hunter Osterhoudt, et al. 2024. “A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Review.” NPJ Digital Medicine 7 (1): 258.
Wagner, Stefan, Marvin Muñoz Barón, Davide Falessi, and Sebastian Baltes. 2025. “Towards Evaluation Guidelines for Empirical Studies Involving LLMs.” In IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering, WSESE@ICSE 2025, May 3, 2025, 24–27. IEEE. https://doi.org/10.1109/WSESE66602.2025.00011.
Wan, Mengting, Tara Safavi, Sujay Kumar Jauhar, Yujin Kim, Scott Counts, Jennifer Neville, Siddharth Suri, et al. 2024. “TnT-LLM: Text Mining at Scale with Large Language Models.” In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2024, edited by Ricardo Baeza-Yates and Francesco Bonchi, 5836–47. ACM. https://doi.org/10.1145/3637528.3671647.
Wang, Angelina, Jamie Morgenstern, and John P. Dickerson. 2025. “Large Language Models That Replace Human Participants Can Harmfully Misportray and Flatten Identity Groups.” Nature Machine Intelligence 7: 400–411. https://doi.org/10.1038/s42256-025-00986-z.
Wang, Chong, Kaifeng Huang, Jian Zhang, Yebo Feng, Lyuye Zhang, Yang Liu, and Xin Peng. 2024. “How and Why LLMs Use Deprecated APIs in Code Completion? An Empirical Study.” CoRR abs/2406.09834. https://doi.org/10.48550/ARXIV.2406.09834.
Unchanged text
Wang, Fanyu, Chetan Arora, Yonghui Liu, Kaicheng Huang, Chakkrit Tantithamthavorn, Aldeida Aleti, Dishan Sambathkumar, and David Lo. 2025. “Multi-Modal Requirements Data-Based Acceptance Criteria Generation Using LLMs.” In 40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025, 3334–45. IEEE. https://doi.org/10.1109/ASE63991.2025.00275.
Wang, Shuohang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. “Want to Reduce Labeling Cost? GPT-3 Can Help.” In Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, edited by Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, EMNLP 2021:4195–205. Findings of ACL. Association for Computational Linguistics. https://doi.org/10.18653/V1/2021.FINDINGS-EMNLP.354.
Wang, Xinru, Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjie Miao. 2024. “Human-LLM Collaborative Annotation Through Effective Verification of LLM Labels.” In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI 2024, edited by Florian ’Floyd’Mueller, Penny Kyburz, Julie R. Williamson, Corina Sas, Max L. Wilson, Phoebe O. Toups Dugas, and Irina Shklovski, 303:1–21. ACM. https://doi.org/10.1145/3613904.3641960.
Widder, David Gray, Meredith Whittaker, and Sarah Myers West. 2024. “Why ‘Open’AI Systems Are Actually Closed, and Why This Matters.” Nature 635 (8040): 827–33.
Wiesinger, Julia, Patrick Marlow, and Vladimir Vuskovic. 2025. “Agents.” Google DeepMind. https://gemini.google.com.
Willison, Simon. 2026. “Opus system prompt.” https://simonwillison.net/2026/Apr/18/opus-system-prompt/.
Wohlin, Claes, Per Runeson, Martin Höst, Magnus C. Ohlsson, Björn Regnell, and Anders Wesslén. 2024. Experimentation in Software Engineering, Second Edition. Springer. https://doi.org/10.1007/978-3-662-69306-3.
Xia, Yuying, Haijian Shao, and Xing Deng. 2024. “VulCoBERT: A CodeBERT-Based System for Source Code Vulnerability Detection.” In 2024 International Conference on Generative Artificial Intelligence and Information Security, GAIIS 2024, Kuala Lumpur, Malaysia, May 10-12, 2024. ACM. https://doi.org/10.1145/3665348.3665391.
Xiao, Tao, Christoph Treude, Hideaki Hata, and Kenichi Matsumoto. 2024. “DevGPT: Studying Developer-ChatGPT Conversations.” In 21st IEEE/ACM International Conference on Mining Software Repositories, MSR 2024, Lisbon, Portugal, April 15-16, 2024, edited by Diomidis Spinellis, Alberto Bacchelli, and Eleni Constantinou, 227–30. ACM. https://doi.org/10.1145/3643991.3648400.
Xu, Cheng, Shuhao Guan, Derek Greene, and M. Tahar Kechadi. 2024. “Benchmark Data Contamination of Large Language Models: A Survey.” CoRR abs/2406.04244. https://doi.org/10.48550/ARXIV.2406.04244.
Xu, Ruoxi, Yingfei Sun, Mengjie Ren, Shiguang Guo, Ruotong Pan, Hongyu Lin, Le Sun, and Xianpei Han. 2024. “AI for Social Science and Social Science of AI: A Survey.” Inf. Process. Manag. 61 (2): 103665. https://doi.org/10.1016/J.IPM.2024.103665.
Yan, Litao, Alyssa Hwang, Zhiyuan Wu, and Andrew Head. 2024. “Ivie: Lightweight Anchored Explanations of Just-Generated Code.” In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI 2024, edited by Florian ’Floyd’Mueller, Penny Kyburz, Julie R. Williamson, Corina Sas, Max L. Wilson, Phoebe O. Toups Dugas, and Irina Shklovski, 140:1–15. ACM. https://doi.org/10.1145/3613904.3642239.
Yang, Guang, Yu Zhou, Xiang Chen, Xiangyu Zhang, Tingting Han, and Taolue Chen. 2023. “ExploitGen: Template-Augmented Exploit Code Generation Based on CodeBERT.” J. Syst. Softw. 197: 111577. https://doi.org/10.1016/J.JSS.2022.111577.
Yang, Yifan, Jiho Shin, Byeonggyu Choi, Minjun Park, Dayun Ju, Changmin Lee, Sanghyuk Chun, Dongjin Kang, Jiin Kim, and Sungroh Yoon. 2025. “From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence.” arXiv Preprint arXiv:2502.11827. https://arxiv.org/abs/2502.11827.
Yuan, Jiayi, Hao Li, Xinheng Ding, Wenya Xie, Yu-Jhe Li, Wentian Zhao, Kun Wan, Jing Shi, Xia Hu, and Zirui Liu. 2025. “Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference.” In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025. https://openreview.net/forum?id=Q3qAsZAEZw.
Unchanged text
Zhao, Chenguang, Meirewuti Habule, and Wei Zhang. 2025. “Large Language Models (LLMs) as Research Subjects: Status, Opportunities and Challenges.” New Ideas in Psychology 79: 101167. https://doi.org/10.1016/j.newideapsych.2025.101167.
Zhou, Ruiyang, Lu Chen, and Kai Yu. 2024. “Is LLM a Reliable Reviewer? A Comprehensive Evaluation of LLM on Automatic Paper Reviewing Tasks.” In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, edited by Nicoletta Calzolari, Min-Yen Kan, Véronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, 9340–51. ELRA; ICCL. https://aclanthology.org/2024.lrec-main.816.
Zhou, Wangchunshu, Yuchen Eleanor Jiang, Long Li, Jialong Wu, Tiannan Wang, Shi Qiu, Jintian Zhang, et al. 2023. “Agents: An Open-Source Framework for Autonomous Language Agents.” CoRR abs/2309.07870. https://doi.org/10.48550/ARXIV.2309.07870.
Zhu, Yiming, Peixian Zhang, Ehsan ul Haq, Pan Hui, and Gareth Tyson. 2023. “Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks.” CoRR abs/2304.10145. https://doi.org/10.48550/ARXIV.2304.10145.

2026.05

References

ACM Publications Board. 2020. “Artifact Review and Badging – Current.” https://www.acm.org/publications/policies/artifact-review-and-badging-current.
Unchanged text
Agarwal, Rishabh, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Marc G. Bellemare. 2021. “Deep Reinforcement Learning at the Edge of the Statistical Precipice.” In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, Virtual, edited by Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, 29304–20. https://proceedings.neurips.cc/paper/2021/hash/f514cec81cb148559cf475e7426eed5e-Abstract.html.
Ahmed, Nur, Muntasir Wahed, and Neil C. Thompson. 2023. “The Growing Influence of Industry in AI Research.” Science 379 (6635): 884–86. https://doi.org/10.1126/science.ade2420.
Ahmed, Toufique, Premkumar T. Devanbu, Christoph Treude, and Michael Pradel. 2025. “Can LLMs Replace Manual Annotation of Software Engineering Artifacts?” In 22nd IEEE/ACM International Conference on Mining Software Repositories, MSR@ICSE 2025, Ottawa, ON, Canada, April 28-29, 2025, 526–38. IEEE. https://doi.org/10.1109/MSR66628.2025.00086.
Ahuja, Sanchit, Varun Gumma, and Sunayana Sitaram. 2024. “Contamination Report for Multilingual Benchmarks.” CoRR abs/2410.16186. https://doi.org/10.48550/ARXIV.2410.16186.
Angermeir, Florian, Maximilian Amougou, Mark Kreitz, Andreas Bauer, Matthias Linhuber, Davide Fucci, Fabiola Moyón Constante, Daniel Méndez, and Tony Gorschek. 2025. “Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies.” CoRR abs/2510.25506. https://doi.org/10.48550/ARXIV.2510.25506.
Anthropic. 2025. “Demystifying Evals for AI Agents.” https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents.
———. 2026. “Claude Code Tools Reference.” https://code.claude.com/docs/en/tools-reference.
Argyle, Lisa P., Ethan C. Busby, Nancy Fulda, Joshua Gubler, Christopher Michael Rytting, and David Wingate. 2022. “Out of One, Many: Using Language Models to Simulate Human Samples.” CoRR abs/2209.06899. https://doi.org/10.48550/ARXIV.2209.06899.
Arize AI. 2026. “OpenInference: AI Observability and Evaluation Specification.” https://github.com/Arize-ai/openinference.
Association for Computing Machinery. 2023. “ACM Policy on Authorship.” https://www.acm.org/publications/policies/new-acm-policy-on-authorship.
Austin, Jacob, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, et al. 2021. “Program Synthesis with Large Language Models.” CoRR abs/2108.07732. https://arxiv.org/abs/2108.07732.
Azanza, Maider, Juanan Pereira, Arantza Irastorza, and Aritz Galdos. 2024. “Can LLMs Facilitate Onboarding Software Developers? An Ongoing Industrial Case Study.” In 36th International Conference on Software Engineering Education and Training, CSEE&t 2024, 1–6. IEEE. https://doi.org/10.1109/CSEET62301.2024.10662989.
Baltes, Sebastian, and Paul Ralph. 2022. “Sampling in Software Engineering Research: A Critical Review and Guidelines.” Empir. Softw. Eng. 27 (4): 94. https://doi.org/10.1007/s10664-021-10072-8.
Bano, Muneera, Hashini Gunatilake, and Rashina Hoda. 2025. “What Does a Software Engineer Look Like? Exploring Societal Stereotypes in LLMs.” In 47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Society, ICSE-SEIS 2025, Ottawa, ON, Canada, April 27 - May 3, 2025, 173–84. IEEE. https://doi.org/10.1109/ICSE-SEIS66351.2025.00023.
Unchanged text
Bano, Muneera, Rashina Hoda, Didar Zowghi, and Christoph Treude. 2024. “Large Language Models for Qualitative Research in Software Engineering: Exploring Opportunities and Challenges.” Autom. Softw. Eng. 31 (1): 8. https://doi.org/10.1007/S10515-023-00407-8.
Bano, Muneera, Didar Zowghi, and Jon Whittle. 2023. “Exploring Qualitative Research Using LLMs.” CoRR abs/2306.13298. https://doi.org/10.48550/ARXIV.2306.13298.
Barros, Cauã Ferreira, Bruna Borges Azevedo, Valdemar Vicente Graciano Neto, Mohamad Kassab, Marcos Kalinowski, Hugo Alexandre Dantas do Nascimento, and Michelle C. G. S. P. Bandeira. 2025. “Large Language Model for Qualitative Research: A Systematic Mapping Study.” In IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering, WSESE@ICSE 2025, Ottawa, ON, Canada, May 3, 2025, 48–55. IEEE. https://doi.org/10.1109/WSESE66602.2025.00015.
Bavaresco, Anna, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, et al. 2024. “LLMs Instead of Human Judges? A Large Scale Empirical Study Across 20 NLP Evaluation Tasks.” CoRR abs/2406.18403. https://doi.org/10.48550/ARXIV.2406.18403.
Bean, Andrew M., Ryan Othniel Kearns, Angelika Romanou, Franziska Sofia Hafner, Harry Mayne, Jan Batzner, Negar Foroutan, et al. 2025. “Measuring What Matters: Construct Validity in Large Language Model Benchmarks.” In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, Datasets and Benchmarks Track. https://arxiv.org/abs/2511.04703.
Begg, C., M. Cho, S. Eastwood, R. Horton, D. Moher, I. Olkin, R. Pitkin, et al. 1996. “Improving the Quality of Reporting of Randomized Controlled Trials. The CONSORT statement.” JAMA 276 (8): 637–39. https://doi.org/10.1001/jama.276.8.637.
Bjarnason, Bjarni Haukur, André Silva, and Martin Monperrus. 2026. “On Randomness in Agentic Evals.” CoRR abs/2602.07150. https://doi.org/10.48550/ARXIV.2602.07150.
Blackwell, Robert E., Jon Barry, and Anthony G. Cohn. 2024. “Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scores.” CoRR abs/2410.03492. https://doi.org/10.48550/ARXIV.2410.03492.
Borsboom, Denny. 2005. Measuring the Mind: Conceptual Issues in Contemporary Psychometrics. Cambridge University Press. https://doi.org/10.1017/CBO9780511490026.
Bouzenia, Islem, and Michael Pradel. 2025. “Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories.” In 40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025, 2846–57. IEEE. https://doi.org/10.1109/ASE63991.2025.00234.
Cai, Yuchen, Aashish Yadavally, Abhishek Mishra, Genesis Montejo, and Tien N. Nguyen. 2024. “Programming Assistant for Exception Handling with CodeBERT.” In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024, 94:1–13. ACM. https://doi.org/10.1145/3597503.3639188.
Cao, Jialun, Yuk-Kit Chan, Zixuan Ling, Wenxuan Wang, Shuqing Li, Mingwei Liu, Ruixi Qiao, et al. 2026. “Rigor, Reliability, and Reproducibility Matter: A Decade-Scale Survey of 572 Code Benchmarks.” CoRR abs/2501.10711. https://arxiv.org/abs/2501.10711.
Chandra, Satish. 2025. “Benchmarks for AI in Software Engineering (BLOG@CACM).” https://cacm.acm.org/blogcacm/benchmarks-for-ai-in-software-engineering/.
Unchanged text
Chann, Sherman. 2023. “Non-determinism in GPT-4 is caused by Sparse MoE.” https://152334h.github.io/blog/non-determinism-in-gpt-4/.
Chen, Lingjiao, Matei Zaharia, and James Zou. 2024. “How Is ChatGPT’s Behavior Changing over Time?” Harvard Data Science Review 6 (2). https://doi.org/10.1162/99608f92.5317da47.
Chen, Mark, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, et al. 2021. “Evaluating Large Language Models Trained on Code.” CoRR abs/2107.03374. https://arxiv.org/abs/2107.03374.
Chen, Yizheng, Zhoujie Ding, Lamya Alowain, Xinyun Chen, and David A. Wagner. 2023. “DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection.” In Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses, RAID 2023, Hong Kong, China, October 16-18, 2023, 654–68. ACM. https://doi.org/10.1145/3607199.3607242.
Cheng, Adam, Aaron Calhoun, and Gabriel Reedy. 2025. “Artificial Intelligence-Assisted Academic Writing: Recommendations for Ethical Use.” Advances in Simulation 10 (1): 26. https://doi.org/10.1186/s41077-025-00350-6.
Choi, Hyeong Kyu, Maxim Khanov, Hongxin Wei, and Yixuan Li. 2025. “How Contaminated Is Your Benchmark? Quantifying Dataset Leakage in Large Language Models with Kernel Divergence.” CoRR abs/2502.00678. https://doi.org/10.48550/ARXIV.2502.00678.
Unchanged text
Cohen, Jacob. 1992. “A Power Primer.” Psychological Bulletin 112 (1): 155–59. https://doi.org/10.1037/0033-2909.112.1.155.
DAIR.AI. 2024. “Elements of a Prompt.” https://www.promptingguide.ai/introduction/elements.
Dhar, Rudra, Karthik Vaidhyanathan, and Vasudeva Varma. 2024. “Can LLMs Generate Architectural Design Decisions? - an Exploratory Empirical Study.” In 21st IEEE International Conference on Software Architecture, ICSA 2024, Hyderabad, India, June 4-8, 2024, 79–89. IEEE. https://doi.org/10.1109/ICSA59870.2024.00016.
Du, Xueying, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2024. “Evaluating Large Language Models in Class-Level Code Generation.” In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024, 81:1–13. ACM. https://doi.org/10.1145/3597503.3639219.
Dybå, Tore, Vigdis By Kampenes, and Dag I. K. Sjøberg. 2006. “A Systematic Review of Statistical Power in Software Engineering Experiments.” Inf. Softw. Technol. 48 (8): 745–55. https://doi.org/10.1016/J.INFSOF.2005.08.009.
Eghbali, Aryaz, and Michael Pradel. 2022. “CrystalBLEU: Precisely and Efficiently Measuring the Similarity of Code.” In 37th IEEE/ACM International Conference on Automated Software Engineering, ASE 2022, 28:1–12. ACM. https://doi.org/10.1145/3551349.3556903.
El-Hajjami, Abdelkarim, and Camille Salinesi. 2025. “How Good Are Synthetic Requirements? Evaluating LLM-Generated Datasets for AI4RE.” CoRR abs/2506.21138. https://doi.org/10.48550/ARXIV.2506.21138.
Gallegos, Isabel O., Ryan A. Rossi, Joe Barrow, Md. Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. 2024. “Bias and Fairness in Large Language Models: A Survey.” Comput. Linguistics 50 (3): 1097–1179. https://doi.org/10.1162/COLI\_A\_00524.
Gallifant, Jack, Majid Afshar, Saleem Ameen, Yindalon Aphinyanaphongs, Shan Chen, Giovanni Cacciamani, Dina Demner-Fushman, et al. 2025. “The TRIPOD-LLM Reporting Guideline for Studies Using Large Language Models.” Nature Medicine 31 (1): 60–69. https://doi.org/10.1038/s41591-024-03425-5.
Unchanged text
Galster, Matthias, Seyedmoein Mohsenimofidi, Jai Lal Lulla, Muhammad Auwal Abubakar, Christoph Treude, and Sebastian Baltes. 2026. “Configuring Agentic AI Coding Tools: An Exploratory Study.” In Proceedings of the 3rd ACM International Conference on AI-Powered Software (AIware 2026).
Gerosa, Marco Aurélio, Bianca Trinkenreich, Igor Steinmacher, and Anita Sarma. 2024. “Can AI Serve as a Substitute for Human Subjects in Software Engineering Research?” Autom. Softw. Eng. 31 (1): 13. https://doi.org/10.1007/S10515-023-00409-6.
Gibney, Elizabeth. 2024. “Not all ‘open source’ AI models are actually open.” Nature News. https://doi.org/10.1038/d41586-024-02012-5.
Gilardi, Fabrizio, Meysam Alizadeh, and Maël Kubli. 2023. “ChatGPT Outperforms Crowd Workers for Text-Annotation Tasks.” Proceedings of the National Academy of Sciences 120 (30): e2305016120. https://doi.org/10.1073/pnas.2305016120.
Gonçalves, José, Miguel Silva, Bernardo Cabral, Tiago Dias, Eva Maia, Isabel Praça, Ricardo Severino, and Luı́s Lino Ferreira. 2025. “Evaluating LLaMA 3.2 for Software Vulnerability Detection.” In Cybersecurity - 9th European Interdisciplinary Cybersecurity Conference, EICC 2025, Rennes, France, June 18-19, 2025, Proceedings, edited by Isabel Praça, Simona Bernardi, and Pedro R. M. Inácio, 38–51. Communications in Computer and Information Science. Springer. https://doi.org/10.1007/978-3-031-94855-8\_3.
Graziotin, Daniel. 2024. “Acmsigsoft/Open-Science-Policies: V1.0.0.” Zenodo. https://doi.org/10.5281/zenodo.10796477.
Unchanged text
Guba, Egon G. 1981. “Criteria for Assessing the Trustworthiness of Naturalistic Inquiries.” ECTJ 29 (2): 75–91. https://doi.org/10.1007/BF02766777.
Guo, Daya, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, et al. 2024. “DeepSeek-Coder: When the Large Language Model Meets Programming - the Rise of Code Intelligence.” CoRR abs/2401.14196. https://doi.org/10.48550/ARXIV.2401.14196.
Harding, Jacqueline, William D’Alessandro, N. G. Laskowski, and Robert Long. 2024. “AI Language Models Cannot Replace Human Research Participants.” AI Soc. 39 (5): 2603–5. https://doi.org/10.1007/S00146-023-01725-X.
He, Jia, Mukund Rungta, David Koleczek, Arshdeep Sekhon, Franklin X. Wang, and Sadid Hasan. 2024. “Does Prompt Formatting Have Any Impact on LLM Performance?” CoRR abs/2411.10541. https://doi.org/10.48550/ARXIV.2411.10541.
He, Junda, Jieke Shi, Terry Yue Zhuo, Christoph Treude, Jiamou Sun, Zhenchang Xing, Xiaoning Du, and David Lo. 2026. “LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead.” ACM Transactions on Software Engineering and Methodology. https://doi.org/10.1145/3797276.
He, Zeyu, Chieh-Yang Huang, Chien-Kuang Cornelia Ding, Shaurya Rohatgi, and Ting-Hao Kenneth Huang. 2024. “If in a Crowdsourced Data Annotation Pipeline, a GPT-4.” In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI 2024, edited by Florian ’Floyd’Mueller, Penny Kyburz, Julie R. Williamson, Corina Sas, Max L. Wilson, Phoebe O. Toups Dugas, and Irina Shklovski, 1040:1–25. ACM. https://doi.org/10.1145/3613904.3642834.
Holtzman, Ari, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. “The Curious Case of Neural Text Degeneration.” In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. https://openreview.net/forum?id=rygGQyrFvH.
Hou, Xinyi, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. “Large Language Models for Software Engineering: A Systematic Literature Review.” ACM Trans. Softw. Eng. Methodol. 33 (8): 220:1–79. https://doi.org/10.1145/3695988.
Hu, Xing, Feifei Niu, Junkai Chen, Xin Zhou, Junwei Zhang, Junda He, Xin Xia, and David Lo. 2025. “Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Tasks.” CoRR abs/2505.08903. https://doi.org/10.48550/ARXIV.2505.08903.
Unchanged text
Huang, Fan, Haewoon Kwak, and Jisun An. 2023. “Is ChatGPT Better Than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech.” In Companion Proceedings of the ACM Web Conference 2023, WWW 2023, edited by Ying Ding, Jie Tang, Juan F. Sequeda, Lora Aroyo, Carlos Castillo, and Geert-Jan Houben, 294–97. ACM. https://doi.org/10.1145/3543873.3587368.
Huang, Jiangping, Bochen Yi, Weisong Sun, Bangrui Wan, Yang Xu, Yebo Feng, Wenguang Ye, and Qinjun Qin. 2024. “Enhancing Review Classification via LLM-Based Data Annotation and Multi-Perspective Feature Representation Learning.” SSRN Electronic Journal, 1–15. https://doi.org/10.2139/ssrn.5002351.
Hui, Binyuan, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, et al. 2024. “Qwen2.5-Coder Technical Report.” CoRR abs/2409.12186. https://doi.org/10.48550/ARXIV.2409.12186.
Husain, Hamel, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. “CodeSearchNet Challenge: Evaluating the State of Semantic Code Search.” CoRR abs/1909.09436. http://arxiv.org/abs/1909.09436.
Hymel, Cory, and Hiroe Johnson. 2025. “Analysis of LLMs Vs Human Experts in Requirements Engineering.” CoRR abs/2501.19297. https://doi.org/10.48550/ARXIV.2501.19297.
Jahic, Jasmin, and Ashkan Sami. 2024. “State of Practice: LLMs in Software Engineering and Software Architecture.” In 21st IEEE International Conference on Software Architecture, ICSA 2024 - Companion, Hyderabad, India, June 4-8, 2024, 311–18. IEEE. https://doi.org/10.1109/ICSA-C63560.2024.00059.
Jain, Naman, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. “LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code.” CoRR abs/2403.07974. https://doi.org/10.48550/ARXIV.2403.07974.
Jimenez, Carlos E., John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. “SWE-bench: Can Language Models Resolve Real-World Github Issues?” In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum?id=VTF8yNQM66.
Jowsey, Tanisha, Virginia Braun, Victoria Clarke, Victoria Clarke, Deborah Lupton, and Michelle Fine. 2025. “We Reject the Use of Generative Artificial Intelligence for Reflexive Qualitative Research.” SSRN. https://doi.org/10.2139/ssrn.5676462.
Unchanged text
Kang, Sungmin, Juyeon Yoon, and Shin Yoo. 2023. “Large Language Models Are Few-Shot Testers: Exploring LLM-Based General Bug Reproduction.” In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, 2312–23. IEEE. https://doi.org/10.1109/ICSE48619.2023.00194.
Khojah, Ranim, Mazen Mohamad, Philipp Leitner, and Francisco Gomes de Oliveira Neto. 2024. “Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice.” Proc. ACM Softw. Eng. 1 (FSE): 1819–40. https://doi.org/10.1145/3660788.
Khraisha, Qusai, Sophie Put, Johanna Kappenberg, Azza Warraitch, and Kristin Hadfield. 2024. “Can Large Language Models Replace Humans in Systematic Reviews? Evaluating GPT-4’s Efficacy in Screening and Extracting Data from Peer-Reviewed and Grey Literature in Multiple Languages.” Research Synthesis Methods 15 (4): 616–26. https://doi.org/10.1002/jrsm.1715.
Kistowski, Jóakim von, Jeremy A. Arnold, Karl Huppler, Klaus-Dieter Lange, John L. Henning, and Paul Cao. 2015. “How to Build a Benchmark.” In Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering, Austin, TX, USA, January 31 - February 4, 2015, edited by Lizy K. John, Connie U. Smith, Kai Sachs, and Catalina M. Lladó, 333–36. ACM. https://doi.org/10.1145/2668930.2688819.
Kong, Aobo, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, Xin Zhou, Enzhi Wang, and Xiaohang Dong. 2024. “Better Zero-Shot Reasoning with Role-Play Prompting.” In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, edited by Kevin Duh, Helena Gómez-Adorno, and Steven Bethard, 4099–4113. Association for Computational Linguistics. https://doi.org/10.18653/V1/2024.NAACL-LONG.228.
Krippendorff, Klaus. 2018. Content Analysis: An Introduction to Its Methodology. 4th ed. SAGE Publications.
Kübler, Jonas M., Kailash Budhathoki, Matthäus Kleindessner, Xiong Zhou, Junming Yin, Ashish Khetan, and George Karypis. 2026. “When LLMs Get Significantly Worse: A Statistical Approach to Detect Model Degradations.” In Proc. Of the International Conference on Learning Representations (ICLR).
Kulal, Sumith, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. 2019. “SPoC: Search-Based Pseudocode to Code.” In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, edited by Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, 11883–94. https://proceedings.neurips.cc/paper/2019/hash/7298332f04ac004a0ca44cc69ecf6f6b-Abstract.html.
Li, David, Kartik Gupta, Mousumi Bhaduri, Paul Sathiadoss, Sahir Bhatnagar, and Jaron Chong. 2024. “Comparing GPT-3.5 and GPT-4 Accuracy and Drift in Radiology Diagnosis Please Cases.” Radiology 310 (1): e232411. https://doi.org/10.1148/radiol.232411.
Li, Raymond, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, et al. 2023. “StarCoder: May the Source Be with You!” Trans. Mach. Learn. Res. 2023. https://openreview.net/forum?id=KoFOg41haE.
Liang, Jenny T., Carmen Badea, Christian Bird, Robert DeLine, Denae Ford, Nicole Forsgren, and Thomas Zimmermann. 2024. “Can GPT-4 Replicate Empirical Software Engineering Research?” Proc. ACM Softw. Eng. 1 (FSE): 1330–53. https://doi.org/10.1145/3660767.
Liu, Jiawei, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. “Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation.” In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, edited by Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine. http://papers.nips.cc/paper\_files/paper/2023/hash/43e9d647ccd3e4b7b5baab53f0368686-Abstract-Conference.html.
Lu, Qinghua, Liming Zhu, Xiwei Xu, Zhenchang Xing, and Jon Whittle. 2024. “Toward Responsible AI in the Era of Generative AI: A Reference Architecture for Designing Foundation Model-Based Systems.” IEEE Softw. 41 (6): 91–100. https://doi.org/10.1109/MS.2024.3406333.
Unchanged text
Lubos, Sebastian, Alexander Felfernig, Thi Ngoc Trang Tran, Damian Garber, Merfat El Mansi, Seda Polat Erdeniz, and Viet-Man Le. 2024. “Leveraging LLMs for the Quality Assurance of Software Requirements.” In 32nd IEEE International Requirements Engineering Conference, RE 2024, Reykjavik, Iceland, June 24-28, 2024, edited by Grischa Liebel, Irit Hadar, and Paola Spoletini, 389–97. IEEE. https://doi.org/10.1109/RE59067.2024.00046.
Madampe, Kashumi, John Grundy, Rashina Hoda, and Humphrey O. Obie. 2024. “The Struggle Is Real! The Agony of Recruiting Participants for Empirical Software Engineering Studies.” In 2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), Liverpool, UK, September 2-6, 2024, 417–22. IEEE. https://doi.org/10.1109/VL/HCC60511.2024.00065.
McDonald, Nora, Sarita Schoenebeck, and Andrea Forte. 2019. “Reliability and Inter-Rater Reliability in Qualitative Research: Norms and Guidelines for CSCW and HCI Practice.” Proc. ACM Hum. Comput. Interact. 3 (CSCW): 72:1–23. https://doi.org/10.1145/3359174.
Menzies, Tim. 2025. “The Case for Compact AI.” Commun. ACM 68 (9): 6–7. https://doi.org/10.1145/3746057.
Microsoft. 2023. “CodeBERT on GitHub.” https://github.com/microsoft/CodeBERT.
Microsoft. 2025. “Azure OpenAI Service models.” https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models.
Mitu, Narcis Eduard, and George Teodor Mitu. 2024. “The Hidden Cost of AI: Carbon Footprint and Mitigation Strategies.” Revista de Stiinte Politice. Revue Des Sciences Politiques 84: 9–16. https://doi.org/10.2139/ssrn.5036344.
Mohamed, Suad, Abdullah Parvin, and Esteban Parra. 2024. “Chatting with AI: Deciphering Developer Conversations with ChatGPT.” In 21st IEEE/ACM International Conference on Mining Software Repositories, MSR 2024, Lisbon, Portugal, April 15-16, 2024, edited by Diomidis Spinellis, Alberto Bacchelli, and Eleni Constantinou, 187–91. ACM. https://doi.org/10.1145/3643991.3645078.
Mohsenimofidi, Seyedmoein, Matthias Galster, Christoph Treude, and Sebastian Baltes. 2026. “Context Engineering for AI Agents in Open-Source Software.” In Proceedings of the 23rd IEEE/ACM International Conference on Mining Software Repositories (MSR 2026). https://arxiv.org/abs/2510.21413.
Montes, Cristina Martinez, Robert Feldt, Cristina Miguel Martos, Sofia Ouhbi, Shweta Premanandan, and Daniel Graziotin. 2025. “Large Language Models in Thematic Analysis: Prompt Engineering, Evaluation, and Guidelines for Qualitative Software Engineering Research.” CoRR abs/2510.18456. https://doi.org/10.48550/ARXIV.2510.18456.
Morais Leça, Matheus de, Lucas Valença, Reydne Santos, and Ronnie de Souza Santos. 2025. “Applications and Implications of Large Language Models in Qualitative Analysis: A New Frontier for Empirical Software Engineering.” In IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering, WSESE@ICSE 2025, Ottawa, ON, Canada, May 3, 2025, 36–43. IEEE. https://doi.org/10.1109/WSESE66602.2025.00013.
Moumoula, Micheline Bénédicte, Abdoul Kader Kaboré, Jacques Klein, and Tegawendé F. Bissyandé. 2024. “Large Language Models for Cross-Language Code Clone Detection.” CoRR abs/2408.04430. https://doi.org/10.48550/ARXIV.2408.04430.
Open Source Initiative (OSI). 2025. “Open Source AI Definition 1.0.” https://opensource.org/ai/open-source-ai-definition.
OpenAI. 2023. “How to make your completions outputs consistent with the new seed parameter.” https://cookbook.openai.com/examples/reproducible_outputs_with_the_seed_parameter.
———. 2025. “OpenAI API Introduction.” https://platform.openai.com/docs/api-reference/chat/streaming.
OpenCode Contributors. 2025. “OpenCode: The Open Source AI Coding Agent.” https://opencode.ai/.
OpenTelemetry Authors. 2026. “OpenTelemetry Semantic Conventions for Generative AI.” https://opentelemetry.io/docs/specs/semconv/gen-ai/.
Ornelas, Tatiane, Allysson Allex Araújo, Júlia Araújo, Marina Araújo, Bianca Trinkenreich, and Marcos Kalinowski. 2025. “LLM-Assisted Thematic Analysis: Opportunities, Limitations, and Recommendations.” CoRR abs/2511.14528. https://doi.org/10.48550/ARXIV.2511.14528.
Pan, Qian, Zahra Ashktorab, Michael Desmond, Martin Santillan Cooper, James Johnson, Rahul Nair, Elizabeth Daly, and Werner Geyer. 2024. “Human-Centered Design Recommendations for LLM-as-a-Judge.” In ACL 2024 Workshop HuCLLM. arXiv. https://doi.org/10.48550/arXiv.2407.03479.
Pangakis, Nicholas, Samuel Wolken, and Neil Fasching. 2023. “Automated Annotation with Generative AI Requires Validation.” CoRR abs/2306.00176. https://doi.org/10.48550/ARXIV.2306.00176.
Panickssery, Arjun, Samuel R. Bowman, and Shi Feng. 2024. “LLM Evaluators Recognize and Favor Their Own Generations.” In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, edited by Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang. http://papers.nips.cc/paper\_files/paper/2024/hash/7f1f0218e45f5414c79c0679633e47bc-Abstract-Conference.html.
Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. “BLEU: A Method for Automatic Evaluation of Machine Translation.” In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, 311–18. ACL. https://doi.org/10.3115/1073083.1073135.
Pezeshkpour, Pouya, and Estevam Hruschka. 2024. “Large Language Models Sensitivity to the Order of Options in Multiple-Choice Questions.” In Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024, edited by Kevin Duh, Helena Gómez-Adorno, and Steven Bethard, NAACL 2024:2006–17. Findings of ACL. Association for Computational Linguistics. https://doi.org/10.18653/V1/2024.FINDINGS-NAACL.130.
Unchanged text
Rabbi, Md. Fazle, Arifa I. Champa, Minhaz Fahim Zibran, and Md. Rakibul Islam. 2024. “AI Writes, We Analyze: The ChatGPT Python Code Saga.” In 21st IEEE/ACM International Conference on Mining Software Repositories, MSR 2024, Lisbon, Portugal, April 15-16, 2024, edited by Diomidis Spinellis, Alberto Bacchelli, and Eleni Constantinou, 177–81. ACM. https://doi.org/10.1145/3643991.3645076.
Ralph, Paul, Nauman bin Ali, Sebastian Baltes, Domenico Bianculli, Jessica Diaz, Yvonne Dittrich, Neil Ernst, et al. 2021. “Empirical Standards for Software Engineering Research.” https://arxiv.org/abs/2010.03525.
Ralph, Paul, Miikka Kuutila, Hera Arif, and Bimpe Ayoola. 2024. “Teaching Software Metrology: The Science of Measurement for Software Engineering.” In Handbook on Teaching Empirical Software Engineering, edited by Daniel Méndez, Paris Avgeriou, Marcos Kalinowski, and Nauman Bin Ali, 101–54. Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-71769-7\_5.
Ralph, Paul, and Ewan D. Tempero. 2018. “Construct Validity in Software Engineering Research and Software Metrics.” In Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering, EASE2018, edited by Austen Rainer, Stephen G. MacDonell, and Jacky W. Keung, 13–23. ACM. https://doi.org/10.1145/3210459.3210461.
Rasheed, Zeeshan, Muhammad Waseem, Aakash Ahmad, Kai-Kristian Kemell, Xiaofeng Wang, Anh Nguyen-Duc, and Pekka Abrahamsson. 2024. “Can Large Language Models Serve as Data Analysts? A Multi-Agent Assisted Approach for Qualitative Data Analysis.” CoRR abs/2402.01386. https://doi.org/10.48550/ARXIV.2402.01386.
Reiss, Michael V. 2023. “Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark.” CoRR abs/2304.11085. https://doi.org/10.48550/ARXIV.2304.11085.
Ren, Shuo, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. “CodeBLEU: A Method for Automatic Evaluation of Code Synthesis.” CoRR abs/2009.10297. https://arxiv.org/abs/2009.10297.
Renze, Matthew. 2024. “The Effect of Sampling Temperature on Problem Solving in Large Language Models.” In Findings of the Association for Computational Linguistics: EMNLP 2024, 7346–56. https://doi.org/10.18653/v1/2024.findings-emnlp.432.
Richards, Jonan, and Mairieli Wessel. 2024. “What You Need Is What You Get: Theory of Mind for an LLM-Based Code Understanding Assistant.” In IEEE International Conference on Software Maintenance and Evolution, ICSME 2024, 666–71. IEEE. https://doi.org/10.1109/ICSME58944.2024.00070.
Ronanki, Krishna, Christian Berger, and Jennifer Horkoff. 2023. “Investigating ChatGPT’s Potential to Assist in Requirements Elicitation Processes.” In 2023 49th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), 354–61. IEEE.
Rozière, Baptiste, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, et al. 2023. “Code Llama: Open Foundation Models for Code.” CoRR abs/2308.12950. https://doi.org/10.48550/ARXIV.2308.12950.
Rozière, Baptiste, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. 2020. “Unsupervised Translation of Programming Languages.” In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, Virtual, edited by Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin. https://proceedings.neurips.cc/paper/2020/hash/ed23fbf18c2cd35f8c7f8de44f85c08d-Abstract.html.
Russo, Daniel. 2024. “Navigating the Complexity of Generative AI Adoption in Software Engineering.” ACM Trans. Softw. Eng. Methodol. 33 (5): 135:1–50. https://doi.org/10.1145/3652154.
Sallou, June, Thomas Durieux, and Annibale Panichella. 2024. “Breaking the Silence: The Threats of Using LLMs in Software Engineering.” In Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results, NIER@ICSE 2024, Lisbon, Portugal, April 14-20, 2024, 102–6. ACM. https://doi.org/10.1145/3639476.3639764.
Unchanged text
Santos, Adrian, Sira Vegas, Markku Oivo, and Natalia Juristo. 2021. “A Procedure and Guidelines for Analyzing Groups of Software Engineering Replications.” IEEE Trans. Software Eng. 47 (9): 1742–63. https://doi.org/10.1109/TSE.2019.2935720.
Schäfer, Max, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024. “An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation.” IEEE Trans. Software Eng. 50 (1): 85–105. https://doi.org/10.1109/TSE.2023.3334955.
Schilke, Oliver, and Martin Reimann. 2025. “The Transparency Dilemma: How AI Disclosure Erodes Trust.” Organizational Behavior and Human Decision Processes 188: 104405. https://doi.org/10.1016/j.obhdp.2025.104405.
Schneider, Kurt, Farnaz Fotrousi, and Rebekka Wohlrab. 2025. “A Reference Model for Empirically Comparing LLMs with Humans.” In 47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Society, ICSE-SEIS 2025, Ottawa, ON, Canada, April 27 - May 3, 2025, 130–34. IEEE. https://doi.org/10.1109/ICSE-SEIS66351.2025.00018.
Schröder, Sarah, Thekla Morgenroth, Ulrike Kuhl, Valerie Vaquet, and Benjamin Paaßen. 2025. “Large Language Models Do Not Simulate Human Psychology.” CoRR abs/2508.06950. https://doi.org/10.48550/ARXIV.2508.06950.
Schroeder, Kayla, and Zach Wood-Doughty. 2024. “Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge.” CoRR abs/2412.12509. https://doi.org/10.48550/ARXIV.2412.12509.
Schulhoff, Sander, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, et al. 2024. “The Prompt Report: A Systematic Survey of Prompting Techniques.” CoRR abs/2406.06608. https://doi.org/10.48550/ARXIV.2406.06608.
Schulz, Kenneth F., Douglas G. Altman, and David Moher. 2010. “CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomised Trials.” BMJ 340: c332. https://doi.org/10.1136/bmj.c332.
Schwartz, Roy, Jesse Dodge, Noah A. Smith, and Oren Etzioni. 2020. “Green AI.” Communications of the ACM 63 (12): 54–63. https://doi.org/10.1145/3381831.
Sclar, Melanie, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. “Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I Learned to Start Worrying about Prompt Formatting.” In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum?id=RIu5lyNXjT.
Shull, Forrest, Janice Singer, and Dag I. K. Sjøberg, eds. 2008. Guide to Advanced Empirical Software Engineering. Springer. https://doi.org/10.1007/978-1-84800-044-5.
Silva, André, and Martin Monperrus. 2025. “RepairBench: Leaderboard of Frontier Models for Program Repair.” In IEEE/ACM International Workshop on Large Language Models for Code, LLM4Code@ICSE 2025, Ottawa, ON, Canada, May 3, 2025, 9–16. IEEE. https://doi.org/10.1109/LLM4CODE66737.2025.00006.
Sjøberg, Dag I. K., and Gunnar Rye Bergersen. 2023. “Construct Validity in Software Engineering.” IEEE Trans. Software Eng. 49 (3): 1374–96. https://doi.org/10.1109/TSE.2022.3176725.
Song, Yifan, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. 2025. “The Good, the Bad, and the Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism.” In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2025 - Volume 1: Long Papers, edited by Luis Chiruzzo, Alan Ritter, and Lu Wang, 4195–4206. Association for Computational Linguistics. https://doi.org/10.18653/V1/2025.NAACL-LONG.211.
Sonnekalb, Tim, Bernd Gruner, Clemens-Alexander Brust, and Patrick Mäder. 2022. “Generalizability of Code Clone Detection on CodeBERT.” In 37th IEEE/ACM International Conference on Automated Software Engineering, ASE 2022, 143:1–3. ACM. https://doi.org/10.1145/3551349.3561165.
Stack Overflow. 2025. “2025 Developer Survey.” https://survey.stackoverflow.co/2025/.
Strubell, Emma, Ananya Ganesh, and Andrew McCallum. 2019. “Energy and Policy Considerations for Deep Learning in NLP.” In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, edited by Anna Korhonen, David R. Traum, and Lluı́s Màrquez, 3645–50. Association for Computational Linguistics. https://doi.org/10.18653/V1/P19-1355.
Sullivan, Gail M, and Joan Sargeant. 2011. “Qualities of Qualitative Research: Part I.” J Grad Med Educ 3 (4): 449–52.
Sumers, Theodore R., Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. 2024. “Cognitive Architectures for Language Agents.” Trans. Mach. Learn. Res. 2024. https://openreview.net/forum?id=1i6ZCvflQJ.
Takerngsaksiri, Wannita, Jirat Pasuksmit, Patanamon Thongtanunam, Chakkrit Tantithamthavorn, Ruixiong Zhang, Fan Jiang, Jing Li, Evan Cook, Kun Chen, and Ming Wu. 2025. “Human-in-the-Loop Software Development Agents.” In 47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice, SEIP@ICSE 2025, Ottawa, ON, Canada, April 27 - May 3, 2025, 342–52. IEEE. https://doi.org/10.1109/ICSE-SEIP66354.2025.00036.
Tam, Thomas Yu Chow, Sonish Sivarajkumar, Sumit Kapoor, Alisa V. Stolyar, Katelyn Polanska, Karleigh R. McCarthy, Hunter Osterhoudt, et al. 2024. “A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Review.” Npj Digit. Medicine 7 (1). https://doi.org/10.1038/S41746-024-01258-7.
Wagner, Stefan, Marvin Muñoz Barón, Davide Falessi, and Sebastian Baltes. 2025. “Towards Evaluation Guidelines for Empirical Studies Involving LLMs.” In IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering, WSESE@ICSE 2025, May 3, 2025, 24–27. IEEE. https://doi.org/10.1109/WSESE66602.2025.00011.
Wan, Mengting, Tara Safavi, Sujay Kumar Jauhar, Yujin Kim, Scott Counts, Jennifer Neville, Siddharth Suri, et al. 2024. “TnT-LLM: Text Mining at Scale with Large Language Models.” In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2024, edited by Ricardo Baeza-Yates and Francesco Bonchi, 5836–47. ACM. https://doi.org/10.1145/3637528.3671647.
Wang, Angelina, Jamie Morgenstern, and John P. Dickerson. 2025. “Large Language Models That Replace Human Participants Can Harmfully Misportray and Flatten Identity Groups.” Nat. Mac. Intell. 7 (3): 400–411. https://doi.org/10.1038/S42256-025-00986-Z.
Wang, Chong, Kaifeng Huang, Jian Zhang, Yebo Feng, Lyuye Zhang, Yang Liu, and Xin Peng. 2024. “How and Why LLMs Use Deprecated APIs in Code Completion? An Empirical Study.” CoRR abs/2406.09834. https://doi.org/10.48550/ARXIV.2406.09834.
Unchanged text
Wang, Fanyu, Chetan Arora, Yonghui Liu, Kaicheng Huang, Chakkrit Tantithamthavorn, Aldeida Aleti, Dishan Sambathkumar, and David Lo. 2025. “Multi-Modal Requirements Data-Based Acceptance Criteria Generation Using LLMs.” In 40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025, 3334–45. IEEE. https://doi.org/10.1109/ASE63991.2025.00275.
Wang, Shuohang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. “Want to Reduce Labeling Cost? GPT-3 Can Help.” In Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, edited by Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, EMNLP 2021:4195–205. Findings of ACL. Association for Computational Linguistics. https://doi.org/10.18653/V1/2021.FINDINGS-EMNLP.354.
Wang, Xinru, Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjie Miao. 2024. “Human-LLM Collaborative Annotation Through Effective Verification of LLM Labels.” In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI 2024, edited by Florian ’Floyd’Mueller, Penny Kyburz, Julie R. Williamson, Corina Sas, Max L. Wilson, Phoebe O. Toups Dugas, and Irina Shklovski, 303:1–21. ACM. https://doi.org/10.1145/3613904.3641960.
Widder, David Gray, Meredith Whittaker, and Sarah Myers West. 2024. “Why ‘Open’AI Systems Are Actually Closed, and Why This Matters.” Nature 635 (8040): 827–33.
Wiesinger, Julia, Patrick Marlow, and Vladimir Vuskovic. 2025. “Agents.” Google DeepMind. https://gemini.google.com.
Willison, Simon. 2026. “Opus system prompt.” https://simonwillison.net/2026/Apr/18/opus-system-prompt/.
Wohlin, Claes, Per Runeson, Martin Höst, Magnus C. Ohlsson, Björn Regnell, and Anders Wesslén. 2024. Experimentation in Software Engineering, Second Edition. Springer. https://doi.org/10.1007/978-3-662-69306-3.
Xia, Yuying, Haijian Shao, and Xing Deng. 2024. “VulCoBERT: A CodeBERT-Based System for Source Code Vulnerability Detection.” In 2024 International Conference on Generative Artificial Intelligence and Information Security, GAIIS 2024, Kuala Lumpur, Malaysia, May 10-12, 2024. ACM. https://doi.org/10.1145/3665348.3665391.
Xiao, Tao, Christoph Treude, Hideaki Hata, and Kenichi Matsumoto. 2024. “DevGPT: Studying Developer-ChatGPT Conversations.” In 21st IEEE/ACM International Conference on Mining Software Repositories, MSR 2024, Lisbon, Portugal, April 15-16, 2024, edited by Diomidis Spinellis, Alberto Bacchelli, and Eleni Constantinou, 227–30. ACM. https://doi.org/10.1145/3643991.3648400.
Xu, Cheng, Shuhao Guan, Derek Greene, and M. Tahar Kechadi. 2024. “Benchmark Data Contamination of Large Language Models: A Survey.” CoRR abs/2406.04244. https://doi.org/10.48550/ARXIV.2406.04244.
Xu, Ruoxi, Yingfei Sun, Mengjie Ren, Shiguang Guo, Ruotong Pan, Hongyu Lin, Le Sun, and Xianpei Han. 2024. “AI for Social Science and Social Science of AI: A Survey.” Inf. Process. Manag. 61 (2): 103665. https://doi.org/10.1016/J.IPM.2024.103665.
Yan, Litao, Alyssa Hwang, Zhiyuan Wu, and Andrew Head. 2024. “Ivie: Lightweight Anchored Explanations of Just-Generated Code.” In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI 2024, edited by Florian ’Floyd’Mueller, Penny Kyburz, Julie R. Williamson, Corina Sas, Max L. Wilson, Phoebe O. Toups Dugas, and Irina Shklovski, 140:1–15. ACM. https://doi.org/10.1145/3613904.3642239.
Yang, Guang, Yu Zhou, Xiang Chen, Xiangyu Zhang, Tingting Han, and Taolue Chen. 2023. “ExploitGen: Template-Augmented Exploit Code Generation Based on CodeBERT.” J. Syst. Softw. 197: 111577. https://doi.org/10.1016/J.JSS.2022.111577.
Yang, Yifan, Jiho Shin, Byeonggyu Choi, Minjun Park, Dayun Ju, Changmin Lee, Sanghyuk Chun, Dongjin Kang, Jiin Kim, and Sungroh Yoon. 2025. “From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence.” arXiv Preprint arXiv:2502.11827.
Yuan, Jiayi, Hao Li, Xinheng Ding, Wenya Xie, Yu-Jhe Li, Wentian Zhao, Kun Wan, Jing Shi, Xia Hu, and Zirui Liu. 2025. “Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference.” In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025. https://openreview.net/forum?id=Q3qAsZAEZw.
Zhao, Chenguang, Meirewuti Habule, and Wei Zhang. 2025. “Large Language Models (LLMs) as Research Subjects: Status, Opportunities and Challenges.” New Ideas in Psychology 79: 101167. https://doi.org/10.1016/j.newideapsych.2025.101167.
Unchanged text
Zhou, Ruiyang, Lu Chen, and Kai Yu. 2024. “Is LLM a Reliable Reviewer? A Comprehensive Evaluation of LLM on Automatic Paper Reviewing Tasks.” In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, edited by Nicoletta Calzolari, Min-Yen Kan, Véronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, 9340–51. ELRA; ICCL. https://aclanthology.org/2024.lrec-main.816.
Zhou, Wangchunshu, Yuchen Eleanor Jiang, Long Li, Jialong Wu, Tiannan Wang, Shi Qiu, Jintian Zhang, et al. 2023. “Agents: An Open-Source Framework for Autonomous Language Agents.” CoRR abs/2309.07870. https://doi.org/10.48550/ARXIV.2309.07870.
Zhu, Yiming, Peixian Zhang, Ehsan ul Haq, Pan Hui, and Gareth Tyson. 2023. “Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks.” CoRR abs/2304.10145. https://doi.org/10.48550/ARXIV.2304.10145.
Unchanged sections (1)
Acknowledgments Unchanged

2026.04

Acknowledgments

Our study types and guidelines are based on discussion sessions with researchers at the 2024 International Software Engineering Research Network (ISERN) meeting and at the 2nd Copenhagen Symposium on Human-Centered Software Engineering AI (supported by the Carlsberg Foundation with grant CF24-0693 and the Alfred P. Sloan Foundation with grant G-2024-22586 to Daniel Russo). LLM-based tools, including ChatGPT (GPT-5) and Claude Code (Opus 4.6 and 4.7), were used interactively for language editing and structural revision suggestions across all sections of this paper. All LLM-generated suggestions were reviewed and revised by the authors.

2026.05

Acknowledgments

Our study types and guidelines are based on discussion sessions with researchers at the 2024 International Software Engineering Research Network (ISERN) meeting and at the 2nd Copenhagen Symposium on Human-Centered Software Engineering AI (supported by the Carlsberg Foundation with grant CF24-0693 and the Alfred P. Sloan Foundation with grant G-2024-22586 to Daniel Russo). LLM-based tools, including ChatGPT (GPT-5) and Claude Code (Opus 4.6 and 4.7), were used interactively for language editing and structural revision suggestions across all sections of this paper. All LLM-generated suggestions were reviewed and revised by the authors.