Study Types

Empirical guidelines are needed for SE studies involving LLMs to ensure the validity and reproducibility of their results. However, such guidelines must be tailored to different study types that each pose unique challenges. Therefore, we created a taxonomy of study types to contextualize our recommendations. We use the term study type to refer to categories of LLM involvement in empirical research, rather than to research methodologies (e.g., experiments, case studies). A single empirical study may involve multiple such study types. Each study type section starts with a description, followed by examples from the SE research community and beyond. The advantages and challenges of using LLMs are discussed in a summary subsection at the end of this section, synthesizing cross-cutting themes across all study types.

---

Table of contents

• LLMs for Research
• S1: LLMs as Annotators
• S2: LLMs as Judges
• S3: LLMs for Synthesis
• S4: LLMs as Subjects
• LLMs for SE
• S5: Studying LLM Usage
• S6: LLMs for Tools
• S7: Benchmarking LLMs
• Advantages and Challenges