LLM Guidelines for SE

Evaluation Guidelines for Empirical Studies in Software Engineering involving LLMs.

This website hosts a DRAFT of community guidlines for empirical studies in software engineering involving LLMs. Besides our motivation and scope, we present a first taxonomy of LLM study types and corresponding guidelines.

This project was initiated by a position paper as well as discussion during the ISERN 2024 meeting and the 2nd Copenhagen Symposium on Human-Centered Software Engineering AI. To contribute to the guidelines, you can open an issue or a pull request in our GitHub repository.

Workstream Leads:

• Sebastian Baltes, University of Bayreuth (Germany)
• Stefan Wagner, Technical University of Munich (Germany)

Team:

• Paris Avgeriou, University of Groningen (Netherlands)
• Marvin Muñoz Barón, Technical University of Munich (Germany)
• Lukas Böhme, HPI, University of Potsdam (Germany)
• Fabio Calefato, University of Bari (Italy)
• Chunyang Chen, Technical University of Munich (Germany)
• Neil Ernst, University of Victoria (Canada)
• Davide Falessi, University of Rome Tor Vergata (Italy)
• Brian Fitzgerald, Lero and University of Limerick (Ireland)
• Davide Fucci, Blekinge Institute of Technology (Sweden)
• Marcos Kalinowski, Pontifical Catholic University of Rio de Janeiro (Brazil)
• Stefano Lambiase, University of Salerno (Italy)
• Mircea Lungu, IT University of Copenhagen (Denmark)
• Margaret-Anne Storey, University of Victoria (Canada)
• Christoph Treude, Singapore Management University (Singapore)