Benefitting from Grey Literature in Software Engineering Research


Grey literature is becoming more and more important as a source of knowledge because software engineering practitioners write and share information in different forms of grey literature (GL) like blogs, videos or white papers. The overall goal of this tutorial is to present ways how software engineering research can benefit from the vast amount of information covered by GL. The participants of this tutorial will learn how GL can be used for various aspects of software engineering research, e.g., shaping new directions of research, or using knowledge and evidence from grey literature in empirical studies in software engineering. First, the concept of GL in general and from the perspective of different disciplines like health sciences or social sciences are presented. Second, the concept of GL in software engineering and types of GL are presented and discussed with the participants. Third, ways how GL can be used in primary studies and secondary studies in software engineering are presented. The discussed application scenarios in primary studies comprise analysis of GL materials with a qualitative approach, analysis of GL with a quantitative approach, and reference of GL sources. In secondary studies, GL can be incorporated into multivocal literature reviews and grey literature reviews. The instructors will present their guidelines for these reviews and explore them with the participants of the tutorial. Finally, the challenges and benefits of using GL in software engineering are discussed. The examples presented during the tutorial are from the domains of software processes and software testing. It is sufficient if potential participants are interested in the topic and intend to use grey literature in their empirical studies in software engineering. Knowledge of systematic literature review and empirical studies in software engineering are and advantage but not required. Participants should bring their notebook or tablet to enable participation in the practical part.


Michael Felderer is a professor at the Department of Computer Science at the University of Innsbruck, Austria and a guest professor at the Department of Software Engineering at the Blekinge Institute of Technology, Sweden. His fields of expertise and interest include software quality, testing, software and security processes, risk management, data-driven engineering, software analytics and measurement, requirements engineering, model-based software engineering and empirical research methodology in software and security engineering. Michael Felderer holds a habilitation degree from the University of Innsbruck, co-authored more than 130 publications and received nine best paper awards. His research has a strong empirical focus also using methods of data science and is directed towards development and evaluation of efficient and effective methods to improve quality and value of industrial software systems and processes in close collaboration with companies. Michael Felderer himself has more than ten years of industrial experience as a senior executive consultant, project manager, and software engineer. He has been supporting the software engineering research community as an editorial board member of the journal Information and Software Technology (IST), organizer of conferences (e.g., General Chair of PROFES 2017, Program Co-Chair of SEAA 2017, Industry Co-Chair of ESEC/FSE 2019) and regular PC member of premier conferences (e.g., ESEM, ICST, EASE or ICSSP). Furthermore, he is also a member of the steering committee of PROFES, RET, TAIC PART, and ISERN. Michael Felderer is an experienced educator. He has given many courses on software engineering topics in general at several universities and research methodology in particular over the last ten year. He has also already given tutorials at software engineering conferences including ESEM 2017 and PROFES 2016. Finally, he has already given a tutorial on grey literature and multivocal literature reviews in software engineering as part of an seminar at the University of Oulu in Finland in 2019. For more information, visit his website at

Vahid Garousi is an Associate Professor of Software Engineering in the Information Technology Group of Wageningen University, Netherlands. Previously, he was an Associate Professor of Software Engineering at Hacettepe University in Ankara, Turkey (2015-2017) and an Associate Professor of Software Engineering at the University of Calgary, Canada (2006-2014). Vahid received his PhD in Software Engineering at Carleton University, Canada, in 2006. His research interests in software engineering include software testing, empirical studies, search-based software testing, action-research, and industry-academia collaborations. Vahid was selected as a Distinguished Speaker for the IEEE Computer Society from 2012 to 2015. He is a member of the IEEE and the IEEE Computer Society and is also a licensed professional engineer (PEng) in the Canadian province of Alberta. Vahid Garousi has taught courses on Research Methods in Software Engineering. Furthermore, he has published numerous systematic literature studies, including MLRs, with his previous students.

Mika Mäntylä is a professor of Software Engineering at the University of Oulu, Finland. He received a D. Sc. degree in 2009 in software engineering from the Helsinki University of Technology, Finland. His research interests include empirical software engineering, software testing, software maintenance, mining software repositories, and behavioral software engineering. He has previously worked as a post-doc at the Lund University, Sweden and as an assistant professor at the Aalto University, Finland. His studies have appeared in journals such as IEEE Transaction on Software Engineering, IEEE Software, Empirical Software Engineering, and Information and Software Technology. He currently serves as an associated editor for IEEE Software. Mika Mäntylä has taught a yearly course in Emerging Trends in Software Engineering at the University of Oulu, Finland. In the course, students perform bibliometrics analysis and natural language processing both to academic literature, accessed from Scopus, and to grey literature, accessed from Scopus and Twitter. For more information

Austen Rainer worked in academia for about 20 years. He co-authored the first discipline-specific book on case study research in software engineering. In 2018 he was the General Chair for the EASE’18 conference. Austen was also the Founding Chair of Software Innovation NZ, which is the national software engineering network in New Zealand and which he chaired for two years. He was Head of the Department of Computer Science and Software Engineering at the University of Canterbury for two years (2015 – 2017). He has published about 70-ish peer-reviewed papers. Austen Rainer has taught courses on empirical evaluation in software engineering for ten years. He published an initial paper at EASE 2006 and then a follow-up paper at EASE 2008. He returned to this research area in 2016, publishing about ten papers since 2016 on topics related to grey literature reviews (a part of multivocal literature reviews).