{rfName}
De

Indexed in

License and use

Citations

14

Altmetrics

Article has an altmetric score of 7

Grant support

This work was supported by the Universidad Politecnica de Madrid through the project Research-based teaching in the DevOps domain under Grant IE1920.6104.

Analysis of institutional authors

Perez, JeAuthorGonzalez-Prieto, AAuthorDiaz, JCorresponding AuthorLopez-Fernandez, DAuthorGarcia-Martin, JAuthorYague, AAuthor

Share

Publications
>
Article

DevOps Research-Based Teaching Using Qualitative Research and Inter-Coder Agreement

Publicated to:Ieee Transactions On Software Engineering. 48 (9): 3378-3393 - 2022-01-01 48(9), DOI: 10.1109/TSE.2021.3092705

Authors: Perez, Jorge E.; Gonzalez-Prieto, Angel; Diaz, Jessica; Lopez-Fernandez, Daniel; Garcia-Martin, Javier; Yague, Agustin;

Affiliations

Univ Autonoma Madrid, Dept Matemat, E-28049 Madrid, Spain - Author
Univ Autonoma Madrid, Inst Ciencias Matemat CSIC UAM UC3M UCM, E-28049 Madrid, Spain - Author
Univ Politecn Madrid, ETSI Sistemas Informat, Madrid 28031, Spain - Author

Abstract

DevOps is becoming a main competency required by the software industry. However, academic institutions have been slow to provide DevOps training in software engineering (SE) curricula. One reason for this is the fact that the problems addressed by DevOps may be hard to understand to students who have not previously worked in the industry or on projects of meaningful size and complexity. This paper shows an experience that integrates DevOps in SE curricula through research-based teaching (RBT). We aim to expose students to the problems that have led companies to adopt DevOps by researching and analyzing real cases of companies, thereby placing students at the center of learning. The contribution of this work is to innovate the application of RBT in software engineering by showing that the RBT approach is, at least, as good as the traditional approach and that it also leads to some extra benefits. This innovative solution has been implemented by using (i) qualitative analysis, specifically coding techniques, to discover knowledge and (ii) inter-coder agreement (ICA), specifically Krippendorff's alpha coefficients, to measure the extent of students' learning. These techniques allow teachers to determine whether students' learning in the subject is homogeneous and to analyze disagreements among students during their analysis. This approach provides teachers with new tools (Krippendorff's a coefficients) to identify those concepts that are less understood by students and to evaluate whether improvements in the research instruments (e.g., the codebook used in the qualitative analysis) also generate improvements in the students' agreement. This RBT experience shows evidence that can be used to assess whether a similar experience and the use of ICA could be applied in similar learning contexts with similar research contexts.

Keywords

DevopsInter-coder agreementQualitative researchReliabilityResearch-based teachingStudents

Quality index

Bibliometric impact. Analysis of the contribution and dissemination channel

The work has been published in the journal Ieee Transactions On Software Engineering due to its progression and the good impact it has achieved in recent years, according to the agency WoS (JCR), it has become a reference in its field. In the year of publication of the work, 2022, it was in position 5/108, thus managing to position itself as a Q1 (Primer Cuartil), in the category Computer Science, Software Engineering. Notably, the journal is positioned above the 90th percentile.

From a relative perspective, and based on the normalized impact indicator calculated from the Field Citation Ratio (FCR) of the Dimensions source, it yields a value of: 4.34, which indicates that, compared to works in the same discipline and in the same year of publication, it ranks as a work cited above average. (source consulted: Dimensions Jun 2025)

Specifically, and according to different indexing agencies, this work has accumulated citations as of 2025-06-10, the following number of citations:

  • WoS: 1
  • Scopus: 3
  • Google Scholar: 10
  • OpenCitations: 4

Impact and social visibility

From the perspective of influence or social adoption, and based on metrics associated with mentions and interactions provided by agencies specializing in calculating the so-called "Alternative or Social Metrics," we can highlight as of 2025-06-10:

  • The use, from an academic perspective evidenced by the Altmetric agency indicator referring to aggregations made by the personal bibliographic manager Mendeley, gives us a total of: 39.
  • The use of this contribution in bookmarks, code forks, additions to favorite lists for recurrent reading, as well as general views, indicates that someone is using the publication as a basis for their current work. This may be a notable indicator of future more formal and academic citations. This claim is supported by the result of the "Capture" indicator, which yields a total of: 38 (PlumX).

With a more dissemination-oriented intent and targeting more general audiences, we can observe other more global scores such as:

  • The Total Score from Altmetric: 6.55.
  • The number of mentions on the social network X (formerly Twitter): 7 (Altmetric).

Leadership analysis of institutional authors

There is a significant leadership presence as some of the institution’s authors appear as the first or last signer, detailed as follows: First Author (PEREZ MARTINEZ, JORGE ENRIQUE) and Last Author (YAGUE PANADERO, AGUSTIN).

the author responsible for correspondence tasks has been DIAZ FERNANDEZ, JESSICA.