June 9, 2019
Publications
>
Article
Green

Understanding replication of experiments in software engineering: A classification

Publicated to: INFORMATION AND SOFTWARE TECHNOLOGY. 56 (8): 1033-1048 - 2014-08-01 56(8), DOI: 10.1016/j.infsof.2014.04.004

Authors:

Gómez, OS; Juristo, N; Vegas, S
[+]

Affiliations

Univ Autonoma Yucatan, Fac Matemat, Merida 97119 - Author
Univ Oulu, Dept Informat Proc Sci - Author
Univ Politecn Madrid, Fac Informat - Author
See more

Abstract

Context: Replication plays an important role in experimental disciplines. There are still many uncertainties about how to proceed with replications of SE experiments. Should replicators reuse the baseline experiment materials? How much liaison should there be among the original and replicating experimenters, if any? What elements of the experimental configuration can be changed for the experiment to be considered a replication rather than a new experiment? Objective: To improve our understanding of SE experiment replication, in this work we propose a classification which is intend to provide experimenters with guidance about what types of replication they can perform. Method: The research approach followed is structured according to the following activities: (1) a literature review of experiment replication in SE and in other disciplines, (2) identification of typical elements that compose an experimental configuration, (3) identification of different replications purposes and (4) development of a classification of experiment replications for SE. Results: We propose a classification of replications which provides experimenters in SE with guidance about what changes can they make in a replication and, based on these, what verification purposes such a replication can serve. The proposed classification helped to accommodate opposing views within a broader framework, it is capable of accounting for less similar replications to more similar ones regarding the baseline experiment. Conclusion: The aim of replication is to verify results, but different types of replication serve special verification purposes and afford different degrees of change. Each replication type helps to discover particular experimental conditions that might influence the results. The proposed classification can be used to identify changes in a replication and, based on these, understand the level of verification. (C) 2014 Elsevier B.V. All rights reserved.
[+]

Keywords

Experimental software engineeringExperimentationNeedReplicationSoftware engineering

Quality index

Bibliometric impact. Analysis of the contribution and dissemination channel

The work has been published in the journal INFORMATION AND SOFTWARE TECHNOLOGY due to its progression and the good impact it has achieved in recent years, according to the agency Scopus (SJR), it has become a reference in its field. In the year of publication of the work, 2014, it was in position , thus managing to position itself as a Q1 (Primer Cuartil), in the category Information Systems. Notably, the journal is positioned above the 90th percentile.

From a relative perspective, and based on the normalized impact indicator calculated from World Citations provided by WoS (ESI, Clarivate), it yields a value for the citation normalization relative to the expected citation rate of: 3.79. This indicates that, compared to works in the same discipline and in the same year of publication, it ranks as a work cited above average. (source consulted: ESI Nov 13, 2025)

This information is reinforced by other indicators of the same type, which, although dynamic over time and dependent on the set of average global citations at the time of their calculation, consistently position the work at some point among the top 50% most cited in its field:

  • Weighted Average of Normalized Impact by the Scopus agency: 3.29 (source consulted: FECYT Mar 2025)

Specifically, and according to different indexing agencies, this work has accumulated citations as of 2026-04-24, the following number of citations:

  • WoS: 94
  • Scopus: 111
  • Google Scholar: 144
[+]

Impact and social visibility

From the perspective of influence or social adoption, and based on metrics associated with mentions and interactions provided by agencies specializing in calculating the so-called "Alternative or Social Metrics," we can highlight as of 2026-04-24:

  • The use of this contribution in bookmarks, code forks, additions to favorite lists for recurrent reading, as well as general views, indicates that someone is using the publication as a basis for their current work. This may be a notable indicator of future more formal and academic citations. This claim is supported by the result of the "Capture" indicator, which yields a total of: 104 (PlumX).

With a more dissemination-oriented intent and targeting more general audiences, we can observe other more global scores such as:

    It is essential to present evidence supporting full alignment with institutional principles and guidelines on Open Science and the Conservation and Dissemination of Intellectual Heritage. A clear example of this is:

    • The work has been submitted to a journal whose editorial policy allows open Open Access publication.
    • Assignment of a Handle/URN as an identifier within the deposit in the Institutional Repository: https://oa.upm.es/37429/

    As a result of the publication of the work in the institutional repository, statistical usage data has been obtained that reflects its impact. In terms of dissemination, we can state that, as of

    • Views: 448
    • Downloads: 1,358
    [+]

    Leadership analysis of institutional authors

    This work has been carried out with international collaboration, specifically with researchers from: Finland; Mexico.

    There is a significant leadership presence as some of the institution’s authors appear as the first or last signer, detailed as follows: Last Author (VEGAS HERNANDEZ, SIRA).

    [+]

    Awards linked to the item

    This work was supported by research grant TIN2011-23216 of the Spanish Ministry of Economy and Competitiveness, and research grant 206747 of the Mexico's National Council of Science and Technology (CONACyT).
    [+]