{rfName}
Ma

Indexed in

License and use

Icono OpenAccess

Citations

10

Altmetrics

Article has an altmetric score of 1

Grant support

This study is supported by the European Union's Horizon 2020 research and innovation program under Grant Agreement no 779963, project Eurobench.

Analysis of institutional authors

Torres-Pardo, AAuthorPinto, DAuthor

Share

Publications
>
Article

Making Bipedal Robot Experiments Reproducible and Comparable: The Eurobench Software Approach

Publicated to:Frontiers In Robotics And Ai. 9 951663- - 2022-01-01 9(), DOI: 10.3389/frobt.2022.951663

Authors: Remazeilles, A; Dominguez, A; Barralon, P; Torres-Pardo, A; Pinto, D; Aller, F; Mombaur, K; Conti, R; Saccares, L; Thorsteinsson, F; Prinsen, E; Canton, A; Castilla, J; Sanz-Morere, CB; Tornero, J; Torricelli, D

Affiliations

Abstract

This study describes the software methodology designed for systematic benchmarking of bipedal systems through the computation of performance indicators from data collected during an experimentation stage. Under the umbrella of the European project Eurobench, we collected approximately 30 protocols with related testbeds and scoring algorithms, aiming at characterizing the performances of humanoids, exoskeletons, and/or prosthesis under different conditions. The main challenge addressed in this study concerns the standardization of the scoring process to permit a systematic benchmark of the experiments. The complexity of this process is mainly due to the lack of consistency in how to store and organize experimental data, how to define the input and output of benchmarking algorithms, and how to implement these algorithms. We propose a simple but efficient methodology for preparing scoring algorithms, to ensure reproducibility and replicability of results. This methodology mainly constrains the interface of the software and enables the engineer to develop his/her metric in his/her favorite language. Continuous integration and deployment tools are then used to verify the replicability of the software and to generate an executable instance independent of the language through dockerization. This article presents this methodology and points at all the metrics and documentation repositories designed with this policy in Eurobench. Applying this approach to other protocols and metrics would ease the reproduction, replication, and comparison of experiments.

Keywords

AlgorithmBenchmarkingExoskeletonExoskeletonsHumanoidPerformance indicatorReplicabilitySoftware

Quality index

Bibliometric impact. Analysis of the contribution and dissemination channel

The work has been published in the journal Frontiers In Robotics And Ai due to its progression and the good impact it has achieved in recent years, according to the agency Scopus (SJR), it has become a reference in its field. In the year of publication of the work, 2022, it was in position , thus managing to position itself as a Q2 (Segundo Cuartil), in the category Computer Science Applications.

From a relative perspective, and based on the normalized impact indicator calculated from the Field Citation Ratio (FCR) of the Dimensions source, it yields a value of: 4.89, which indicates that, compared to works in the same discipline and in the same year of publication, it ranks as a work cited above average. (source consulted: Dimensions Jun 2025)

Specifically, and according to different indexing agencies, this work has accumulated citations as of 2025-06-01, the following number of citations:

  • WoS: 1
  • Scopus: 9
  • OpenCitations: 4

Impact and social visibility

From the perspective of influence or social adoption, and based on metrics associated with mentions and interactions provided by agencies specializing in calculating the so-called "Alternative or Social Metrics," we can highlight as of 2025-06-01:

  • The use, from an academic perspective evidenced by the Altmetric agency indicator referring to aggregations made by the personal bibliographic manager Mendeley, gives us a total of: 12.
  • The use of this contribution in bookmarks, code forks, additions to favorite lists for recurrent reading, as well as general views, indicates that someone is using the publication as a basis for their current work. This may be a notable indicator of future more formal and academic citations. This claim is supported by the result of the "Capture" indicator, which yields a total of: 12 (PlumX).

With a more dissemination-oriented intent and targeting more general audiences, we can observe other more global scores such as:

  • The Total Score from Altmetric: 1.
  • The number of mentions on the social network X (formerly Twitter): 3 (Altmetric).

It is essential to present evidence supporting full alignment with institutional principles and guidelines on Open Science and the Conservation and Dissemination of Intellectual Heritage. A clear example of this is:

  • The work has been submitted to a journal whose editorial policy allows open Open Access publication.

Leadership analysis of institutional authors

This work has been carried out with international collaboration, specifically with researchers from: Canada; Germany; Iceland; Italy; Netherlands.