{rfName}
Op

Indexed in

Citations

1

Altmetrics

Analysis of institutional authors

Reviriego, PedroAuthorPozo, AlejandoAuthor

Share

January 12, 2025
Publications
>
Article
No

Operating Conversational Large Language Models (LLMs)in the Presenceof Errors

Publicated to: IEEE Nanotechnology Magazine. 19 (1): 31-37 - 2025-02-01 19(1), DOI: 10.1109/MNANO.2024.3513112

Authors:

Gao, Z; Deng, J; Reviriego, P; Liu, SS; Pozo, A; Lombardi, F
[+]

Affiliations

Northeastern Univ, Dept Elect & Comp Engn, Boston, MA 02215 USA - Author
Tianjin Univ, Sch Elect & Informat Engn, Tianjin 300072, Peoples R China - Author
Tianjin Univ, Sch Future Technol, Tianjin 300072, Peoples R China - Author
Univ Elect Sci & Technol China, Sch Informat & Commun Engn, Chengdu 611731, Peoples R China - Author
Univ Politecn Madrid, ETSI Telecomunicac, Madrid 28040, Spain - Author
See more

Abstract

Conversational Large Language Models have taken the center stage of the artificial intelligence landscape. As they are pervasive, there is a need to evaluate their dependability, i.e., performance when errors appear due to the underlying hardware implementation. In this paper we consider the evaluation of the dependability of a widely used conversational LLM: Mistral-7B. Error injection is conducted, and the Multitask Language Understanding (MMLU) benchmark is used to evaluate the impact on performance. The drop in the percentage of correct answers due to errors is analyzed and the results provide interesting insights: Mistral-7B has a large intrinsic tolerance to errors even at high bit error rates. This opens the door to the use of nanotechnologies that trade-off errors for energy dissipation and complexity to further improve the LLM implementation. Also, the error tolerance is larger for 8-bit quantization than for 4-bit quantization, so suggesting that there will be also a trade-off between quantization optimizations to reduce memory requirements and error tolerance. In addition, we also show the different impact of errors on different types of weights, which is valuable information for selective protection designs.
[+]

Keywords

Benchmark testingCodesComputational modelingDependabilityErrorError-injectionErrorsErrors toleranceFits and tolerancesGenerative artificial intelligenceHardwareHardware implementationsIntegrated circuit modelingLanguage modelLarge language modelLarge language modelsLogic gatesMemory managementPerformanceQuantisationQuantization (signal)Trade offTransformersTranslation

Quality index

Bibliometric impact. Analysis of the contribution and dissemination channel

The work has been published in the journal IEEE Nanotechnology Magazine due to its progression and the good impact it has achieved in recent years, according to the agency Scopus (SJR), it has become a reference in its field. In the year of publication of the work, 2025, it was in position , thus managing to position itself as a Q2 (Segundo Cuartil), in the category Electrical and Electronic Engineering. Notably, the journal is positioned en el Cuartil Q3 for the agency WoS (JCR) in the category Nanoscience & Nanotechnology.

[+]

Impact and social visibility

From the perspective of influence or social adoption, and based on metrics associated with mentions and interactions provided by agencies specializing in calculating the so-called "Alternative or Social Metrics," we can highlight as of 2026-04-09:

  • The use of this contribution in bookmarks, code forks, additions to favorite lists for recurrent reading, as well as general views, indicates that someone is using the publication as a basis for their current work. This may be a notable indicator of future more formal and academic citations. This claim is supported by the result of the "Capture" indicator, which yields a total of: 4 (PlumX).

It is essential to present evidence supporting full alignment with institutional principles and guidelines on Open Science and the Conservation and Dissemination of Intellectual Heritage. A clear example of this is:

  • Assignment of a Handle/URN as an identifier within the deposit in the Institutional Repository: https://oa.upm.es/86955/

As a result of the publication of the work in the institutional repository, statistical usage data has been obtained that reflects its impact. In terms of dissemination, we can state that, as of

  • Views: 226
  • Downloads: 289
[+]

Leadership analysis of institutional authors

This work has been carried out with international collaboration, specifically with researchers from: China; United States of America.

There is a significant leadership presence as some of the institution’s authors appear as the first or last signer, detailed as follows: Last Author (Lombardi, Fabrizio).

[+]

Awards linked to the item

This work was supported in part by the Natural Science Funds of China under Grant 62171313, in part by the FUN-4DATE under Grant PID2022-136684OB-C22 project funded by the Spanish Agencia Estatal de Investigacion (AEI) 10.13039/501100011033, and in part by the Chips Act Joint Undertaking project SMARTY under Grant 101140087.
[+]