Systematic literature review of validation methods for AI systems

Show full item record



Permalink

http://hdl.handle.net/10138/333412

Citation

Myllyaho , L S , Raatikainen , M , Männistö , T , Mikkonen , T & Nurminen , J K 2021 , ' Systematic literature review of validation methods for AI systems ' , The Journal of Systems and Software , vol. 181 , 111050 . https://doi.org/10.1016/j.jss.2021.111050

Title: Systematic literature review of validation methods for AI systems
Author: Myllyaho, Lalli Santeri; Raatikainen, Mikko; Männistö, Tomi; Mikkonen, Tommi; Nurminen, Jukka K
Contributor: University of Helsinki, Department of Computer Science
University of Helsinki, Empirical Software Engineering research group / Tomi Männistö
University of Helsinki, Empirical Software Engineering research group / Tomi Männistö
University of Helsinki, Empirical Software Engineering research group / Tomi Männistö
University of Helsinki, Department of Computer Science
Date: 2021-11
Language: eng
Number of pages: 22
Belongs to series: The Journal of Systems and Software
ISSN: 0164-1212
URI: http://hdl.handle.net/10138/333412
Abstract: Context: Artificial intelligence (AI) has made its way into everyday activities, particularly through new techniques such as machine learning (ML). These techniques are implementable with little domain knowledge. This, combined with the difficulty of testing AI systems with traditional methods, has made system trustworthiness a pressing issue. Objective: This paper studies the methods used to validate practical AI systems reported in the literature. Our goal is to classify and describe the methods that are used in realistic settings to ensure the dependability of AI systems. Method: A systematic literature review resulted in 90 papers. Systems presented in the papers were analysed based on their domain, task, complexity, and applied validation methods. Results: The validation methods were synthesized into a taxonomy consisting of trial, simulation, model-centred validation, and expert opinion. Failure monitors, safety channels, redundancy, voting, and input and output restrictions are methods used to continuously validate the systems after deployment. Conclusions: Our results clarify existing strategies applied to validation. They form a basis for the synthesization, assessment, and refinement of AI system validation in research and guidelines for validating individual systems in practice. While various validation strategies have all been relatively widely applied, only few studies report on continuous validation.
Subject: 113 Computer and information sciences
Artificial intelligence
Machine learning
Validation
Testing
V&V
Systematic literature review
SOFTWARE PRODUCT LINES
Rights:


Files in this item

Total number of downloads: Loading...

Files Size Format View
1_s2.0_S0164121221001473_main.pdf 1.064Mb PDF View/Open

This item appears in the following Collection(s)

Show full item record