The science of testing : an automotive perspective

Abstract

Increasing automation in the automotive systems has re-focused the industry's attention on verification and validation methods and especially on the development of test scenarios. The complex nature of Advanced Driver Assistance Systems (ADASs) and Automated Driving (AD) systems warrant the adoption of new and innovative means of evaluating and establishing the safety of such systems. In this paper, the authors discuss the results from a semi-structured interview study, which involved interviewing ADAS and AD experts across the industry supply chain. \ud \ud Eighteen experts (each with over 10 years' of experience in testing and development of automotive systems) from different countries were interviewed on two themes: test methods and test scenarios. Each of the themes had three guiding questions which had some follow-up questions. The interviews were transcribed and a thematic analysis via coding was conducted on the transcripts. A two-stage coding analysis process was done to first identify codes from the transcripts and subsequently, the codes were grouped into categories. \ud \ud The analysis of transcripts for the question about the biggest challenge in the area of test methods revealed two specific themes. Firstly, the definition of pass/fail criteria and secondly the quality of requirements (completeness and consistency). The analysis of the questions on test scenarios revealed that "good" scenario is one that is able to test a safety goal and ways in which a system may fail. Based on the analysis of the transcripts, the authors propose two types of testing for ADAS and AD systems: Requirements-Based Testing (traditional method) and Hazard Based Testing. The proposed approach not only generates test scenarios for testing how the system works, but also how the system may fail.