The VALU3S ECSEL project: Verification and validation of automated systems safety and security

Manufacturers of automated systems and their components have been allocating an enormous amount of time and effort in R&D activities, which led to the availability of prototypes demonstrating new capabilities as well as the introduction of such systems to the market within different domains. Manufacturers need to make sure that the systems function in the intended way and according to specifications. This is not a trivial task as system complexity rises dramatically the more integrated and interconnected these systems become with the addition of automated functionality and features to them. This effort translates into an overhead on the V&V (verification and validation) process making it time-consuming and costly. In this paper, we present VALU3S, an ECSEL JU (joint undertaking) project that aims to evaluate the state-of-the-art V&V methods and tools, and design a multi-domain framework to create a clear structure around the components and elements needed to conduct the V&V process. The main expected benefit of the framework is to reduce time and cost needed to verify and validate automated systems with respect to safety, cyber-security, and privacy requirements. This is done through identification and classification of evaluation methods, tools, environments and concepts for V&V of automated systems with respect to the mentioned requirements. VALU3S will provide guidelines to the V&V community including engineers and researchers on how the V&V of automated systems could be improved considering the cost, time and effort of conducting V&V processes. To this end, VALU3S brings together a consortium with partners from 10 different countries, amounting to a mix of 25 industrial partners, 6 leading research institutes, and 10 universities to reach the project goal.     

A strategy for evaluating a fuzzy case-based construction procurement selection system

System Verification and Validation (V&V) is an essential element in the development and implementation of any computer-based decision tools. The unique concepts of Case-Based Reasoning (CBR), such as the use of mega-knowledge and nearest matching have generated extra challenges to system developers to ensure that the system is built right and the right system is built. However, little attention has been attributed to verifying and validating a CBR system. Recently, a fuzzy CBR prototype known as CaPS has been developed for the selection of appropriate construction procurement systems. To ensure that the procurement system is acceptable to the procurement experts in the construction industry, a series of tests have been conducted with domain experts using real cases (stored in the case base) and projects (as scenarios for retrieval and comparison). This paper reports on the findings of the V&V that have been performed on CaPS. Techniques available for verifying and validating a CBR system are first discussed. The V&V procedures applied to the prototype system are subsequently outlined. The results confirm that the cases stored in CaPS are correct, consistent, and irredundant. More importantly, the solutions generated by CaPS are accurate and innovative, and these are necessary for today's construction projects.     

Early verification and validation of mission critical systems

Complex software and systems are pervasive in today’s world. In a growing number of fields they come to play a critical role. In order to provide a high assurance level, verification and validation (V&V) should be considered early in the development process. This paper shows how this can be achieved based on a goal-oriented requirements engineering framework which combines complementary semi-formal and formal notations. This allows the analyst to formalize only when and where needed and also preserves optimal communication with stakeholders and developers. For the industrial application of the methodology, a supporting toolbox was developed. It consist of a number of tightly integrated tools for performing V&V tasks at requirements level. This is achieved through the use of (1) a roundtrip mapping between the requirements language and the specific formal languages used in the underlying formal tools (such as SAT or constraint solvers) and (2) graphical views using domain-based representations. This paper will focus on two major and representative tools: the Refinement Checker (about verification) and the Animator (about validation).     

Challenges of working with artifacts in requirements engineering and software engineering

When developing or evolving software systems of non-trivial size, having the requirements properly documented is a crucial success factor. The time and effort required for creating and maintaining non-code artifacts are significantly influenced by the tools with which practitioners view, navigate and edit these artifacts. This is not only true for requirements, but for any artifacts used when developing or evolving systems. However, there is not much evidence about how practitioners actually work with artifacts and how well software tools support them. Therefore, we conducted an exploratory study based on 29 interviews with software practitioners to understand the current practice of presenting and manipulating artifacts in tools, how practitioners deal with the challenges encountered, and how these challenges affect the usability of the tools used. We found that practitioners typically work with several interrelated artifacts concurrently, less than half of these artifacts can be displayed entirely on a large screen, the artifact interrelationship information is often missing, and practitioners work collaboratively on artifacts without sufficient support. We identify the existing challenges of working with artifacts and discuss existing solutions proposed addressing them. Our results contribute to the body of knowledge about how practitioners work with artifacts when developing or evolving software, the challenges they are faced with, and the attempts to address these challenges.     

Testing scientific software: A systematic literature review

ContextScientific software plays an important role in critical decision making, for example making weather predictions based on climate models, and computation of evidence for research publications. Recently, scientists have had to retract publications due to errors caused by software faults. Systematic testing can identify such faults in code.ObjectiveThis study aims to identify specific challenges, proposed solutions, and unsolved problems faced when testing scientific software.MethodWe conducted a systematic literature survey to identify and analyze relevant literature. We identified 62 studies that provided relevant information about testing scientific software.ResultsWe found that challenges faced when testing scientific software fall into two main categories: (1) testing challenges that occur due to characteristics of scientific software such as oracle problems and (2) testing challenges that occur due to cultural differences between scientists and the software engineering community such as viewing the code and the model that it implements as inseparable entities. In addition, we identified methods to potentially overcome these challenges and their limitations. Finally we describe unsolved challenges and how software engineering researchers and practitioners can help to overcome them.ConclusionsScientific software presents special challenges for testing. Specifically, cultural differences between scientist developers and software engineers, along with the characteristics of the scientific software make testing more difficult. Existing techniques such as code clone detection can help to improve the testing process. Software engineers should consider special challenges posed by scientific software such as oracle problems when developing testing techniques.     

Practical design considerations for successful industrial application of model-based fault detection techniques to aircraft systems

This paper discusses some key factors which may arise for successful application of model-based Fault Detection (FD) techniques to aircraft systems. The paper reports on the results and the lessons learned during flight V&V (Validation & Verification) activities, implementation in the A380 Flight Control Computer (FCC) and A380 flight tests at Airbus (Toulouse, France). The paper does not focus on new theoretical materials, but rather on a number of practical design considerations to provide viable technological solutions and mechanization schemes. The selected case studies are taken from past and on-going research actions between Airbus and the University of Bordeaux (France). One of the presented solutions has received final certification on new generation Airbus A350 aircraft and is flying (first commercial flight: January 15, 2015).     

Usability engineering laboratories: limitations and challenges toward a unifying tools/practices environment

This article discusses the limitations and challenges surrounding the present usability engineering (UE) labs. Usability laboratories include hardware and software tools to observe users, collect and analyse diverse data about the users' interactions, behaviours, actions, and reactions including their raw feedback regarding their experiences. Using statistical analysis and data mining software, these qualitative and quantitative data are transformed into design insights and recommendations for future usability improvements. First, we survey the existing stationary, portable, and remote laboratories. We then review the current usability tools while highlighting the gap between the existing tools/labs and the UE practices. We will show how this gap can be closed via a roadmap using a computer-assisted usability engineering environment (CAUTE). A CAUTE provides a unifying user interface that exploits a process-sensitive architecture for integrating the large variety of the existing tools into the best UE practices. Beyond the technical problems, there is also a need to address research issues including determining the interest of the CAUTE approach in comparison with the current usability labs.     

Migration of existing software systems to mobile computing platforms: a systematic mapping study

Mobile computing has fast emerged as a pervasive technology to replace the old computing paradigms with portable computation and context-aware communication. Existing software systems can be migrated (while preserving their data and logic) to mobile computing platforms that support portability, context-sensitivity, and enhanced usability. In recent years, some research and development efforts have focused on a systematic migration of existing software systems to mobile computing platforms.To investigate the research state-of-the-art on the migration of existing software systems to mobile computing platforms. We aim to analyze the progression and impacts of existing research, highlight challenges and solutions that reflect dimensions of emerging and futuristic research.We followed evidence-based software engineering (EBSE) method to conduct a systematic mapping study (SMS) of the existing research that has progressed over more than a decade (25 studies published from 1996–2017).We have derived a taxonomical classification and a holistic mapping of the existing research to investigate its progress, impacts, and potential areas of futuristic research and development.The SMS has identified three types of migration namely Static, Dynamic, and State-based Migration of existing software systems to mobile computing platforms.Migration to mobile computing platforms enables existing software systems to achieve portability, context-sensitivity, and high connectivity. However, mobile systems may face some challenges such as resource poverty, data security, and privacy. The emerging and futuristic research aims to support patterns and tool support to automate the migration process. The results of this SMS can benefit researchers and practitioners–by highlighting challenges, solutions, and tools, etc., –to conceptualize the state-ofthe- art and futuristic trends that support migration of existing software to mobile computing.     

A distributed approach to verification and validation of electronic structure simulation data using ESTEST

We present a Verification and Validation (V&V) approach for electronic structure computations based on a network of distributed servers running the ESTEST (Electronic Structure TEST) software. This network-based infrastructure enables remote verification, validation, comparison and sharing of electronic structure data obtained with different simulation codes. The implementation and configuration of the distributed framework is described. ESTEST features are enhanced by server communication and data sharing, minimizing the duplication of effort by separate research groups. We discuss challenges that arise from the use of a distributed network of ESTEST servers and outline possible solutions. A community web portal called ESTEST Discovery is introduced for the purpose of facilitating the collection and annotation of contents from multiple ESTEST servers. We describe examples of use of the framework using two currently running servers at the University of California Davis and at the Centre Européen de Calcul Atomique et Moléculaire (CECAM).     

Blockchain-empowered security and privacy protection technologies for smart grid

With the rapid development in society of the economy and of computational technologies, it is particularly important to build a secure, efficient and reliable smart grid architecture to provide users with high-quality electricity services. However, data collection and energy trading in public networks creates security and privacy challenges in smart grids. Blockchain technologies have the excellent characteristics of decentralization, immutability and traceability, which can resolve the security, integration and coordination problems faced by the traditional centralized networks for smart grids. The goal of this paper is to introduce and compare blockchain-based technologies in addressing the problems of privacy protection, identity authentication, data aggregation and electricity pricing for the data collection and power energy trading processes in smart grids. In addition, the existing challenges and future research directions of smart grids are discussed.     

Combining unit and specification-based testing for meta-model validation and verification

Meta-models play a cornerstone role in Model-Driven Engineering as they are used to define the abstract syntax of modelling languages, and so models and all sorts of model transformations depend on them. However, there are scarce tools and methods supporting their Validation and Verification (V&V), which are essential activities for the proper engineering of meta-models.In order to fill this gap, we propose two complementary meta-model V&V languages. The first one has similar philosophy to the xUnit framework, as it enables the definition of meta-model unit test suites comprising model fragments and assertions on their (in-)correctness. The second one is directed to express and verify expected properties of a meta-model, including domain and design properties, quality criteria and platform-specific requirements.As a proof of concept, we have developed tooling for both languages in the Eclipse platform, and illustrate its use within an example-driven approach for meta-model construction. The expressiveness of our languages is demonstrated by their application to build a library of meta-model quality issues, which has been evaluated over the ATL zoo of meta-models and some OMG specifications. The results show that integrated support for meta-model V&V (as the one we propose here) is urgently needed in meta-modelling environments.     

Model learning: a survey of foundations,tools and applications

Software systems are present all around us and playing their vital roles in our daily life. The correct functioning of these systems is of prime concern. In addition to classical testing techniques, formal techniques like model checking are used to reinforce the quality and reliability of software systems. However, obtaining of behavior model, which is essential for model-based techniques, of unknown software systems is a challenging task. To mitigate this problem, an emerging black-box analysis technique, called Model Learning, can be applied. It complements existing model-based testing and verification approaches by providing behavior models of blackbox systems fully automatically. This paper surveys the model learning technique, which recently has attracted much attention from researchers, especially from the domains of testing and verification. First, we review the background and foundations of model learning, which form the basis of subsequent sections. Second, we present some well-known model learning tools and provide their merits and shortcomings in the form of a comparison table. Third, we describe the successful applications of model learning in multidisciplinary fields, current challenges along with possible future works, and concluding remarks.     

EQUITAS: A tool-chain for functional safety and reliability improvement in automotive systems

To support advanced features such as hybrid engine control, intelligent energy management, and advanced driver assistance systems, automotive embedded systems must use advanced technologies. As a result, systems are becoming distributed and include dozens of Electronic Control Units (ECU). On the one hand, this tendency raises the issue of robustness and reliability, due to the increase in the error ratio with the integration level and the clock frequency. On the other hand, due to a lack of automation, software Validation and Verification (V&V) tends to swallow up 40% to 50% of the total development cost. The ``Enhanced Quality Using Intensive Test Analysis on Simulators'' (EQUITAS¹) project aims (1) to improve reliability and functional safety and (2) to limit the impact of software V&V on embedded systems costs and time-to-market. These two achievements are obtained by (1) developing a continuous tool-chain to automate the V&V process, (2) improving the relevance of the test campaigns by detecting redundant tests using equivalence classes, (3) providing assistance for hardware failure effect analysis (FMEA) and finally (4) assessing the tool-chain under the ISO 26262 requirements.     

Approximate declarative semantics for rule base anomalies

Despite the fact that there has been a surge of publications in verification and validation of knowledge-based systems and expert systems in the past decade, there are still gaps in the study of verification and validation (V&V) of expert systems, not the least of which is the lack of appropriate semantics for expert system programming languages. Without a semantics, it is hard to formally define and analyze knowledge base anomalies such as inconsistency and redundancy, and it is hard to assess the effectiveness of V&V tools, methods and techniques that have been developed or proposed. In this paper, we develop an approximate declarative semantics for rule-based knowledge bases and provide a formal definition and analysis of knowledge base inconsistency, redundancy, circularity and incompleteness in terms of theories in the first order predicate logic. In the paper, we offer classifications of commonly found cases of inconsistency, redundancy, circularity and incompleteness. Finally, general guidelines on how to remedy knowledge base anomalies are given.     

Tracing requirements to defect reports: an application of information retrieval techniques

To support debugging, maintenance, verification and validation (V&V) and/or independent V&V (IV&V), it is necessary to understand the relationship between defect reports and their related artifacts. For example, one cannot correct a code-related defect report without being able to find the code that is affected. Information retrieval (IR) techniques have been used effectively to trace textual artifacts to each other. This has generally been applied to the problem of dynamically generating a trace between artifacts in the software document hierarchy after the fact (after development has proceeded to at least the next lifecycle phase). The same techniques can also be used to trace textual artifacts of the software engineering lifecycle to defect reports. We have applied the term frequency–inverse document frequency (TF-IDF) technique with relevance feedback, as implemented in our requirements tracing on-target (RETRO) tool, to the problem of tracing textual requirement elements to related textual defect reports. We have evaluated the technique using a dataset for a NASA scientific instrument. We found that recall of over 85% and precision of 69%, and recall of 70% and precision of 99% could be achieved, respectively, on two subsets of the dataset.     

Systems & Control for the future of humanity,research agenda: Current and future roles,impact and grand challenges

Following in the footsteps of the renowned report “Control in an Information Rich World,” Report of the Panel on “Future Directions in Control, Dynamics, and Systems” chaired by Richard Murray (2002), this paper aims to demonstrate that Systems & Control is at the heart of the Information and Communication Technologies to most application domains. As such, Systems & Control should be acknowledged as a priority by funding agencies and supported at the levels necessary to enable technologies addressing critical societal challenges. A second intention of this paper is to present to the industrials and the young research generation, a global picture of the societal and research challenges where the discipline of Systems & Control will play a key role. Throughout, this paper demonstrates the extremely rich, current and future, cross-fertilization between five critical societal challenges and seven key research and innovation Systems & Control scientific challenges. This paper is authored by members of the IFAC Task Road Map Committee, established following the 19th IFAC World Congress in Cape Town. Other experts who authored specific parts are listed below.     

Automatic Generation of Test Oracles—From Pilot Studies to Application

We describe a progression from pilot studies to development and use of domain-specific verification and validation (V&V) automation. Our domain is the testing of an AI planning system that forms a key component of an autonomous spacecraft. We used pilot studies to ascertain opportunities for, and suitability of, automating various analyses whose results would contribute to V&V in our domain. These studies culminated in development of an automatic generator of automated test oracles. This was then applied and extended in the course of testing the spacecraft's AI planning system.Richardson et al. (1992, In Proceedings of the 14th International Conference on Software Engineering, Melbourne, Australia, pp. 105–118), presents motivation for automatic test oracles, and considered the issues and approaches particular to test oracles derived from specifications. Our work, carried through from conception to application, confirms many of their insights. Generalizing from our specific domain, we present some additional insights and recommendations concerning the use of test oracles for V&V of knowledge-based systems.     

An integrated semantic framework supporting universal accessibility to ICT

As we are moving rapidly to a digital economy, accessing and effectively using Information and Communication Technologies (ICT) in everyday life is widely recognized as an important requirement. However, the accessibility technologies that we have up to date are meeting the needs of only some, at a very high cost and, as a consequence, accessible ICT for all people still remains a major research and development goal. This work presents an integrated ontological framework for the semantic representation of terms and concepts (i.e., related to user needs and preferences (N&P) with respect to ICT use, as well as solutions, platforms and devices) that are required for addressing the universal accessibility in the scope of the Cloud4all project and the Global Public Inclusive Infrastructure (GPII). Cloud4all aims at advancing and building upon the concept of GPII through the development of the necessary tools and models for making ICT accessible for all by exploiting the cloud computing paradigm. The main goal of the proposed framework lays in the separation between generalized accessibility concepts, user interaction mechanisms and N&P with the particular details of different ICT artifacts. Thus, the framework aims at integrating concepts related with user N&P, as well as ICT solutions, platforms, devices and their customizable settings along with information concerning their vendors or implementers, in order to (a) offer the necessary expressiveness for defining/representing personal N&P across applications, platforms and devices, (b) link N&P with the conditions/context according to which these shall be applicable for (e.g., considering the user activity and the physical environment), (c) link interaction requirements (originated from user characteristics) with N&P and (d) support the Cloud4all matchmaking process through the mapping between N&P and application-specific settings based on semantic rules and automatic reasoning techniques.     

Verification and validation of knowledge-based systems

Knowledge-based systems (KBSs) are being used in many applications areas where their failures can be costly because of losses in services, property or even life. To ensure their reliability and dependability, it is therefore important that these systems are verified and validated before they are deployed. This paper provides perspectives on issues and problems that impact the verification and validation (V&V) of KBSs. Some of the reasons why V&V of KBSs is difficult are presented. The paper also provides an overview of different techniques and tools that have been developed for performing V&V activities. Finally, some of the research issues that are relevant for future work in this field are discussed