工业软件现场测试中的拆分及其测试数据设计

在工业软件的用户生产现场测试中,可能由于操作风险、用户生产限制等约束而导致测试不充分,针对实践中的难点提出新的现场测试过程及其测试数据生成方法。定义了测试拆分子集及相关属性概念,根据现场因素设计的测试拆分子集提高了现场测试可主动性;综合现场因素设计拆分方案,采用针对性的辅助管理程序引导每个测试拆分子集的现场测试的实施;通过对生产现场的历史数据的处理建立针对各测试拆分子集的测试准备数据库,以搜索测试准备数据库的结果为依据并结合现场因素设计现场主动测试的输入数据,并说明了实践中的综合实施过程。通过石化生产优化控制系统的现场测试的实际案例,说明了这种方法能够在预防风险的前提下显著提高现场测试的充分性和测试效率。     

Mission-critical development with open source software: lessons learned

Using open source software components in a mission-critical project not only can keep the project within budget but can also result in a more robust and flexible tool. When considering an open source component, prospective users should evaluate the project for several characteristics: maturity, longevity, and flexibility. For greatest benefit, the users should also build and maintain a strong working relationship with the component's developers. We compiled our experiences developing SAP into a developer's guide for those considering using open source in their mission-critical application. In addition to discussing how to evaluate open source components suitability for inclusion in a mission-critical application, the guide suggests strategies for working with open source development teams.     

Usability testing in a competive market: lessons learned

Abstract

Usability testing is a relatively new and rapidly developing field. Newcomers to usability testing typically enter the profession with a knowledge of social science research methods and the belief that usability testing is conducted in a lab using thinking aloud techniques to identify usability defects. Our practice of usability testing at Mead Data Central has shown that these core beliefs represent an approach of limited utility in designing products like the LEX1S®NEXIS® research systems. In this paper, we describe our experiences in conducting traditional usability testing and how we used the results of those efforts to develop more effective methods of testing for Mead Data Central's products and customers.     

Software development productivity on a new platform: an industrial case study

The high non-functional requirements on mobile telecommunication applications call for new solutions. An example of such a solution can be a software platform that provides high performance and availability. The introduction of such a platform may, however, affect the development productivity. In this study, we present experiences from research carried out at Ericsson. The purpose of the research was productivity improvement and assessment when using the new platform. In this study, we quantify and evaluate the current productivity level by comparing it with UNIX development. The comparison is based on two large, commercially, available systems. We reveal a factor of four differences in productivity. Later, we decompose the problem into two issues: code writing speed and average amount of code necessary to deliver a certain functionality. We assess the impact of both these issues. We describe the nature of the problem by identifying factors that affect productivity and estimating their importance. To the issues identified we suggest a number of remedies. The main methods used in the study are interviews and historical data research.     

Software generated device exception for more intensive device-related software testing: An industrial field study

It is more important to properly handle exceptions, than to prevent exceptions from occurring, because they arise from so many different causes. In embedded systems, a vast number of exceptions are caused by hardware devices. In such cases, numerous software components are involved in these hardware device-originated exceptions, ranging from the device itself to the device driver, the kernel, and applications. Therefore, it takes a lot of time to debug software that fails to handle exceptions. This paper proposes a lightweight device exception testing method, and a related automation tool, AMOS v3.0. The proposed method artificially triggers more realistic device exceptions in runtime, and monitors how software components handle exceptions in detail. AMOS v3.0 has been applied to the exception testing of car-infotainment systems in an automobile company. The results based on this industrial field study have revealed that 39.13% of the failures in exception handling were caused by applications, 36.23% of the failures were caused by device drivers, and 24.64% were derived from the kernel. We conclude that the proposed method is highly effective, in that it can allow developers to identify the root cause of failure for exception handling.     

Assessing software product line potential: an exploratory industrial case study

Corporate organizations sometimes offer similar software products in certain domains due to former company mergers or due to the complexity of the organization. The functional overlap of such products is an opportunity for future systematic reuse to reduce software development and maintenance costs. Therefore, we have tailored existing domain analysis methods to our organization to identify commonalities and variabilities among such products and to assess the potential for software product line (SPL) approaches. As an exploratory case study, we report on our experiences and lessons learned from conducting the domain analysis in four application cases with large-scale software products. We learned that the outcome of a domain analysis was often a smaller integration scenario instead of an SPL and that business case calculations were less relevant for the stakeholders and managers from the business units during this phase. We also learned that architecture reconstruction using a simple block diagram notation aids domain analysis and that large parts of our approach were reusable across application cases.     

Applying empirical software engineering to software architecture: challenges and lessons learned

In the last 15 years, software architecture has emerged as an important software engineering field for managing the development and maintenance of large, software-intensive systems. Software architecture community has developed numerous methods, techniques, and tools to support the architecture process (analysis, design, and review). Historically, most advances in software architecture have been driven by talented people and industrial experience, but there is now a growing need to systematically gather empirical evidence about the advantages or otherwise of tools and methods rather than just rely on promotional anecdotes or rhetoric. The aim of this paper is to promote and facilitate the application of the empirical paradigm to software architecture. To this end, we describe the challenges and lessons learned when assessing software architecture research that used controlled experiments, replications, expert opinion, systematic literature reviews, observational studies, and surveys. Our research will support the emergence of a body of knowledge consisting of the more widely-accepted and well-formed software architecture theories.     

Evaluating software reuse alternatives: a model and its application to an industrial case study

We propose a model that enables software developers to systematically evaluate and compare all possible alternative reuse scenarios. The model supports the clear identification of the basic operations involved and associates a cost component with each basic operation in a focused and precise way. The model is a practical tool that assists developers to weigh and evaluate different reuse scenarios, based on accumulated organizational data, and then to decide which option to select in a given situation. The model is currently being used at six different companies for cost-benefit analysis of alternative reuse scenarios; we give a case study that illustrates how it has been used in practice.     

Requirements for a COTS software component: A case study

The goal of the GUARDS project is to design and develop a generic fault-tolerant computer architecture that can be built from predefined standardised components. The architecture favours the use of commercial off-the-shelf (COTS) hardware and software components. However, the assessment and selection of COTS components is a non-trivial task as it requires balancing a myriad of requirements from end-users and the preliminary architecture design. In this paper, we present the requirements and assessment criteria for a specific COTS software component, the operating system kernel. As an interface specification constitutes a major compatibility criterion for the selection of COTS components in GUARDS, a particular emphasis is placed on operating system conformance to the POSIX 1003.1 standard. We discuss the general lessons learned from the assessment process and raise a number of questions relevant to the assessment of any COTS software component.     

Using aspects for testing of embedded software: experiences from two industrial case studies

Aspect-oriented software testing is emerging as an important alternative to conventional procedural and object-oriented testing techniques. This paper reports experiences from two case studies where aspects were used for the testing of embedded software in the context of an industrial application. In the first study, we used code-level aspects for testing non-functional properties. The methodology we used for deriving test aspect code was based on translating high-level requirements into test objectives, which were then implemented using test aspects in AspectC++. In the second study, we used high-level visual scenario-based models for the test specification, test generation, and aspect-based test execution. To specify scenario-based tests, we used a UML2-compliant variant of live sequence charts. To automatically generate test code from the models, a modified version of the S2A Compiler, outputting AspectC++ code, was used. Finally, to examine the results of the tests, we used the Tracer, a prototype tool for model-based trace visualization and exploration. The results of the two case studies show that aspects offer benefits over conventional techniques in the context of testing embedded software; these benefits are discussed in detail. Finally, towards the end of the paper, we also discuss the lessons learned, including the technological and other barriers to the future successful use of aspects in the testing of embedded software in industry.     

A systematic test case selection methodology for product lines: results and insights from an industrial case study

In the context of product lines, test case selection aims at obtaining a set of relevant test cases for a product from the entire set of test cases available for a product line. While working on a research-based innovation project on automated testing of product lines of Video Conferencing Systems (VCSs) developed by Cisco, we felt the need to devise a cost-effective way of selecting relevant test cases for a product. To fulfill such need, we propose a systematic and automated test selection methodology using: 1) Feature Model for Testing (FM_T) to capture commonalities and variabilities of a product line; 2) Component Family Model for Testing (CFM_T) to model the structure of test case repository; 3) A tool to automatically build restrictions from CFM_T to FM_T and traces from CFM_T to the actual test cases. Using our methodology, a test engineer is only required to select relevant features through FM_T at a higher level of abstraction for a product and the corresponding test cases will be obtained automatically. We evaluate our methodology by applying it to a VCS product line called Saturn with seven commercial products and the results show that our methodology can significantly reduce cost measured as test selection time and at the same time achieves higher effectiveness (feature coverage, feature pairwise coverage and fault detection) as compared with the current manual process. Moreover, we conduct a questionnaire-based study to solicit the views of test engineers who are involved in developing FM_T and CFM_T. The results show that test engineers are positive about adapting our methodology in their current practice. Finally, we present a set of lessons learnt while applying product line engineering at Cisco for test case selection.     

Dynamic test planning: a study in an industrial context

Testing accounts for a relevant part of the production cost of complex or critical software systems. Nevertheless, time and resources budgeted to testing are often underestimated with respect to the target quality goals. Test managers need engineering methods to perform appropriate choices in spending testing resources, so as to maximize the outcome. We present a method to dynamically allocate testing resources to software components minimizing the estimated number of residual defects and/or the estimated residual defect density. We discuss the application to a real-world critical system in the homeland security domain. We describe a support tool aimed at easing industrial technology transfer by hiding to practitioners the mathematical details of the method application.     

Software reuse and competition: Consumer preferences in a software component market

Research on software reuse has traditionally focused on reuse among employees within the same organization. In contrast, our research investigates consumer preferences in a software component market with multiple producers and consumers of software components. We propose that empirically studying competition in software component markets is crucial to discovering success factors for improving software reuse. To facilitate such research, we have developed an experimental market laboratory called SofTrade; we report here the results of two empirical studies made possible by this environment. Our results support some existing reuse maxims, such as the concept that consumers prefer to reuse components that are general and flexible. Other established reuse beliefs are not supported by our results, such as the expectation that consumers will prefer objects with unique, complex features. Overall, the research reported here strongly supports the claim that the practice of exploring software reuse activity within real or artificial software component markets produces unique insights into how we can improve software reuse.     

Usage and usefulness of technical software documentation: An industrial case study

ContextSoftware documentation is an integral part of any software development process. However, software practitioners are often concerned about the value, degree of usage and usefulness of documentation during development and maintenance.ObjectiveMotivated by the needs of NovAtel Inc. (NovAtel), a world-leading company developing software systems in support of global navigation satellite systems, and based on the results of a former systematic mapping study, we aimed at better understanding of the usage and the usefulness of various technical documents during software development and maintenance.MethodWe utilized the results of a former systematic mapping study and performed an industrial case study at NovAtel. From the joint definition of the analysis goals, the research method incorporates qualitative and quantitative analysis of 55 documents (design, test and process related) and 1630 of their revisions. In addition, we conducted a survey on the usage and usefulness of documents. A total of 25 staff members from the industrial partner, all having a medium to high level of experience, participated in the survey.ResultsIn the context of the case study, a number of findings were derived. They include that (1) technical documentation was consulted least frequently for maintenance purpose and most frequently as an information source for development, (2) source code was considered most frequently as the preferred information source during software maintenance, (3) there is no significant difference between the usage of various documentation types during both development and maintenance, and (4) initial hypotheses stating that up-to-date information, accuracy and preciseness have the highest impact on usefulness of technical documentation.ConclusionsIt is concluded that the usage of documentation differs for various purposes and it depends on the type of the information needs as well as the tasks to be completed (e.g., development and maintenance). The results have been confirmed to be helpful for the company under study, and the firm is currently implementing some of the recommendations given.     

How productivity improves in hands-free continuous dictation tasks: lessons learned from a longitudinal study

Speech recognition technology continues to improve, but users still experience significant difficulty using the software to create and edit documents. The reported composition speed using speech software is only between 8 and 15 words per minute [Proc CHI 99 (1999) 568; Universal Access Inform Soc 1 (2001) 4], much lower than people's normal speaking speed of 125–150 words per minute. What causes the huge gap between natural speaking and composing using speech recognition? Is it possible to narrow the gap and make speech recognition more promising to users? In this paper we discuss users' learning processes and the difficulties they experience as related to continuous dictation tasks using state of the art Automatic Speech Recognition (ASR) software. Detailed data was collected for the first time on various aspects of the three activities involved in document composition tasks: dictation, navigation, and correction. The results indicate that navigation and error correction accounted for big chunk of the dictation task during the early stages of interaction. As users gained more experience, they became more efficient at dictation, navigation and error correction. However, the major improvements in productivity were due to dictation quality and the usage of navigation commands. These results provide insights regarding the factors that cause the gap between user expectation with speech recognition software and the reality of use, and how those factors changed with experience. Specific advice is given to researchers as to the most critical issues that must be addressed.     

Automated product line test case selection: industrial case study and controlled experiment

Automated test case selection for a new product in a product line is challenging due to several reasons. First, the variability within the product line needs to be captured in a systematic way; second, the reusable test cases from the repository are required to be identified for testing a new product. The objective of such automated process is to reduce the overall effort for selection (e.g., selection time), while achieving an acceptable level of the coverage of testing functionalities. In this paper, we propose a systematic and automated methodology using a feature model for testing (FM_T) to capture commonalities and variabilities of a product line and a component family model for testing (CFM_T) to capture the overall structure of test cases in the repository. With our methodology, a test engineer does not need to manually go through the repository to select a relevant set of test cases for a new product. Instead, a test engineer only needs to select a set of relevant features using FM_T at a higher level of abstraction for a product and a set of relevant test cases will be selected automatically. We evaluated our methodology via three different ways: (1) We applied our methodology to a product line of video conferencing systems called Saturn developed by Cisco, and the results show that our methodology can reduce the selection effort significantly; (2) we conducted a questionnaire-based study to solicit the views of test engineers who were involved in developing FM_T and CFM_T. The results show that test engineers are positive about adapting our methodology and models (FM_T and CFM_T) in their current practice; (3) we conducted a controlled experiment with 20 graduate students to assess the performance (i.e., cost, effectiveness and efficiency) of our automated methodology as compared to the manual approach. The results showed that our methodology is cost-effective as compared to the manual approach, and at the same time, its efficiency is not affected by the increased complexity of products.     

Lessons learned from developing mbeddr: a case study in language engineering with MPS

Software and Systems Modeling - Language workbenches are touted as a promising technology to engineer languages for use in a wide range of domains, from programming to science to business. However,...