Integrating risk-based testing in industrial test processes

Risk-based testing has a high potential to improve the software development and test process as it helps to optimize the allocation of resources and provides decision support for the management. But for many organizations, its integration into an existing test process is a challenging task. In this article, we provide a comprehensive overview of existing work and present a generic testing methodology enhancing an established test process to address risks. On this basis, we develop a procedure on how risk-based testing can be introduced in a test process and derive a stage model for its integration. We then evaluate our approach for introducing risk-based testing by means of an industrial study and discuss benefits, prerequisites and challenges to introduce it. Potential benefits of risk-based testing identified in the studied project are faster detection of defects resulting in an earlier release, a more reliable release quality statement as well as the involved test-process optimization. As necessary prerequisites for risk-based testing, we identified an inhomogeneous distribution of risks associated with the various parts of the tested software system as well as consolidated technical and business views on it. Finally, the identified challenges of introducing risk-based testing are reliable risk assessment in the context of complex systems, the availability of experts for risk assessment as well as established tool supports for test management.     

A multiple case study on risk-based testing in industry

In many development projects, testing has to be conducted under severe pressure due to limited resources and a challenging time schedule. Risk-based testing, which utilizes identified risks of the system for testing purposes, has a high potential to improve testing as it helps to optimize the allocation of resources and provides decision support for management. But for many organizations, the integration of a risk-based approach into established testing activities is a challenging task, and there are several options to do so. In this article, we analyze how risk is defined, assessed, and applied to support and improve testing activities in projects, products, and processes. We investigate these questions empirically by a multiple case study of currently applied risk-based testing activities in industry. The case study is based on three cases from different backgrounds, i.e., a test project in context of the extension of a large Web-based information system, product testing of a measurement and diagnostic equipment for the electrical power industry, as well as a test process of a system integrator of telecommunication solutions. By analyzing and comparing these different industrial cases, we draw conclusions on the state of risk-based testing and discuss possible improvements.     

Risk-Based Stopping Criteria for Test Sequencing

Testing complex manufacturing systems, like ASML lithographic machines, can take up to 45% of the total development time. The decision of when to stop testing is often difficult to make because less testing may leave critical faults in the system, while more testing increases time-to-market. In this paper, we solve the problem of deciding when to stop testing by introducing a test-sequencing method that incorporates several stopping criteria. These stopping criteria consist of objectives and constraints on the test cost and the remaining risk cost. For a given problem, a suitable stopping criterion can be chosen. For example, with the risk-based stopping criterion, testing stops when the test time or cost exceeds the risk cost. Furthermore, we show that it also is possible to model reliability problems with this test-sequencing method. The method is demonstrated on ASML systems with two case studies. The first case study was conducted in the test phase during the development of the software that is used to control an ASML lithographic machine. The second case study was conducted on the reliability testing of a lithographic machine.      

CHOOSE: Towards a metamodel for enterprise architecture in small and medium-sized enterprises

Enterprise architecture (EA) is a coherent whole of principles, methods, and models that are used in the design and realization of an enterprise’s organizational structure, business processes, information systems, and IT infrastructure. Recent research indicates the need for EA in small and medium-sized enterprises (SMEs), important drivers of the economy, as they struggle with problems related to a lack of structure and overview of their business. However, existing EA frameworks are perceived as too complex and, to date, none of the EA approaches are sufficiently adapted to the SME context. Therefore, this paper presents the CHOOSE metamodel for EA in SMEs that was developed and evaluated through action research in an SME and further refined and validated through case study research in five other SMEs. This metamodel is based on the essential dimensions of EA frameworks and is kept simple so that it may be applied in an SME context. The final CHOOSE metamodel includes only four essential concepts (i.e. goal, actor, operation, object), one for each most frequently used EA focus. As an example, an extract is included from the specific model that was created for the SME used in our action research. Finally, the CHOOSE metamodel is evaluated according to the dimensions essential in EA and the requirements for EA in an SME context.     

基于风险测试揭错能力分析

在介绍了基于风险测试的基本原理后,通过简化的测试模型讨论了基于风险测试的方法在软件揭错能力上的改善。并进一步讨论了 该方法在实际操作中的实施关健,认识其本身风险性及规避方法。     

An investigation of ‘build vs. buy’ decision for software acquisition by small to medium enterprises

ContextThe prevalence of computing and communication technologies, combined with the availability of sophisticated and highly specialised software packages from software vendors has made package acquisition a viable option for many organisations. While some research has addressed the factors that influence the selection of the software acquisition method in large organisations, little is known about the factors affecting SMEs.ObjectiveTo provide an understanding of factors that affect the decision process of software acquisition for SMEs. It is expected that results from this study: (i) will assist the SME decision process for software acquisition, and (ii) will assist policy makers in terms of developing appropriate guidelines for SME software acquisition.MethodA positivist research perspective has been adopted involving semi-structured interviews in eight SMEs in Thailand with the interviewees assigning to each of the potential factors.ResultsThe study found that the following factors affect both SMEs and large organisations: requirements fit, cost, scale and complexity, commoditization/flexibility, time, in-house experts, support structure, and operational factors. Factors mainly applying to large organisations were strategic role of the software, intellectual property concerns, and risk, Factors particularly relevant to SMEs (ubiquitous systems, availability of free download, and customizable to specific government/tax regulations).ConclusionThe results suggest that: (i) when deciding on their software acquisition method, SMEs are generally less likely to pursue a long-term vision compared with larger organisations, possibly because SMEs mainly serve their local markets; and (ii) contrary to the large organisations, the role that the IT plays in SMEs may not be as vital to the SMEs’ core business processes, to their supply chains, and/or to the management of their customer relationship. Furthermore, neither the level of technological intensity nor size of the SME appears to affect the ranks given by the interviewees for the various factors.     

A taxonomy of risk-based testing

Software testing has often to be done under severe pressure due to limited resources and a challenging time schedule facing the demand to assure the fulfillment of the software requirements. In addition, testing should unveil those software defects that harm the mission-critical functions of the software. Risk-based testing uses risk (re-)assessments to steer all phases of the test process to optimize testing efforts and limit risks of the software-based system. Due to its importance and high practical relevance, several risk-based testing approaches were proposed in academia and industry. This paper presents a taxonomy of risk-based testing providing a framework to understand, categorize, assess, and compare risk-based testing approaches to support their selection and tailoring for specific purposes. The taxonomy is aligned with the consideration of risks in all phases of the test process and consists of the top-level classes risk drivers, risk assessment, and risk-based test process. The taxonomy of risk-based testing has been developed by analyzing the work presented in available publications on risk-based testing. Afterwards, it has been applied to the work on risk-based testing presented in this special section of the International Journal on Software Tools for Technology Transfer.     

SPI success factors within product usability evaluation

This article presents an experience report where we compare 8 years of experience of product related usability testing and evaluation with principles for software process improvement (SPI). In theory the product and the process views are often seen to be complementary, but studies of industry have demonstrated the opposite. Therefore, more empirical studies are needed to understand and improve the present situation. We find areas of close agreement as well as areas where our work illuminates new characteristics. It has been identified that successful SPI is dependent upon being successfully combined with a business orientation. Usability and business orientation also have strong connections although this has not been extensively addressed in SPI publications. Reasons for this could be that usability focuses on product metrics whilst today's SPI mainly focuses on process metrics. Also because today's SPI is dominated by striving towards a standardized, controllable, and predictable software engineering process; whilst successful usability efforts in organisations are more about creating a creative organisational culture advocating a useful product throughout the development and product life cycle. We provide a study and discussion that supports future development when combining usability and product focus with SPI, in particular if these efforts are related to usability process improvement efforts.     

Generic manufacturing information systems development via template prototyping

There are a number of difficulties with software support for production management decision making in small manufacturing enterprises (SME). System requirement specification and total system cost are two major hurdles. A generic approach to the system design will enable the development costs to be spread across a group of users, while prototyping offers the potential to involve users in the system specification and design. However existing prototyping methodologies have not focused on producing generic software. The template prototyping concepts developed here, together with the derived methodology, based on object-orientated structures, contribute significantly to overcoming these difficulties. The use of the developed software in practice and as a learning tool in computer aided production management for SME environments is outlined.     

功能点法在软件造价评估中的应用

With the continuous improvement of IT technology, informatization and digitization have become the important driving en- gines for the management improvement and business promotion of most medium and large enterprises, and the demand for informa- tion system software of various types of enterprises is also increasing. However, in terms of software cost evaluation, it is difficult for Party A and Party B to reach an agreement on the project scale, workload and R & D cost, so the unified evaluation standard and basis are urgently needed. This paper introduces the function point method, and explains the application of NESMA function point method in software cost in detail, which provides a standard for the unified understanding of project Party A and Party B, and pro- vides the basis for enterprises to standardize the cost evaluation system.     

ProcessTalk A natural language interface for industrial applications

Many commercial software packages are available for industrial applications. However, small and medium size enterprises, SME, normally lack the in-house qualified personnel to benefit from such packages. As natural languages are the best means of communication, a natural language interface can therefore be a good means to grant the SME an easy access to the sophisticated computerized software packages, thus alleviating the problem of the lack of qualified personnel and reducing the cost of employees' learning curve. At present, no such interfaces exist.

This paper presents the design of a human-machine interface, ProcessTalk, that would accept natural language commands in a text form, then would invoke and execute the appropriate functional/procedural commands required by the desired commercial software package the same way as qualified analysts and engineers would operate. It also exposes its benefits and some of its industrial applications.     

一类小型软件企业实施CMM2级的研究

分析了基于SSR需求的测试项目的特点和以该类项目为主的小型软件企业的特点，并通过实例阐述了这类组织实施CMM2级的一些要点和实施过程中要注意的问题。根据对项目和组织的特点分析及对实施CMM2级后组织的能力成熟度的提升和软件过程改进后的效果的分析，得出实施CMM2级能够使以基于SSR需求的测试项目为主的小型软件企业实现有效的软件过程改进。     

软件执行过程的加权复杂网络

软件规模庞大,结构复杂,使软件测试和软件可信性评价成为一个亟待解决的理论问题,可信软件研究也因此成为目前软件工程研究的热点。实验证明大型软件的静态函数调用网络和动态执行所得到的函数调用加权（函数调用次数）网络都具有小世界效应和无标度特性,这就为采用基于关键模块和关键路径的软件测试方法提供了理论依据,从而提高测试效率,降低测试费用。提出基于关键路径的测试方法,利用较少的测试用例覆盖较多的软件执行过程。     

Organizational information systems competences in small and medium-sized enterprises

We used resource-based theory and evidence from empirical studies to evolve a framework of IS competences in small and medium-sized enterprises (SMEs). The framework significantly improved our understanding of internal IS expertise in SMEs. We used relevant IS competence and SME literature, as well as empirical data from SME case studies. Our set of twenty two IS competences were organized around six macro competences. Each competence refers to a specific ability at the organizational rather than the individual level and they cover a broad range of activities, such as those associated with recognising business opportunities, IS planning, accessing IS knowledge, defining requirements, software and hardware sourcing, applications development, and managing relationships with IS suppliers. The framework was tested against prior literature, including studies of IS adoption, IS success, and entrepreneurial competence. Each competence was fully explained and discussed using evidence from the case studies. The framework creates a comprehensive set of IS competences that can be used in both SME practice and research.     

Potential of critical e-applications for engaging SMEs in e-business: a provider perspective

Against a background of the low engagement of small and medium-sized enterprises (SMEs) in e-business, this paper investigates the emergence of, and potential for, critical e-applications defined as ‘an e-business application, promoted by a trusted third party, which engages a significant number of SMEs by addressing an important shared business concern within an aggregation.’ By a review of secondary data and empirical investigation with service providers and other intermediaries, the research shows that such applications can facilitate the engagement of SME aggregations. There are three key findings, namely: the emergence of aggregation-specific e-business applications; the emergence of collaboratively based ‘one to many’ business models; and the importance of trusted third parties in the adoption of higher-level complexity e-business applications by SMEs. Significantly, this work takes a deliberately provider perspective and complements the already considerable literature on SME IT adoption from a user and network perspective. In terms of future research, the importance of a better conceptual understanding of the impact of complexity on the adoption of information technologies by SMEs is highlighted.     

Software process improvement in small and medium software enterprises: a systematic review

Small and medium enterprises are a very important cog in the gears of the world economy. The software industry in most countries is composed of an industrial scheme that is made up mainly of small and medium software enterprises—SMEs. To strengthen these types of organizations, efficient Software Engineering practices are needed—practices which have been adapted to their size and type of business. Over the last two decades, the Software Engineering community has expressed special interest in software process improvement (SPI) in an effort to increase software product quality, as well as the productivity of software development. However, there is a widespread tendency to make a point of stressing that the success of SPI is only possible for large companies. In this article, a systematic review of published case studies on the SPI efforts carried out in SMEs is presented. Its objective is to analyse the existing approaches towards SPI which focus on SMEs and which report a case study carried out in industry. A further objective is that of discussing the significant issues related to this area of knowledge, and to provide an up-to-date state of the art, from which innovative research activities can be thought of and planned.

Cluster analysis application for understanding SME manufacturing strategies

Small and medium size enterprises’ (SME) manufacturing strategy configurations are identified in a small developed economy with the aim to explore how SME manufacturing strategy configurations affect business stability and performance during a period of macroeconomic shock. Drawing on a survey-based dataset, our two-step cluster analysis results suggest that three distinctive manufacturing strategy configurations can be observed among the SMEs of the Finnish manufacturing sector, namely: Responsive niche-innovators, Subcontractors, and Engineer-servers. Furthermore, we are able to establish a link between the strategy configurations and business stability and performance. The results support conclusions that the nature of manufacturing strategy taxonomies are driven by the business context, and that volume flexibility, design flexibility and service provision capabilities enable better business outcomes during macroeconomic shocks, in comparison to the more easily achievable conformance quality as well as delivery speed and dependability. In light of this research, best performing cluster under the macroeconomic shock is the Engineer-servers, emphasizing flexibility-oriented broad product line and after sales service, while having less priority concerning low price and volume flexibility. The results offer important insights for managers, but also for other stakeholders in the form of for example expert systems development for SME funding decisions.     

An evaluation of simulation to support contract costing

As global markets have become more customer oriented, rapid response rates are now often among the most important metrics in business. To achieve the required agility many companies have turned to outsourcing in order to focus on developing their core activities. By its nature, outsourcing increases the complexity of supply chain networks as more companies are drawn into global logistics networks. The relationships between each of these companies are controlled by contractual agreements. The fast pace of modern industry means that these contracts are entered into relatively quickly often without a full understanding of the true cost implications. This paper presents research conducted as part of a project with the aim of developing contract costing software for outsourcing enterprises. Findings are presented from a study conducted on a number of companies in the electronics sector. A simulation-based study focused on one of these companies, with some associated experimentation, is also presented.     

Software security varies greatly

Veracode has analyzed more than 9,000 applications over the past 18 months, across 40 different industry sectors. These applications are both internally developed enterprise applications and those purchased by enterprises from software vendors. We measured the security quality of third-party software from large and small software vendors and compared the security quality of soft-ware written different languages for different industry sectors. The paper will show that there are significant differences in the quantity and types of vulnerabilities in software due to differences in where the software was developed, the type of software it is, in what language it was developed, and for what type of business the software was developed for.     

A Risk Poker Based Testing Model for Scrum

In agile software development, project estimation often depends on group discussion and expert opinions. Literature claims that group discussion in risk analysis helps to identify some of the crucial issues that might affect development, testing, and implementation. However, risk prioritization often relies on individual expert judgment. Therefore, Risk Poker, a lightweight risk-based testing methodology in which risk analysis is performed through group discussion that outperforms the individual analyst’s estimation is introduced in agile methods. Keeping in view aforementioned benefits Risk Poker can offer, unfortunately, no study has been conducted to empirically prove its ability to improve the testing process to date. Therefore, this research is aimed at closing this research gap by (i) deploying Risk Poker technique as a risk-based strategy in the agile development lifecycle, and (ii) empirically evaluating improvement of the proposed test process. For this purpose, Risk Poker technique is coupled with test coverage for an innovated testing process in an agile project following Scrum in order to provide adequate test coverage for testing activity. A case study was conducted with 6 teams of undergraduate students to estimate test coverage using Risk Poker for an e-commerce system. Three teams estimated their user stories using Risk Poker, while the rest estimated individually and used an average to obtain the statistical combination. The results showed that the proposed usage of Risk Poker for risk analysis and estimate test coverage outperformed the averaged statistical estimation of risk analysis for user stories.