Web application testing: A systematic literature review

ContextThe web has had a significant impact on all aspects of our society. As our society relies more and more on the web, the dependability of web applications has become increasingly important. To make these applications more dependable, for the past decade researchers have proposed various techniques for testing web-based software applications. Our literature search for related studies retrieved 193 papers in the area of web application testing, which have appeared between 2000 and 2013.ObjectiveAs this research area matures and the number of related papers increases, it is important to systematically identify, analyze, and classify the publications and provide an overview of the trends and empirical evidence in this specialized field.MethodsWe systematically review the body of knowledge related to functional testing of web application through a systematic literature review (SLR) study. This SLR is a follow-up and complimentary study to a recent systematic mapping (SM) study that we conducted in this area. As part of this study, we pose three sets of research questions, define selection and exclusion criteria, and synthesize the empirical evidence in this area.ResultsOur pool of studies includes a set of 95 papers (from the 193 retrieved papers) published in the area of web application testing between 2000 and 2013. The data extracted during our SLR study is available through a publicly-accessible online repository. Among our results are the followings: (1) the list of test tools in this area and their capabilities, (2) the types of test models and fault models proposed in this domain, (3) the way the empirical studies in this area have been designed and reported, and (4) the state of empirical evidence and industrial relevance.ConclusionWe discuss the emerging trends in web application testing, and discuss the implications for researchers and practitioners in this area. The results of our SLR can help researchers to obtain an overview of existing web application testing approaches, fault models, tools, metrics and empirical evidence, and subsequently identify areas in the field that require more attention from the research community.     

Graphical user interface (GUI) testing: Systematic mapping and repository

ContextGUI testing is system testing of a software that has a graphical-user interface (GUI) front-end. Because system testing entails that the entire software system, including the user interface, be tested as a whole, during GUI testing, test cases—modeled as sequences of user input events—are developed and executed on the software by exercising the GUI’s widgets (e.g., text boxes and clickable buttons). More than 230 articles have appeared in the area of GUI testing since 1991.ObjectiveIn this paper, we study this existing body of knowledge using a systematic mapping (SM).MethodThe SM is conducted using the guidelines proposed by Petersen et al. We pose three sets of research questions. We define selection and exclusion criteria. From the initial pool of 230 articles, published in years 1991–2011, our final pool consisted of 136 articles. We systematically develop a classification scheme and map the selected articles to this scheme.ResultsWe present two types of results. First, we report the demographics and bibliometrics trends in this domain, including: top-cited articles, active researchers, top venues, and active countries in this research area. Moreover, we derive the trends, for instance, in terms of types of articles, sources of information to derive test cases, types of evaluations used in articles, etc. Our second major result is a publicly-accessible repository that contains all our mapping data. We plan to update this repository on a regular basis, making it a “live” resource for all researchers.ConclusionOur SM provides an overview of existing GUI testing approaches and helps spot areas in the field that require more attention from the research community. For example, much work is needed to connect academic model-based techniques with commercially available tools. To this end, studies are needed to compare the state-of-the-art in GUI testing in academic techniques and industrial tools.     

GUI回归测试的测试脚本修复技术研究

回归测试对修改后的软件重新进行测试,确认修改没有引入新的错误或导致其他代码产生错误。研究GUI回归测试脚本的自动化修复技术,采用有限状态机描述GUI的交互行为,并基于FSM产生测试用例集;使用自动化测试工具生成测试脚本,建立状态迁移和测试脚本的映射关系表;通过比较GUI新、旧版本的FSM得到GUI的变化信息,根据变化信息和映射表对原测试脚本自动修复。实例验证了脚本修复方法的可行性和有效性。     

代码审查自动化研究综述

随着现代软件规模的不断扩大,协作开发成为软件开发的主流趋势,代码审查成为了现代化软件开发的重要工作流程.但由于人工代码评审往往耗费审查者较大精力,且存在审查者不匹配或审查者水平有限等问题,人工代码评审的质量和效率难以保证,且审查后的代码修复也十分费时费力.因此亟需研究人员为代码审查流程进行改进,提供自动化思路.本文对代码审查自动化相关研究进行系统梳理和总结,并重点介绍4种主要方向:审查者推荐、代码变更质量评估、审查意见生成和代码自动修复.并整理了相关方向的148篇研究,对每个方向的研究进行技术分类与分析.随后,本文整理了各方向研究任务的评估方法,并整理出常用的数据集与开源工具.最后,对代码审查自动化领域面临的问题进行梳理,并对未来研究进行展望.     

基于持续集成的Android自动化测试

Android测试方面的研究大多集中在测试工具和框架的实现上,有些工具和框架可以实现测试用例的自动生成和测试脚本的自动执行。然而在项目开发过程中,测试这个活动是需要人工启动的,不能及时有效地保证新增或者修改代码的质量。在 Robotium 测试框架的基础上,通过研究持续集成方案,包括被测代码和测试代码的托管、版本控制,应用的自动构建,测试的自动执行,实现了Android的自动化测试平台。使用该测试平台,可以及时自动地对被测代码的修改进行测试,直观可控地保证了Android应用的质量。     

Analogy-based software development effort estimation: A systematic mapping and review

ContextAnalogy-based Software development Effort Estimation (ASEE) techniques have gained considerable attention from the software engineering community. However, existing systematic map and review studies on software development effort prediction have not investigated in depth several issues of ASEE techniques, to the exception of comparisons with other types of estimation techniques.ObjectiveThe objective of this research is twofold: (1) to classify ASEE studies which primary goal is to propose new or modified ASEE techniques according to five criteria: research approach, contribution type, techniques used in combination with ASEE methods, and ASEE steps, as well as identifying publication channels and trends and (2) to analyze these studies from five perspectives: estimation accuracy, accuracy comparison, estimation context, impact of the techniques used in combination with ASEE methods, and ASEE tools.MethodWe performed a systematic mapping of studies for which the primary goal is to develop or to improve ASEE techniques published in the period 1990–2012, and reviewed them based on an automated search of four electronic databases.ResultsIn total, we identified 65 studies published between 1990 and 2012, and classified them based on our predefined classification criteria. The mapping study revealed that most researchers focus on addressing problems related to the first step of an ASEE process, that is, feature and case subset selection. The results of our detailed analysis show that ASEE methods outperform the eight techniques with which they were compared, and tend to yield acceptable results especially when combining ASEE techniques with Fuzzy Logic (FL) or Genetic Algorithms (GA).ConclusionBased on the findings of this study, the use of other techniques such FL and GA in combination with an ASEE method is promising to generate more accurate estimates. However, the use of ASEE techniques by practitioners is still limited: developing more ASEE tools may facilitate the application of these techniques and then lead to increasing the use of ASEE techniques in industry.     

代码自然性及其应用研究进展

代码自然性(code naturalness)研究是自然语言处理领域和软件工程领域共同的研究热点之一,旨在通过构建基于自然语言处理技术的代码自然性模型,以解决各种软件工程任务.近年来,随着开源软件社区中源代码和数据规模的不断扩大,越来越多的研究人员注重钻研源代码中蕴藏的信息,并且取得了一系列研究成果.但与此同时,代码自然性研究在代码语料库构建、模型构建和任务应用等环节面临许多挑战.鉴于此,从代码自然性技术的代码语料库构建、模型构建和任务应用等方面对近年来代码自然性研究及应用进展进行梳理和总结.主要内容包括:(1)介绍了代码自然性的基本概念及其研究概况;(2)归纳目前代码自然性研究的语料库,并对代码自然性模型建模方法进行分类与总结;(3)总结代码自然性模型的实验验证方法和模型评价指标;(4)总结并归类了目前代码自然性的应用现状;(5)归纳代码自然性技术的关键问题;(6)展望代码自然性技术的未来发展.     

Knowledge management initiatives in software testing: A mapping study

ContextSoftware testing is a knowledge intensive process, and, thus, Knowledge Management (KM) principles and techniques should be applied to manage software testing knowledge.ObjectiveThis study conducts a survey on existing research on KM initiatives in software testing, in order to identify the state of the art in the area as well as the future research. Aspects such as purposes, types of knowledge, technologies and research type are investigated.MethodThe mapping study was performed by searching seven electronic databases. We considered studies published until December 2013. The initial resulting set was comprised of 562 studies. From this set, a total of 13 studies were selected. For these 13, we performed snowballing and direct search to publications of researchers and research groups that accomplished these studies.ResultsFrom the mapping study, we identified 15 studies addressing KM initiatives in software testing that have been reviewed in order to extract relevant information on a set of research questions.ConclusionsAlthough only a few studies were found that addressed KM initiatives in software testing, the mapping shows an increasing interest in the topic in the recent years. Reuse of test cases is the perspective that has received more attention. From the KM point of view, most of the studies discuss aspects related to providing automated support for managing testing knowledge by means of a KM system. Moreover, as a main conclusion, the results show that KM is pointed out as an important strategy for increasing test effectiveness, as well as for improving the selection and application of suited techniques, methods and test cases. On the other hand, inadequacy of existing KM systems appears as the most cited problem related to applying KM in software testing.     

航天嵌入式软件代码逻辑分析

为提高航天嵌入式软件的测试质量、确保航天型号任务的圆满完成,对航天嵌入式软件代码审查重要内容之一的代码逻辑分析进行了研究.通过对软件缺陷的机理、缺陷查找过程、缺陷暴露过程、以及缺陷引发后果的分析,结合多年软件测试工程实践经验的总结,提出了场景分析法、时序分析法、假想故障追源法等10种主要的代码逻辑分析方法.开展了代码逻辑分析方法的应用分析、代码审查与其它测试手段之间的对比分析,通过分析,给出了代码审查的工程适用性说明.研究成果已在航天型号软件第三方评测中全面推广应用,实践数据表明,应用效果良好,使代码审查的缺陷发现率由业界公认的30％～70％提升至90％以上.相关分析方法和分析思路对动态测试设计以及软件缺陷自动化检测工具的研发均具有一定的参考作用.     

Development of service-oriented architectures using model-driven development: A mapping study

ContextModel-Driven Development (MDD) and Service-Oriented Architecture (SOA) are two challenging research areas in software engineering. MDD is about improving software development whilst SOA is a service-based conceptual development style, therefore investigating the available proposals in the literature to use MDD when developing SOA may be insightful. However, no studies have been found with this purpose.ObjectiveThis work aims at assessing the state of the art in MDD for SOA systems. It mainly focuses on: what are the characteristics of MDD approaches that support SOA; what types of SOA are supported; how do they handle non-functional requirements.MethodWe conducted a mapping study following a rigorous protocol. We identified the representative set of venues that should be included in the study. We applied a search string over the set of selected venues. As result, 129 papers were selected and analysed (both frequency analysis and correlation analysis) with respect to the defined classification criteria derived from the research questions. Threats to validity were identified and mitigated whenever possible.ResultsThe analysis allows us to answer the research questions. We highlight: (1) predominance of papers from Europe and written by researchers only; (2) predominance of top-down transformation in software development activities; (3) inexistence of consolidated methods; (4) significant percentage of works without tool support; (5) SOA systems and service compositions more targeted than single services and SOA enterprise systems; (6) limited use of metamodels; (7) very limited use of NFRs; and (8) limited application in real cases.ConclusionThis mapping study does not just provide the state of the art in the topic, but also identifies several issues that deserve investigation in the future, for instance the need of methods for activities other than software development (e.g., migration) or the need of conducting more real case studies.     

源代码漏洞静态分析技术

漏洞这一名词伴随着计算机软件领域的发展已经走过了数十载。自世界上第一个软件漏洞被公开以来,软件安全研究者和工程师们就一直在探索漏洞的挖掘与分析方法。源代码漏洞静态分析是一种能够贯穿整个软件开发生命周期的、帮助软件开发人员及早发现漏洞的技术,在业界有着广泛的使用。然而,随着软件的体量越来越大,软件的功能越来越复杂,如何表示和建模软件源代码是当前面临的一个难题;此外,近年来的研究倾向于将源代码漏洞静态分析和机器学习相结合,试图通过引入机器学习模型提升漏洞挖掘的精度,但如何选择和构建合适的机器学习模型是该研究方向的一个核心问题。本文将目光聚焦于源代码漏洞静态分析技术(以下简称:静态分析技术),通过对该领域相关工作的回顾,将静态分析技术的研究分为两个方向:传统静态分析和基于学习的静态分析。传统静态分析主要是利用数据流分析、污点分析等一系列软件分析技术对软件的源代码进行建模分析;基于学习的静态分析则是将源代码以数值的形式表示并提交给学习模型,利用学习模型挖掘源代码的深层次表征特征和关联性。本文首先阐述了软件漏洞分析技术的基本概念,对比了静态分析技术和动态分析技术的优劣;然后对源代码的表示方法进行了说明。接着,本文对传统静态分析和基于学习的静态分析的一般步骤进行了总结,同时对这两个研究方向典型的研究成果进行了系统地梳理,归纳了它们的技术特点和工作流程,提出了当前静态分析技术中存在的问题,并对该方向上未来的研究工作进行了展望。     

The maturity of maturity model research: A systematic mapping study

ContextMaturity models offer organizations a simple but effective possibility to measure the quality of their processes. Emerged out of software engineering, the application fields have widened and maturity model research is becoming more important. During the last two decades the publication amount steadily rose as well. Until today, no studies have been available summarizing the activities and results of the field of maturity model research.ObjectiveThe objective of this paper is to structure and analyze the available literature of the field of maturity model research to identify the state-of-the-art research as well as research gaps.MethodA systematic mapping study was conducted. It included relevant publications of journals and IS conferences. Mapping studies are a suitable method for structuring a broad research field concerning research questions about contents, methods, and trends in the available publications.ResultsThe mapping of 237 articles showed that current maturity model research is applicable to more than 20 domains, heavily dominated by software development and software engineering. The study revealed that most publications deal with the development of maturity models and empirical studies. Theoretical reflective publications are scarce. Furthermore, the relation between conceptual and design-oriented maturity model development was analyzed, indicating that there is still a gap in evaluating and validating developed maturity models. Finally, a comprehensive research framework was derived from the study results and implications for further research are given.ConclusionThe mapping study delivers the first systematic summary of maturity model research. The categorization of available publications helps researchers gain an overview of the state-of-the-art research and current research gaps. The proposed research framework supports researchers categorizing their own projects. In addition, practitioners planning to use a maturity model may use the study as starting point to identify which maturity models are suitable for their domain and where limitations exist.     

Tools used in Global Software Engineering: A systematic mapping review

ContextThis systematic mapping review is set in a Global Software Engineering (GSE) context, characterized by a highly distributed environment in which project team members work separately in different countries. This geographic separation creates specific challenges associated with global communication, coordination and control.ObjectiveThe main goal of this study is to discover all the available communication and coordination tools that can support highly distributed teams, how these tools have been applied in GSE, and then to describe and classify the tools to allow both practitioners and researchers involved in GSE to make use of the available tool support in GSE.MethodWe performed a systematic mapping review through a search for studies that answered our research question, “Which software tools (commercial, free or research based) are available to support Global Software Engineering?” Applying a range of related search terms to key electronic databases, selected journals, and conferences and workshops enabled us to extract relevant papers. We then used a data extraction template to classify, extract and record important information about the GSD tools from each paper. This information was synthesized and presented as a general map of types of GSD tools, the tool’s main features and how each tool was validated in practice.ResultsThe main result is a list of 132 tools, which, according to the literature, have been, or are intended to be, used in global software projects. The classification of these tools includes lists of features for communication, coordination and control as well as how the tool has been validated in practice. We found that out the total of 132, the majority of tools were developed at research centers, and only a small percentage of tools (18.9%) are reported as having been tested outside the initial context in which they were developed.ConclusionThe most common features in the GSE tools included in this study are: team activity and social awareness, support for informal communication, Support for Distributed Knowledge Management and Interoperability with other tools. Finally, there is the need for an evaluation of these tools to verify their external validity, or usefulness in a wider global environment.     

企业级海量代码的检索与管理技术

在大型IT企业中,尤其像Google或者百度,代码搜索已是软件开发过程中不可或缺且频繁的活动,其通过借鉴或复用已有代码,加速开发过程的速度.多年以来,已有大量的研究人员关注代码搜索,且设计出很多优秀的工具.但是已有的研究和工具主要是在小规模或者编程语言单一的代码数据集上,没有从企业实际搜索需求出发,且对用户的查询输入也有所限制,尚缺少一套针对企业级海量代码的检索与管理技术方案.提出了一套企业级海量数据代码搜索引擎的方案和系统实现,面向开发过程中用户最直接的需求,通过离线分析与在线分析,完成对海量代码库的索引构建与检索.其中,离线分析负责代码相关数据的获取与分析、构建索引集群.在线过程负责变换用户的query、对搜索的结果进行高级排序、生成摘要.本系统部署在百度代码库上,为数十TB级的Git代码库构建了索引,平均一次检索时间在1s之内.在百度推出应用以来,访问量逐步增加,现每周平均用户有数千人,每周查询平均有数万次,广受百度工程师好评.     

A systematic literature review on the usage of eye-tracking in software engineering

ContextEye-tracking is a mean to collect evidence regarding some participants’ cognitive processes. Eye-trackers monitor participants’ visual attention by collecting eye-movement data. These data are useful to get insights into participants’ cognitive processes during reasoning tasks.ObjectiveThe Evidence-based Software Engineering (EBSE) paradigm has been proposed in 2004 and, since then, has been used to provide detailed insights regarding different topics in software engineering research and practice. Systematic Literature Reviews (SLR) are also useful in the context of EBSE by bringing together all existing evidence of research and results about a particular topic. This SLR evaluates the current state of the art of using eye-trackers in software engineering and provides evidence on the uses and contributions of eye-trackers to empirical studies in software engineering.MethodWe perform a SLR covering eye-tracking studies in software engineering published from 1990 up to the end of 2014. To search all recognised resources, instead of applying manual search, we perform an extensive automated search using Engineering Village. We identify 36 relevant publications, including nine journal papers, two workshop papers, and 25 conference papers.ResultsThe software engineering community started using eye-trackers in the 1990s and they have become increasingly recognised as useful tools to conduct empirical studies from 2006. We observe that researchers use eye-trackers to study model comprehension, code comprehension, debugging, collaborative interaction, and traceability. Moreover, we find that studies use different metrics based on eye-movement data to obtain quantitative measures. We also report the limitations of current eye-tracking technology, which threaten the validity of previous studies, along with suggestions to mitigate these limitations.ConclusionHowever, not withstanding these limitations and threats, we conclude that the advent of new eye-trackers makes the use of these tools easier and less obtrusive and that the software engineering community could benefit more from this technology.     

FADEC可靠性试验HIL仿真自动化技术研究

针对航空发动机全权限数字电子控制(FADEC)系统电子控制器可靠性试验中硬件在环(HIL)仿真测试的执行过程,开展了测试自动化技术研究,提出按名义执行时间设计脚本的分层封装方法,实现脚本模块执行时间误差受控,提高了脚本的模块化程度,有利于各层脚本执行时间的精确调试和顶层测试任务脚本的开发。设计了外部辅助脚本的自动流程整合方式,实现非自动化软件工具的快速整合,提高了自动运行流程的构建效率。经过某型电子控制器1000 h可靠性摸底试验验证,测试流程自动化运行流畅,报表输出完整,任务脚本执行时间误差稳定受控,满足试验过程的自动化需求。     

一种基于C#的Cantata工具变更过程改进方法

航空发动机控制软件在升级过程中使用Cantata工具开展单元测试活动时,存在未变更函数的Cantata测试脚本需重新人工隔离插桩导致时间和人力耗费的问题.通过研究Cantata自动生成测试脚本的过程及插桩特点,提出了一种基于C#的Cantata工具变更过程改进方法.该方法通过C#语言结合正则表达式进行代码分析,识别出升级过程中的变更函数和全局变量,并按照Cantata插桩格式,自动完成测试脚本更新工作.详细介绍了该方法的设计过程,并在某型航空发动机控制软件升级过程中进行实践应用.实践结果表明,该方法可准确识别源码信息并完成变更前后的差异比对,能正确快速地实现未变更函数的自动隔离插桩工作,有效解决了人力和时间消耗的问题,对回归测试效率有极大提升.     

基于图嵌入的软件项目源代码检索方法

源代码检索是软件工程领域的一项重要研究问题,其主要任务是检索和复用软件项目API（application program interface,应用程序接口）.随着软件项目的规模越来越大、越来越复杂,当前,源代码检索一方面需要提高基于自然语言API查询的准确性,另一方面需要定位和展示目标API及其相关代码之间的关联,以更好地辅助用户理解API的实现逻辑和使用场景.为此,提出一种基于图嵌入的软件项目源代码检索方法.该方法能够基于软件项目源代码自动构建其代码结构图,并通过图嵌入对源代码进行信息表示.在此基础上,用户可以输入自然语言问题、检索并返回相关的API及其关联信息构成的连通代码子图,从而提高API检索和复用的效率.在以开源项目Apache Lucene和POI为例的检索实验中,该方法检索结果的F1值比现有基于最短路径的方法提高了10%,同时显著缩短了平均响应时间.     

代码知识图谱构建及智能化软件开发方法研究

智能化软件开发正在经历从简单的代码检索到语义赋能的代码自动生成的转变,传统的语义表达方式无法有效地支撑人、机器和代码之间的语义交互,探索机器可理解的语义表达机制迫在眉睫.首先指出了代码知识图谱是实现智能化软件开发的基础,进而分析了大数据时代智能化软件开发的新特点以及基于代码知识图谱进行智能化软件开发的新挑战;随后回顾了智能化软件开发和代码知识图谱的研究现状,指出了现有智能化软件开发的研究仍然处于较低水平,而现有知识图谱的研究主要面向开放领域知识图谱,无法直接应用于代码领域知识图谱.因此,从代码知识图谱的建模与表示、构建与精化、存储与演化管理、查询语义理解以及智能化应用这5个方面详细探讨了研究新趋势,以更好地满足基于代码知识图谱进行智能化软件开发的需要.     

基于深度学习的源代码缺陷检测研究综述

源代码缺陷检测是判别程序代码中是否存在非预期行为的过程,广泛应用于软件测试、软件维护等软件工程任务,对软件的功能保障与应用安全方面具有至关重要的作用.传统的缺陷检测研究以程序分析为基础,通常需要很强的领域知识与复杂的计算规则,面临状态爆炸问题,导致检测性能有限,在误报漏报率上都有较大提高空间.近年来,开源社区的蓬勃发展积累了以开源代码为核心的海量数据,在此背景下,利用深度学习的特征学习能力能够自动学习语义丰富的代码表示,从而为缺陷检测提供一种新的途径.搜集了该领域最新的高水平论文,从缺陷代码数据集与深度学习缺陷检测模型两方面系统地对当前方法进行了归纳与阐述.最后对该领域研究所面临的主要挑战进行总结,并展望了未来可能的研究重点.