Usability evaluation methods for the web: A systematic mapping study

Context

In recent years, many usability evaluation methods (UEMs) have been employed to evaluate Web applications. However, many of these applications still do not meet most customers’ usability expectations and many companies have folded as a result of not considering Web usability issues. No studies currently exist with regard to either the use of usability evaluation methods for the Web or the benefits they bring.

Objective

The objective of this paper is to summarize the current knowledge that is available as regards the usability evaluation methods (UEMs) that have been employed to evaluate Web applications over the last 14 years.

Method

A systematic mapping study was performed to assess the UEMs that have been used by researchers to evaluate Web applications and their relation to the Web development process. Systematic mapping studies are useful for categorizing and summarizing the existing information concerning a research question in an unbiased manner.

Results

The results show that around 39% of the papers reviewed reported the use of evaluation methods that had been specifically crafted for the Web. The results also show that the type of method most widely used was that of User Testing. The results identify several research gaps, such as the fact that around 90% of the studies applied evaluations during the implementation phase of the Web application development, which is the most costly phase in which to perform changes. A list of the UEMs that were found is also provided in order to guide novice usability practitioners.

Conclusions

From an initial set of 2703 papers, a total of 206 research papers were selected for the mapping study. The results obtained allowed us to reach conclusions concerning the state-of-the-art of UEMs for evaluating Web applications. This allowed us to identify several research gaps, which subsequently provided us with a framework in which new research activities can be more appropriately positioned, and from which useful information for novice usability practitioners can be extracted.     

Network mapping services for dynamic selection of web services: promises and challenges

Web services provide the possibility of dynamically integrating distributed service components scattered over the Internet to fulfill sophisticated business demands. The effectiveness of service integration depends critically on the selection of best-fit service components. In this paper, we first review several research streams related to server selection. We then propose a framework of dynamic service discovery, termed Network Mapping Services (NMS), which combines prior efforts on network-aware applications and server selection into a coherent structure. As successful implementation of NMS requires the collaboration among all participating service providers, we also discuss several incentive schemes and compare their effectiveness in supporting collaboration.This research was sponsored in part by the National Science Foundation under contract number IIS-0219825.     

A systematic mapping study of gamification models oriented to motivational characteristics

ABSTRACT

Context: Gamification focuses on the improvement of users' engagement when performing tasks by making use of game mechanics and elements in order to increase their motivations. Many researches have developed gamification models supporting a variety of motivational characteristics to provide engagement solutions for different areas. Objective: This paper carries out a systematic mapping in the field of gamification, looking for models with motivational characteristics in an attempt to characterise the state of the art of this field, identifying gaps and tendencies for further research. Method: We carried out a systematic mapping aiming at finding the primary studies in the existing literature, which were later classified and analysed according to twelve criteria. Results: We analysed 70 papers that resulted in 17 primary studies, published until September 2016. Most of them focus on Education, making use of Gamification to increase the motivation of a learning process. The gamification mechanics and elements most used were Badges/Achievements and Points/ExperiencePoints(XP), and most of the studies were not validated, thus not providing empirical evidence of the impact of gamification. Conclusions: Existing research in the field is somehow preliminary, and more research effort to analyse the applicability of the models and their respective evaluations would be needed.     

Software evolution visualization: A systematic mapping study

BackgroundSoftware evolution is an important topic in software engineering. It generally deals with large amounts of data, as one must look at whole project histories as opposed to their current snapshot. Software visualization is the field of software engineering that aims to help people to understand software through the use of visual resources. It can be effectively used to analyze and understand the large amount of data produced during software evolution.ObjectiveThis study investigates Software Evolution Visualization (SEV) approaches, collecting evidence about how SEV research is structured, synthesizing current evidence on the goals of the proposed approaches and identifying key challenges for its use in practice.MethodsA mapping study was conducted to analyze how the SEV area is structured. Selected primary studies were classified and analyzed with respect to nine research questions.ResultsSEV has been used for many different purposes, especially for change comprehension, change prediction and contribution analysis. The analysis identified gaps in the studies with respect to their goals, strategies and approaches. It also pointed out to a widespread lack of empirical studies in the area.ConclusionResearchers have proposed many SEV approaches during the past years, but some have failed to clearly state their goals, tie them back to concrete problems, or formally validate their usefulness. The identified gaps indicate that there still are many opportunities to be explored in the area.     

Extensible soft semantic web services agent

Web services technology is critical for the success of business integration and other application fields such as bioinformatics. However, there are two challenges facing the practicality of Web services: (a) efficient location of the Web service registries that contain the requested Web services and (b) efficient retrieval of the requested services from these registries with high quality of service (QoS). The main reason for this problem is that current Web services technology is not semantic-oriented. Several proposals have been made to add semantics to Web services to facilitate discovery and composition of relevant Web services. Such proposals are being referred to as semantic Web services (SWS). However, most of these proposals do not address the second problem of retrieval of web services with high QoS. In this paper, we propose a framework called soft semantic Web services agent (soft SWS agent) for providing high QoS Semantic Web services using soft computing methodology. Since different application domains have different requirement for QoS, it is impractical to use classical mathematical modeling methods to evaluate the QoS of semantic Web services. We use fuzzy neural networks with Genetic Algorithms (GA) as our study case. Simulation results show that the soft computing methodology is practicable to handle fuzzy and uncertain QoS metrics effectively.     

Software development in startup companies: A systematic mapping study

ContextSoftware startups are newly created companies with no operating history and fast in producing cutting-edge technologies. These companies develop software under highly uncertain conditions, tackling fast-growing markets under severe lack of resources. Therefore, software startups present a unique combination of characteristics which pose several challenges to software development activities.ObjectiveThis study aims to structure and analyze the literature on software development in startup companies, determining thereby the potential for technology transfer and identifying software development work practices reported by practitioners and researchers.MethodWe conducted a systematic mapping study, developing a classification schema, ranking the selected primary studies according their rigor and relevance, and analyzing reported software development work practices in startups.ResultsA total of 43 primary studies were identified and mapped, synthesizing the available evidence on software development in startups. Only 16 studies are entirely dedicated to software development in startups, of which 10 result in a weak contribution (advice and implications (6); lesson learned (3); tool (1)). Nineteen studies focus on managerial and organizational factors. Moreover, only 9 studies exhibit high scientific rigor and relevance. From the reviewed primary studies, 213 software engineering work practices were extracted, categorized and analyzed.ConclusionThis mapping study provides the first systematic exploration of the state-of-art on software startup research. The existing body of knowledge is limited to a few high quality studies. Furthermore, the results indicate that software engineering work practices are chosen opportunistically, adapted and configured to provide value under the constrains imposed by the startup context.     

Client-side selection of replicated web services: An empirical assessment

Replicating web services over physically distributed servers can offer client applications a number of QoS benefits, including higher availability and reduced response time. However, selecting the “best” service replica to invoke at the client-side is not a trivial task, as this requires taking into account factors such as local and external network conditions, and the servers’ current workload. This paper presents an empirical assessment of five representative client-side service selection policies for accessing replicated web services. The assessment measured the response time obtained with each of the five policies, at two different client configurations, when accessing a world-wide replicated service with four replicas located in three continents. The assessment’s results were analyzed both quantitatively and qualitatively. In essence, the results show that, in addition to the QoS levels provided by the external network and the remote servers, characteristics of the local client environment can have a significant impact on the performance of some of the policies investigated. In this regard, the paper presents a set of guidelines to help application developers in identifying a server selection policy that best suits a particular service replication scenario.     

Gamification in software engineering – A systematic mapping

ContextGamification seeks for improvement of the user’s engagement, motivation, and performance when carrying out a certain task, by means of incorporating game mechanics and elements, thus making that task more attractive. Much research work has studied the application of gamification in software engineering for increasing the engagement and results of developers.ObjectiveThe objective of this paper is to carry out a systematic mapping of the field of gamification in software engineering in an attempt to characterize the state of the art of this field identifying gaps and opportunities for further research.MethodWe carried out a systematic mapping with a view to finding the primary studies in the existing literature, which were later classified and analyzed according to four criteria: the software process area addressed, the gamification elements used, the type of research method followed, and the type of forum in which they were published. A subjective evaluation of the studies was also carried out to evaluate them in terms of methodology, empirical evidence, integration with the organization, and replicability.ResultsAs a result of the systematic mapping we found 29 primary studies, published between January 2011 and June 2014. Most of them focus on software development, and to a lesser extent, requirements, project management, and other support areas. In the main, they consider very simple gamification mechanics such as points and badges, and few provide empirical evidence of the impact of gamification.ConclusionsExisting research in the field is quite preliminary, and more research effort analyzing the impact of gamification in SE would be needed. Future research work should look at other game mechanics in addition to the basic ones and should tackle software process areas that have not been fully studied, such as requirements, project management, maintenance, or testing. Most studies share a lack of methodological support that would make their proposals replicable in other settings. The integration of gamification with an organization’s existing tools is also an important challenge that needs to be taken up in this field.     

QoS驱动的Web services动态计价机制研究

结合Web services的服务质量(QoS)因素和用户经验,对Web services的计价机制进行研究,提出一种QoS驱动的Web services动态计价模型,描述其计价过程,并给出相应的实现算法。通过对模型的简单实现,经测试这一计价模型在准确性和合理性上具有一定优势。     

Quality of service for workflows and web service processes

Workflow management systems (WfMSs) have been used to support various types of business processes for more than a decade now. In workflows or Web processes for e-commerce and Web service applications, suppliers and customers define a binding agreement or contract between the two parties, specifying quality of service (QoS) items such as products or services to be delivered, deadlines, quality of products, and cost of services. The management of QoS metrics directly impacts the success of organizations participating in e-commerce. Therefore, when services or products are created or managed using workflows or Web processes, the underlying workflow engine must accept the specifications and be able to estimate, monitor, and control the QoS rendered to customers. In this paper, we present a predictive QoS model that makes it possible to compute the quality of service for workflows automatically based on atomic task QoS attributes. We also present the implementation of our QoS model for the METEOR workflow system. We describe the components that have been changed or added, and discuss how they interact to enable the management of QoS.     

Software product line testing - A systematic mapping study

Context

Software product lines (SPL) are used in industry to achieve more efficient software development. However, the testing side of SPL is underdeveloped.

Objective

This study aims at surveying existing research on SPL testing in order to identify useful approaches and needs for future research.

Method

A systematic mapping study is launched to find as much literature as possible, and the 64 papers found are classified with respect to focus, research type and contribution type.

Results

A majority of the papers are of proposal research types (64%). System testing is the largest group with respect to research focus (40%), followed by management (23%). Method contributions are in majority.

Conclusions

More validation and evaluation research is needed to provide a better foundation for SPL testing.     

A novel non-functional matchmaking approach between fuzzy user queries and real world web services based on rough sets

We propose in this paper a novel matchmaking approach between fuzzy user queries and real world Web services. The matchmaking spans over a domain dependent classification step that produces fuzzy classification rules for Web services. The elaborated rules leverage a core set of non-functional quality attributes, which is extracted using rough sets theory. Furthermore, these rules are leveraged to classify Web services into categories, which allows reducing the matchmaking space. The experimental results show that our proposed matchmaking approach provides good results in terms of efficiency and precision.     

Combining service-orientation and software product line engineering: A systematic mapping study

ContextService-Orientation (SO) is a rapidly emerging paradigm for the design and development of adaptive and dynamic software systems. Software Product Line Engineering (SPLE) has also gained attention as a promising and successful software reuse development paradigm over the last decade and proven to provide effective solutions to deal with managing the growing complexity of software systems.ObjectiveThis study aims at characterizing and identifying the existing research on employing and leveraging SO and SPLE.MethodWe conducted a systematic mapping study to identify and analyze related literature. We identified 81 primary studies, dated from 2000–2011 and classified them with respect to research focus, types of research and contribution.ResultThe mapping synthesizes the available evidence about combining the synergy points and integration of SO and SPLE. The analysis shows that the majority of studies focus on service variability modeling and adaptive systems by employing SPLE principles and approaches.In particular, SPLE approaches, especially feature-oriented approaches for variability modeling, have been applied to the design and development of service-oriented systems. While SO is employed in software product line contexts for the realization of product lines to reconcile the flexibility, scalability and dynamism in product derivations thereby creating dynamic software product lines.ConclusionOur study summarizes and characterizes the SO and SPLE topics researchers have investigated over the past decade and identifies promising research directions as due to the synergy generated by integrating methods and techniques from these two areas.     

Guidelines for conducting systematic mapping studies in software engineering: An update

ContextSystematic mapping studies are used to structure a research area, while systematic reviews are focused on gathering and synthesizing evidence. The most recent guidelines for systematic mapping are from 2008. Since that time, many suggestions have been made of how to improve systematic literature reviews (SLRs). There is a need to evaluate how researchers conduct the process of systematic mapping and identify how the guidelines should be updated based on the lessons learned from the existing systematic maps and SLR guidelines.ObjectiveTo identify how the systematic mapping process is conducted (including search, study selection, analysis and presentation of data, etc.); to identify improvement potentials in conducting the systematic mapping process and updating the guidelines accordingly.MethodWe conducted a systematic mapping study of systematic maps, considering some practices of systematic review guidelines as well (in particular in relation to defining the search and to conduct a quality assessment).ResultsIn a large number of studies multiple guidelines are used and combined, which leads to different ways in conducting mapping studies. The reason for combining guidelines was that they differed in the recommendations given.ConclusionThe most frequently followed guidelines are not sufficient alone. Hence, there was a need to provide an update of how to conduct systematic mapping studies. New guidelines have been proposed consolidating existing findings.     

一种面向语义网服务的本体映射框架*

本体的异构性阻碍了语义网服务的互操作。从解决语义网服务中本体的异构问题出发,同时考虑到目前的本体映射系统大多效率不高、映射结果不够准确的问题,提出了一种适用于语义网服务的本体映射方法及系统框架。该方法利用机器学习技术来提高本体映射的自动化程度,利用综合评判技术修正映射结果,以提高本体映射的准确率。采用OAEI 2007的基准测试数据集benchmarks进行实验测试,结果表明本系统的性能基本达到预期效果,能够有效地解决语义网服务中的本体异构问题。     

A high performance backoff protocol for fast execution of composite web services

Using web services to expose applications over the Internet is now a widely accepted practice. Currently, there are several ongoing efforts that provide ways to effectively compose web services distributed across different organizations. One of the problems underlying the deployment of such composite services on the web, however, is service co-allocation that arises when a composite service needs to ensure all the required component services to be available for execution at the same time. Motivated by this, this paper presents a new decentralized protocol, named web service co-allocation protocol (WSCP), which can facilitate fast execution of composite web services. The proposed framework is an enhancement of the famous two phase commit protocol through the incorporation of tentative hold phase as well as the employment of a new high performance backoff protocol developed to better address the dynamics of the service co-allocation problem. The simulation results show that the proposed approach yields significant improvements over existing protocols.     

A web services framework for distributed model management

Distributed model management aims to support the wide-spread sharing and usage of decision support models. Web services is a promising technology for supporting distributed model management activities such as model creation and delivery, model composition, model execution and model maintenance to fulfill dynamic decision-support and problem solving requests. We propose a web services based framework for model management (called MM-WS) to support various activities of the model management life cycle. The framework is based on the recently proposed Integrated Service Planning and Execution (ISP & E) approach for web services integration. We discuss encoding of domain knowledge (as individual models) and utilize the MM-WS framework to interleave synthesis of composite models with their execution. A prototypical implementation with an example is used to illustrate the utility of the framework to enable distributed model management and knowledge integration. Benefits and issues of using the framework to support model-based decision-making in organizational contexts are outlined.

Supporting the semi-automatic semantic annotation of web services: A systematic literature review

ContextSemantically annotating web services is gaining more attention as an important aspect to support the automatic matchmaking and composition of web services. Therefore, the support of well-known and agreed ontologies and tools for the semantical annotation of web services is becoming a key concern to help the diffusion of semantic web services.ObjectiveThe objective of this systematic literature review is to summarize the current state-of-the-art for supporting the semantical annotation of web services by providing answers to a set of research questions.MethodThe review follows a predefined procedure that involves automatically searching well-known digital libraries. As a result, a total of 35 primary studies were identified as relevant. A manual search led to the identification of 9 additional primary studies that were not reported during the automatic search of the digital libraries. Required information was extracted from these 44 studies against the selected research questions and finally reported.ResultsOur systematic literature review identified some approaches available for semantically annotating functional and non-functional aspects of web services. However, many of the approaches are either not validated or the validation done lacks credibility.ConclusionWe believe that a substantial amount of work remains to be done to improve the current state of research in the area of supporting semantic web services.     

A framework for QoS-aware binding and re-binding of composite web services

QoS-aware dynamic binding of composite services provides the capability of binding each service invocation in a composition to a service chosen among a set of functionally equivalent ones to achieve a QoS goal, for example minimizing the response time while limiting the price under a maximum value.This paper proposes a QoS-aware binding approach based on Genetic Algorithms. The approach includes a feature for early run-time re-binding whenever the actual QoS deviates from initial estimates, or when a service is not available. The approach has been implemented in a framework and empirically assessed through two different service compositions.