SPI success factors within product usability evaluation

This article presents an experience report where we compare 8 years of experience of product related usability testing and evaluation with principles for software process improvement (SPI). In theory the product and the process views are often seen to be complementary, but studies of industry have demonstrated the opposite. Therefore, more empirical studies are needed to understand and improve the present situation. We find areas of close agreement as well as areas where our work illuminates new characteristics. It has been identified that successful SPI is dependent upon being successfully combined with a business orientation. Usability and business orientation also have strong connections although this has not been extensively addressed in SPI publications. Reasons for this could be that usability focuses on product metrics whilst today's SPI mainly focuses on process metrics. Also because today's SPI is dominated by striving towards a standardized, controllable, and predictable software engineering process; whilst successful usability efforts in organisations are more about creating a creative organisational culture advocating a useful product throughout the development and product life cycle. We provide a study and discussion that supports future development when combining usability and product focus with SPI, in particular if these efforts are related to usability process improvement efforts.     

Reconciling usability and interactive system architecture using patterns

Traditional interactive system architectures such as MVC [Goldberg, A., 1984. Smaltalk-80: The Interactive Programming Environment, Addison-Wesley Publ.] and PAC [Coutaz, J., 1987. PAC, an implementation model for dialog design. In: Interact’87, Sttutgart, September 1987, pp. 431-436; Coutaz, J., 1990. Architecture models for interactive software: faillures and trends. In: Cockton, G. (Ed.), Engineering for Human-Computer Interaction, Elsevier Science Publ., pp. 137-153.] decompose the system into subsystems that are relatively independent, thereby allowing the design work to be partitioned between the user interfaces and underlying functionalities. Such architectures extend the independence assumption to usability, approaching the design of the user interface as a subsystem that can designed and tested independently from the underlying functionality. This Cartesian dichotomy can be fallacious, as functionalities buried in the application’s logic can sometimes affect the usability of the system. Our investigations model the relationships between internal software attributes and externally visible usability factors. We propose a pattern-based approach for dealing with these relationships. We conclude by discussing how these patterns can lead to a methodological framework for improving interactive system architectures, and how these patterns can support the integration of usability in the software design process.     

A method to elicit architecturally sensitive usability requirements: its integration into a software development process

In the Human-Computer Interaction (HCI) community, software usability has primarily been concerned with the presentation of information, more precisely with the user interface. However, some usability problems can prove costly to fix if the changes require modifications that reach beyond the presentation layer, namely those that cannot be easily accommodated by the software architecture. Taking into account some usability requirements earlier in the software development cycle, specifically prior to the architectural design phase, can reduce the cost of these modifications. There is a scarcity of methods and guidelines with the scope to direct users in eliciting the usability requirements that can impact the software architecture. This paper proposes a usability-driven adaptation of the quality attribute workshop (QAW) to assist software development organizations in discovering and documenting usability requirements. It shows how this method can be integrated into a software development process, by discussing how the existing software framework workflows can be adjusted to take this new activity into consideration. A preliminary exercise was conducted to help discern the utility and the limits of the proposed method. Participants with different levels of knowledge of usability and comprehension of the system being developed found the method constructive, as it guided them in identifying the architecturally relevant usability requirements. It also helped determine the usability aspects that would not necessarily have been defined if this technique had not been employed.     

Bridging the gap between information architecture analysis and software engineering in interactive web application development

Web development teams comprise non-computer experts working on the conceptual modeling of non-functional aspects in software applications. Later on, such conceptual information is processed by analysts and software engineers to face the technical phases of the software project. However, this information transfer is often difficult to automate since the information processed by the different professionals involves different abstraction levels, as well as important cost and effort that need to be considered. The main aim of this research is to minimize these problems by increasing automation and interoperability in the development of interactive web applications. To take up this challenge, we have created and evaluated a tool that aims at bridging the gap between the conceptual definitions of web contents–i.e., the information architecture, and the UML elements for analysis and design required by software engineers, connecting functional and non-functional information to achieve the rest of technical activities during the software development process.     

Contextualising design: Aspects of using usability context analysis and hierarchical task analysis for software design

Usability Context Analysis is part of ISO 9241-11 and, as such, is recommended for use in user-centred design. However, studies involving a critical assessment of usability context analysis within the software domain are scarce; this paper details two small studies that involved the use of usability context analysis (UCA) and hierarchical task analysis (HTA). The first system involved designing a small subsite of a larger website, while the second system was a small database designed as part of an exercise by UK university students. Post-design evaluation suggested that the use of both UCA and HTA, rather than one or the other, was beneficial in gathering and analysing user requirements.     

Contextualising design: Aspects of using usability context analysis and hierarchical task analysis for software design

Usability Context Analysis is part of ISO 9241-11 and, as such, is recommended for use in user-centred design. However, studies involving a critical assessment of usability context analysis within the software domain are scarce; this paper details two small studies that involved the use of usability context analysis (UCA) and hierarchical task analysis (HTA). The first system involved designing a small subsite of a larger website, while the second system was a small database designed as part of an exercise by UK university students. Post-design evaluation suggested that the use of both UCA and HTA, rather than one or the other, was beneficial in gathering and analysing user requirements.     

Empirical validation of an automatic usability evaluation method

Today, the success of a software application strongly depends on the usability of its interface, so the evaluation of interfaces has become a crucial aspect of software engineering. It is recognized that automatic tools for graphical user interface evaluation may greatly reduce the costs of traditional activities performed during expert evaluation or user testing in order to estimate the success probability of an application. However, automatic methods need to be empirically validated in order to prove their effectiveness with respect to the attributes they are supposed to evaluate.In this work, we empirically validate a usability evaluation method conceived to assess consistency aspects of a GUI with no need to analyze the back-end. We demonstrate the validity of the approach by means of a comparative experimental study, where four web sites and a stand-alone interactive application are analyzed and the results compared to those of a human-based usability evaluation. The analysis of the results and the statistical correlation between the tool׳s rating and humans׳ average ratings show that the proposed methodology can indeed be a useful complement to standard techniques of usability evaluation.     

Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis

The use of e-learning technology is incontestably recognized as an important and integral part of the educational process. Considerable research studies are carried out in order to apprehend how effective and usable e-learning systems. In this paper, an empirical-based study is conducted to explore how lecturers interact with an e-learning environment based on a predefined task model describing low-level interactions. Client-side log data is collected from university lecturers from the Electrical and Computer Science departments. Subsequently, data analysis is conducted to infer the usability degree from the estimated usage metrics together with further exploratory analysis from user feedback via System Usability Scale. Experimental results reveal that the System Usability Scale score is not a sufficient measure to express the true acceptance and satisfaction level of lecturers for using the e-learning systems. The evaluation must be fulfilled in tandem with analyzing the usage metrics derived from interaction traces in a non-intrusive fashion. The proposed approach is a milestone towards usability evaluation to improve the acceptance and user experience for academic staff and students.     

Software development methods and usability: Perspectives from a survey in the software industry in Norway

This paper investigates the relationship between software development methodologies and usability. The point of departure is the assumption that two important disciplines in software development, one of software development methods (SDMs) and one of usability work, are not integrated in industrial software projects.

Building on previous research we investigate two questions; (1) Will software companies generally acknowledge the importance of usability, but not prioritise it in industrial projects? and (2) To what degree are software development methods and usability perceived by practitioners as being integrated? To this end a survey in the Norwegian IT industry was conducted. From a sample of 259 companies we received responses from 78 companies.

In response to our first research question, our findings show that although there is a positive bias towards usability, the importance of usability testing is perceived to be much less than that of usability requirements. Given the strong time and cost pressures associated with the software industry, we believe that these results highlight that there is a gap between intention and reality. Regarding our second research question our survey revealed that companies perceive usability and software development methods to be integrated. This is in contrast to earlier research, which, somewhat pessimistically, has argued for the existence of two different cultures, one of software development and one of usability. The findings give hope for the future, in particular because the general use of system development methods are pragmatic and adaptable.     

Analysis of architecture evaluation data

The output of 18 software architecture evaluations is analyzed. The goal of the analysis is to find patterns in the important quality attributes and risk themes identified in the evaluations. The major results are

•

A categorization of risk themes.

•

The observation that twice as many risk themes are risks of “omission” as are risks of “commission”.

•

A failure to find a relationship between the business and mission goals of a system and the risk themes from an evaluation of that system.

•

A failure to find a correlation between the domain of a system being evaluated and the important quality attributes for that system.

•

A wide diversity of names used for various quality attributes.

The results of this investigation have application to practitioners by suggesting activities on which developers should put greater focus. They also have application to researchers by suggesting further areas of investigation.     

Validating a model-driven software architecture evaluation and improvement method: A family of experiments

ContextSoftware architectures should be evaluated during the early stages of software development in order to verify whether the non-functional requirements (NFRs) of the product can be fulfilled. This activity is even more crucial in software product line (SPL) development, since it is also necessary to identify whether the NFRs of a particular product can be achieved by exercising the variation mechanisms provided by the product line architecture or whether additional transformations are required. These issues have motivated us to propose QuaDAI, a method for the derivation, evaluation and improvement of software architectures in model-driven SPL development.ObjectiveWe present in this paper the results of a family of four experiments carried out to empirically validate the evaluation and improvement strategy of QuaDAI.MethodThe family of experiments was carried out by 92 participants: Computer Science Master’s and undergraduate students from Spain and Italy. The goal was to compare the effectiveness, efficiency, perceived ease of use, perceived usefulness and intention to use with regard to participants using the evaluation and improvement strategy of QuaDAI as opposed to the Architecture Tradeoff Analysis Method (ATAM).ResultsThe main result was that the participants produced their best results when applying QuaDAI, signifying that the participants obtained architectures with better values for the NFRs faster, and that they found the method easier to use, more useful and more likely to be used. The results of the meta-analysis carried out to aggregate the results obtained in the individual experiments also confirmed these results.ConclusionsThe results support the hypothesis that QuaDAI would achieve better results than ATAM in the experiments and that QuaDAI can be considered as a promising approach with which to perform architectural evaluations that occur after the product architecture derivation in model-driven SPL development processes when carried out by novice software evaluators.     

Quantitative analysis in a heuristic evaluation experiment

In usability context of interactive systems the heuristic evaluation method is widespread. In most applications the results tend to be qualitative, describing such aspects that require some improvement for the benefit of usability. However, these qualitative results do not allow us to determine how usable it is or how it becomes an interactive system. Hence the need for quantitative results may also be very necessary in order to determine the effort that would be needed to get a sufficiently usable system.This article describes, following the idea of the UsabAIPO Project, a new experiment to obtain quantitative results after a heuristic evaluation. This new experimentation has required some variation on the original idea, working with a set of different heuristic categories, while considering the use of the score depending on severity and frequency parameters.     

A comparison of the four prominent user-based methods for evaluating the usability of computer software

The aim was to examine four prominent user-based computer software usability evaluation methods. Four evaluation methods (logged data, questionnaire, inter view, and verbal protocol analysis) were used to evaluate three different business software types (spreadsheet, word processor, and database) using a between-groups design, involving 148 individuals of mixed age and gender. Comparisons were made to examine the efficiency of each evaluation method in terms of its ability to highlight usability problems both between and within the evaluation strategy. Here, the verbal protocol analysis was found to be most efficient. The possibility of further efficiency gains by using two evaluation methods was also examined, where it was found that no statistically significant improvement was obtained over the verbal protocol analysis used by itself. Ways in which the utility of the methods may be enhanced was also discussed.     

An integrated framework for the analysis of dependable interactive systems (IFADIS): Its tool support and evaluation

This paper discusses a method for the analysis of dependable interactive systems using model checking, and its support by a tool designed to make it accessible to a broader community. The method and the tool are designed to be of value to system engineers, usability engineers and software engineers. It has been designed to help usability engineers by making those aspects of the analysis relevant to them explicit while concealing those aspects of modelling and model checking that are not relevant. The paper presents the results of a user evaluation of the effectiveness of aspects of the tool and how it supports the proposed method. The tool was constructed while both authors worked in the Human Computer Interaction Group. Department of Computer Science, University of York, UK.     

Survey of reliability and availability prediction methods from the viewpoint of software architecture

Many future software systems will be distributed across a network, extensively providing different kinds of services for their users. These systems must be highly reliable and provide services when required. Reliability and availability must be engineered into software from the onset of its development, and potential problems must be detected in the early stages, when it is easier and less expensive to implement modifications. The software architecture design phase is the first stage of software development in which it is possible to evaluate how well the quality requirements are being met. For this reason, a method is needed for analyzing software architecture with respect to reliability and availability. In this paper, we define a framework for comparing reliability and availability analysis methods from the viewpoint of software architecture. Our contribution is the comparison of the existing analysis methods and techniques that can be used for reliability and availability prediction at the architectural level. The objective is to discover which methods are suitable for the reliability and availability prediction of today’s complex systems, what are the shortcomings of the methods, and which research activities need to be conducted in order to overcome these identified shortcomings. The comparison reveals that none of the existing methods entirely fulfill the requirements that are defined in the framework. The comparison framework also defines the characteristics required of new reliability and availability analysis methods. Additionally, the framework is a valuable tool for selecting the best suitable method for architecture analysis. Furthermore, the framework can be extended and used for other evaluation methods as well.     

Integrating usability requirements that can be evaluated in design time into Model Driven Engineering of Web Information Systems

In recent years, Web Engineering development projects have grown increasingly complex for and critical to the smooth running of organizations. However, recent studies reveal that a high percentage of these projects fail to attain the quality parameters required by stakeholders. The inadequate consideration of requirements management activities together with the absence of attention to the elicitation and evaluation of requirements and metrics related to certain quality attributes which are of special importance in this kind of systems, such as usability, have proved to be some of the main causes of this failure. This paper attempts to reduce some of the quality failures detected in Web Engineering development projects by proposing the consideration and evaluation of quality attributes from early stages of the development process. The presented approach therefore commences with a reinforcement of the requirements related activities in this discipline, which is carried out by using a requirements metamodel. Once these requirements have been identified, the approach focuses on the extension of the conceptual models used by Web Engineering methodologies with the aim of allowing the explicit consideration of usability requirements along with the evaluation of quality metrics during the design of the system. An example of an application illustrating how the approach can be used, along with the automatic support which was developed for it, are also shown.     

A systematic review of software architecture visualization techniques

ContextGiven the increased interest in using visualization techniques (VTs) to help communicate and understand software architecture (SA) of large scale complex systems, several VTs and tools have been reported to represent architectural elements (such as architecture design, architectural patterns, and architectural design decisions). However, there is no attempt to systematically review and classify the VTs and associated tools reported for SA, and how they have been assessed and applied.ObjectiveThis work aimed at systematically reviewing the literature on software architecture visualization to develop a classification of VTs in SA, analyze the level of reported evidence and the use of different VTs for representing SA in different application domains, and identify the gaps for future research in the area.MethodWe used systematic literature review (SLR) method of the evidence-based software engineering (EBSE) for reviewing the literature on VTs for SA. We used both manual and automatic search strategies for searching the relevant papers published between 1 February 1999 and 1 July 2011.ResultsWe selected 53 papers from the initially retrieved 23,056 articles for data extraction, analysis, and synthesis based on pre-defined inclusion and exclusion criteria. The results from the data analysis enabled us to classify the identified VTs into four types based on the usage popularity: graph-based, notation-based, matrix-based, and metaphor-based VTs. The VTs in SA are mostly used for architecture recovery and architectural evolution activities. We have also identified ten purposes of using VTs in SA. Our results also revealed that VTs in SA have been applied to a wide range of application domains, among which “graphics software” and “distributed system” have received the most attention.ConclusionSA visualization has gained significant importance in understanding and evolving software-intensive systems. However, only a few VTs have been employed in industrial practice. This review has enabled us to identify the following areas for further research and improvement: (i) it is necessary to perform more research on applying visualization techniques in architectural analysis, architectural synthesis, architectural implementation, and architecture reuse activities; (ii) it is essential to pay more attention to use more objective evaluation methods (e.g., controlled experiment) for providing more convincing evidence to support the promised benefits of using VTs in SA; (iii) it is important to conduct industrial surveys for investigating how software architecture practitioners actually employ VTs in architecting process and what are the issues that hinder and prevent them from adopting VTs in SA.     

Using simulation to aid decision making in managing the usability evaluation process

ContextThis paper is developed in the context of Usability Engineering. More specifically, it focuses on the use of modelling and simulation to help decision-making in the scope of usability evaluation.ObjectiveThe main goal of this paper is to present UESim: a System Dynamics simulation model to help decision-making in the make-up of the usability evaluation team during the process of usability evaluation.MethodTo develop this research we followed four main research phases: (a) study identification, (b) study development, (c) running and observation and finally, (d) reflexion. In relation with these phases the paper describes the literature revision, the model building and validation, the model simulation and its results and finally the reflexion on it.ResultsWe developed and validated a model to simulate the usability evaluation process. Through three different simulations we analysed the effects of different compositions of the evaluation team on the outcome of the evaluation. The simulation results show the utility of the model in the decision making of the usability evaluation process by changing the number and expertise of evaluators employed.ConclusionOne of the main advantages of using such a simulation model is that it allows developers to observe the evolution of the key indicators of the evaluation process over time. UESim represents a customisable tool to help decision-making in the management of the usability evaluation process, since it makes it possible to analyse how the key process indicators are affected by the main management options of the Usability Evaluation Process.     

Software architecture evaluation methods based on cost benefit analysis and quantitative decision making

Since many parts of the architecture evaluation steps of the Cost Benefit Analysis Method (CBAM) depend on the stakeholders’ empirical knowledge and intuition, it is very important that such an architecture evaluation method be able to faithfully reflect the knowledge of the experts in determining Architectural Strategy (AS). However, because CBAM requires the stakeholders to make a consensus or vote for collecting data for decision making, it is difficult to accurately reflect the stakeholders’ knowledge in the process. In order to overcome this limitation of CBAM, we propose the two new CBAM-based methods for software architecture evaluation, which respectively adopt the Analytic Hierarchy Process (AHP) and the Analytic Network Process (ANP). Since AHP and ANP use pair-wise comparison they are suitable for a cost and benefit analysis technique since its purpose is not to calculate correct values of benefit and cost but to decide AS with highest return on investment. For that, we first define a generic process of CBAM and develop variations from the generic process by applying AHP and ANP to obtain what we call the CBAM+AHP and CBAM+ANP methods. These new methods not only reflect the knowledge of experts more accurately but also reduce misjudgments. A case study comparison of CBAM and the two new methods is conducted using an industry software project. Because the cost benefit analysis process that we present is generic, new cost benefit analysis techniques with capabilities and characteristics different from the three methods we examine here can be derived by adopting various different constituent techniques.