An empirical approach for evaluating the usability of model-driven tools

MDD tools are very useful to draw conceptual models and to automate code generation. Even though this would bring many benefits, wide adoption of MDD tools is not yet a reality. Various research activities are being undertaken to find why and to provide the required solutions. However, insufficient research has been done on a key factor for the acceptance of MDD tools: usability. With the help of end-users, this paper presents a framework to evaluate the usability of MDD tools. The framework will be used as a basis for a family of experiments to get clear insights into the barriers to usability that prevent MDD tools from being widely adopted in industry. To illustrate the applicability of our framework, we instantiated it for performing a usability evaluation of a tool named INTEGRANOVA. Furthermore, we compared the outcome of the study with another usability evaluation technique based on ergonomic criteria.     

A framework to identify primitives that represent usability within Model-Driven Development methods

ContextNowadays, there are sound methods and tools which implement the Model-Driven Development approach (MDD) satisfactorily. However, MDD approaches focus on representing and generating code that represents functionality, behaviour and persistence, putting the interaction, and more specifically the usability, in a second place. If we aim to include usability features in a system developed with a MDD tool, we need to extend manually the generated code.ObjectiveThis paper tackles how to include functional usability features (usability recommendations strongly related to system functionality) in MDD through conceptual primitives.MethodThe approach consists of studying usability guidelines to identify usability properties that can be represented in a conceptual model. Next, these new primitives are the input for a model compiler that generates the code according to the characteristics expressed in them. An empirical study with 66 subjects was conducted to study the effect of including functional usability features regarding end users’ satisfaction and time to complete tasks. Moreover, we have compared the workload of two MDD analysts including usability features by hand in the generated code versus including them through conceptual primitives according to our approach.ResultsResults of the empirical study shows that after including usability features, end users’ satisfaction improves while spent time does not change significantly. This justifies the use of usability features in the software development process. Results of the comparison show that the workload required to adapt the MDD method to support usability features through conceptual primitives is heavy. However, once MDD supports these features, MDD analysts working with primitives are more efficient than MDD analysts implementing these features manually.ConclusionThis approach brings us a step closer to conceptual models where models represent not only functionality, behaviour or persistence, but also usability features.     

A methodology for evaluating the usability of audiovisual consumer electronic products

Usability evaluation is now considered an essential procedure in consumer product development. Many studies have been conducted to develop various techniques and methods of usability evaluation hoping to help the evaluators choose appropriate methods. However, planning and conducting usability evaluation requires considerations of a number of factors surrounding the evaluation process including the product, user, activity, and environmental characteristics. In this perspective, this study suggested a new methodology of usability evaluation through a simple, structured framework. The framework was outlined by three major components: the interface features of a product as design variables, the evaluation context consisting of user, product, activity, and environment as context variables, and the usability measures as dependent variables. Based on this framework, this study established methods to specify the product interface features, to define evaluation context, and to measure usability. The effectiveness of this methodology was demonstrated through case studies in which the usability of audiovisual products was evaluated by using the methods developed in this study. This study is expected to help the usability practitioners in consumer electronics industry in various ways. Most directly, it supports the evaluators' plan and conduct usability evaluation sessions in a systematic and structured manner. In addition, it can be applied to other categories of consumer products (such as appliances, automobiles, communication devices, etc.) with minor modifications as necessary.     

A test-first view of usability

The foundations of usability evaluation were being established in the early 1990s. In this context, “Usability – Context, Definition, Design and Evaluation” built upon Brian Shackel’s earlier influential work – work that helped define the notion of usability. In this paper, he established key dimensions of usability as well as approaches to integrating the testing of these dimensions, within the whole process of setting requirements. Essentially he argued for usability design as part of the system design process.This commentary describes the context of Professor Shackel’s paper and reviews the influential ideas that appear in much subsequent work.     

A method to elicit architecturally sensitive usability requirements: its integration into a software development process

In the Human-Computer Interaction (HCI) community, software usability has primarily been concerned with the presentation of information, more precisely with the user interface. However, some usability problems can prove costly to fix if the changes require modifications that reach beyond the presentation layer, namely those that cannot be easily accommodated by the software architecture. Taking into account some usability requirements earlier in the software development cycle, specifically prior to the architectural design phase, can reduce the cost of these modifications. There is a scarcity of methods and guidelines with the scope to direct users in eliciting the usability requirements that can impact the software architecture. This paper proposes a usability-driven adaptation of the quality attribute workshop (QAW) to assist software development organizations in discovering and documenting usability requirements. It shows how this method can be integrated into a software development process, by discussing how the existing software framework workflows can be adjusted to take this new activity into consideration. A preliminary exercise was conducted to help discern the utility and the limits of the proposed method. Participants with different levels of knowledge of usability and comprehension of the system being developed found the method constructive, as it guided them in identifying the architecturally relevant usability requirements. It also helped determine the usability aspects that would not necessarily have been defined if this technique had not been employed.     

A model-driven and task-oriented method for the development of collaborative systems

Groupware systems are usually difficult to design when following traditional methodologies and approaches for single user systems. In this regard, model-driven approaches have been gaining attention in recent years. In accordance with this paradigm, we developed the SpacEclipse method in a previous work, which is a method for developing collaborative modeling groupware based on the plug-ins in the Eclipse Modeling Project. However, this method presents some deficiencies that we have tried to overcome in this work. In order to achieve this goal, we have chosen the CIAM methodology, which allows the modeling of collaboration, users, tasks, sessions and interactive issues and which is also supported by Eclipse-based tools. In this paper, we explain how the integration of CIAM elements in the SpacEclipse method has been carried out and resulted in a new method with its own methodological, conceptual and technological frameworks. To prove the validity of the method, we have applied it to a re-engineering process in the development of an existing tool.     

Software development methods and usability: Perspectives from a survey in the software industry in Norway

This paper investigates the relationship between software development methodologies and usability. The point of departure is the assumption that two important disciplines in software development, one of software development methods (SDMs) and one of usability work, are not integrated in industrial software projects.

Building on previous research we investigate two questions; (1) Will software companies generally acknowledge the importance of usability, but not prioritise it in industrial projects? and (2) To what degree are software development methods and usability perceived by practitioners as being integrated? To this end a survey in the Norwegian IT industry was conducted. From a sample of 259 companies we received responses from 78 companies.

In response to our first research question, our findings show that although there is a positive bias towards usability, the importance of usability testing is perceived to be much less than that of usability requirements. Given the strong time and cost pressures associated with the software industry, we believe that these results highlight that there is a gap between intention and reality. Regarding our second research question our survey revealed that companies perceive usability and software development methods to be integrated. This is in contrast to earlier research, which, somewhat pessimistically, has argued for the existence of two different cultures, one of software development and one of usability. The findings give hope for the future, in particular because the general use of system development methods are pragmatic and adaptable.     

A comparison of the four prominent user-based methods for evaluating the usability of computer software

The aim was to examine four prominent user-based computer software usability evaluation methods. Four evaluation methods (logged data, questionnaire, inter view, and verbal protocol analysis) were used to evaluate three different business software types (spreadsheet, word processor, and database) using a between-groups design, involving 148 individuals of mixed age and gender. Comparisons were made to examine the efficiency of each evaluation method in terms of its ability to highlight usability problems both between and within the evaluation strategy. Here, the verbal protocol analysis was found to be most efficient. The possibility of further efficiency gains by using two evaluation methods was also examined, where it was found that no statistically significant improvement was obtained over the verbal protocol analysis used by itself. Ways in which the utility of the methods may be enhanced was also discussed.     

Empirical validation of an automatic usability evaluation method

Today, the success of a software application strongly depends on the usability of its interface, so the evaluation of interfaces has become a crucial aspect of software engineering. It is recognized that automatic tools for graphical user interface evaluation may greatly reduce the costs of traditional activities performed during expert evaluation or user testing in order to estimate the success probability of an application. However, automatic methods need to be empirically validated in order to prove their effectiveness with respect to the attributes they are supposed to evaluate.In this work, we empirically validate a usability evaluation method conceived to assess consistency aspects of a GUI with no need to analyze the back-end. We demonstrate the validity of the approach by means of a comparative experimental study, where four web sites and a stand-alone interactive application are analyzed and the results compared to those of a human-based usability evaluation. The analysis of the results and the statistical correlation between the tool׳s rating and humans׳ average ratings show that the proposed methodology can indeed be a useful complement to standard techniques of usability evaluation.     

A usability evaluation toolkit for In-Vehicle Information Systems (IVISs)

Usability must be defined specifically for the context of use of the particular system under investigation. This specific context of use should also be used to guide the definition of specific usability criteria and the selection of appropriate evaluation methods. There are four principles which can guide the selection of evaluation methods, relating to the information required in the evaluation, the stage at which to apply methods, the resources required and the people involved in the evaluation. This paper presents a framework for the evaluation of usability in the context of In-Vehicle Information Systems (IVISs). This framework guides designers through defining usability criteria for an evaluation, selecting appropriate evaluation methods and applying those methods. These stages form an iterative process of design–evaluation–redesign with the overall aim of improving the usability of IVISs and enhancing the driving experience, without compromising the safety of the driver.     

A systematic mapping study of API usability evaluation methods

An Application Programming Interface (API) provides a programmatic interface to a software component that is often offered publicly and may be used by programmers who are not the API’s original designers. APIs play a key role in software reuse. By reusing high quality components and services, developers can increase their productivity and avoid costly defects. The usability of an API is a qualitative characteristic that evaluates how easy it is to use an API. Recent years have seen a considerable increase in research efforts aiming at evaluating the usability of APIs. An API usability evaluation can identify problem areas and provide recommendations for improving the API. In this systematic mapping study, we focus on 47 primary studies to identify the aim and the method of the API usability studies. We investigate which API usability factors are evaluated, at which phases of API development is the usability of API evaluated and what are the current limitations and open issues in API usability evaluation. We believe that the results of this literature review would be useful for both researchers and industry practitioners interested in investigating the usability of API and new API usability evaluation methods.     

Improving a portlet usability model

Second-generation portals are far from being monolithic pieces of software. Their complexity calls for a component-based approach where portlets are the technical enabler. That being the case nowadays portals tend to be constructed by means of portlets, i.e. a multi-step, user-facing application to be delivered through a Web application. The proposal for and ample support given to the WSRP (Web Services for Remote Portlets) portlet standard predict an emerging portlet market. A main requirement for the blossoming of this market is the existence of portlet quality models that assist portal developers to select the appropriate portlet. This paper focuses on usability. The aim, therefore, is to develop a usability model for portlets. The paper presents such a model and its realisation for a sample case.

A framework for evaluating the usability of mobile phones based on multi-level, hierarchical model of usability factors

As a mobile phone has various advanced functionalities or features, usability issues are increasingly challenging. Due to the particular characteristics of a mobile phone, typical usability evaluation methods and heuristics, most of which are relevant to a software system, might not effectively be applied to a mobile phone. Another point to consider is that usability evaluation activities should help designers find usability problems easily and produce better design solutions. To support usability practitioners of the mobile phone industry, we propose a framework for evaluating the usability of a mobile phone, based on a multi-level, hierarchical model of usability factors, in an analytic way. The model was developed on the basis of a set of collected usability problems and our previous study on a conceptual framework for identifying usability impact factors. It has multi-abstraction levels, each of which considers the usability of a mobile phone from a particular perspective. As there are goal-means relationships between adjacent levels, a range of usability issues can be interpreted in a holistic as well as diagnostic way. Another advantage is that it supports two different types of evaluation approaches: task-based and interface-based. To support both evaluation approaches, we developed four sets of checklists, each of which is concerned, respectively, with task-based evaluation and three different interface types: Logical User Interface (LUI), Physical User Interface (PUI) and Graphical User Interface (GUI). The proposed framework specifies an approach to quantifying usability so that several usability aspects are collectively measured to give a single score with the use of the checklists. A small case study was conducted in order to examine the applicability of the framework and to identify the aspects of the framework to be improved. It showed that it could be a useful tool for evaluating the usability of a mobile phone. Based on the case study, we improved the framework in order that usability practitioners can use it more easily and consistently.     

A survey of usability capability maturity models: implications for practice and research

Improving the position and effectiveness of user-centred design (UCD) in software and product development is a challenge in many companies. One step towards improvements is to carry out a usability capability maturity (UCM) assessment to evaluate the strengths and weaknesses of a development organisation in UCD. While several diverse UCM models have been introduced, little research has been published in the public domain on these models. The paper aims to fill this gap by analysing the main features of the models. The results show that most models represent different approaches – although some of them have the same roots – meaning that understanding one model is not enough to understand the basics of another model. All models can be used for examining the status of UCD in individual development projects. In addition, models provide various means for assessment of the status of UCD in other organisational areas. The level of documentation of models varies a lot, and very few empirical research results exist. Based on the results, implications for practice and research are suggested.     

Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis

The use of e-learning technology is incontestably recognized as an important and integral part of the educational process. Considerable research studies are carried out in order to apprehend how effective and usable e-learning systems. In this paper, an empirical-based study is conducted to explore how lecturers interact with an e-learning environment based on a predefined task model describing low-level interactions. Client-side log data is collected from university lecturers from the Electrical and Computer Science departments. Subsequently, data analysis is conducted to infer the usability degree from the estimated usage metrics together with further exploratory analysis from user feedback via System Usability Scale. Experimental results reveal that the System Usability Scale score is not a sufficient measure to express the true acceptance and satisfaction level of lecturers for using the e-learning systems. The evaluation must be fulfilled in tandem with analyzing the usage metrics derived from interaction traces in a non-intrusive fashion. The proposed approach is a milestone towards usability evaluation to improve the acceptance and user experience for academic staff and students.     

Finding a way to usability: procurement of a taxi dispatch system

Despite the extensive work on human–computer interaction regarding methods of involving users and designing for high degrees of usability, there is surprisingly little published on how procurer organizations understand, reason about, and require usability. This study focuses on how one taxi company dealt with usability requirements when procuring a new dispatch system. We have conducted ten interviews with various stakeholders in the company and analyzed related documentation in order to discover the process. The case shows how the concept of usability matured during over time. The taxi company dealt with requirement elicitation by developing prototypes in small reference groups. They did no formal analysis of the operators cooperation with each other at the operator central, but they did include experienced users, which created implicit scenarios. The supplier company did not focus on the efficiency of the operators or, for that matter, the cooperative demands of the operator central in their original design, which became evident when the procurer organization requested a redesign that emphasized user tasks. This indicates, on one hand, the extent to which procurers must understand usability and cooperation to procure good systems design and, on the other hand, the extent to which designers must understand business and activity processes in order to design good systems.     

Adaptive systems: A solution to usability problems

Improving the usability of computer systems is perhaps the most important goal of human-computer interaction research. Current approaches to usability engineering tend to focus on simply improving the interface. An alternative is to build intelligence into the system. However, in order to do this a more comprehensive analysis is required and systems must be designed so that they can be made adaptive. This paper examines the implications for systems analysis, design and usability specification if adaptive systems are to be a realistic solution to usability problems.     

Early traces of usability as a science and as a profession

Shackel’s paper [Shackel, B., 1991. Usability – context, framework, definition, design and evaluation. In: Shcakel, B., Richardson, S. (Eds.), Human Factors for Informatics Usability. Cambridge University Press, Cambridge, UK] is reviewed in an attempt to assess his contribution to the development of usability as a science and as a profession. Usability related research is first situated in the period around 1990. The contributions to usability as a science then are addressed via Professor Shackel’s definition and evaluation of usability. Finally, his contribution toward usability as a profession is acknowledged via his view of usability in the light of wider business goals.     

A proposal toward the development of accessible e-learning content by human involvement

Most of the existing efforts for supporting the design, preparation, and deployment of accessible e-learning applications propose guidelines that primarily address technical accessibility issues. Little, if any, consideration is given to the real actors involved in the learning experience, such as didactical experts and disabled learners. Moreover, implementing artifacts addressed to the e-learning world requires a wide range of particular skills which are related not only to technical but also to didactical, pedagogical, usability, and accessibility aspects of the produced material. This paper argues that the know-how of a number of stakeholders should be blended into a joint design activity, and that it should be possible to determine the role of each participant in the successive phases of the development lifecycle of e-learning applications. The paper sketches the methodological guidelines of a design framework based on involving the users with disabilities, as well as pedagogical experts, in the development process. The novelty of this proposal mainly stems from being built up around the core of strategies and choices specifically bound to accessibility requirements. Characteristic elements of learner-centered design are then further integrated into processes and methodologies which are typical of participatory and contextual design approaches. Following such guidelines, it will be possible to gain a deeper understanding of the requirements and of the operational context of people needing accessible material, either as learners or educators. The underlying objective is to increase the potential to realize learning systems that better meet different user needs and that provide a more satisfying learning experience. Moreover, when people get involved in the development process, they gain a sense of ownership of the system and are therefore more likely to accept and “promote” it.