Evaluation of user interfaces: EVADIS II— a comprehensive evaluation approach

Abstract

As a result of the importance of the usability approach in system development and the EC's ‘Directive concerning the minimum safety and health requirements for VDT workers’ (EWG 1990), there is an accepted need for practical evaluation methods for user interfaces. The usability approach and the EC Directive are not restricted to user interface design, as they include the design of appropriate hardware and software, as well as organization, job, and task design. Therefore system designers are faced with many, often conflicting, requirements and need to address the question, ‘How can usability requirements comprehensively be considered and evaluated in system development?’ Customers buying hardware and software and introducing them into their organization ask, (How can I select easy-to-use hardware and software?’ Both designers and customers need an evaluation procedure that covers all the organizational, user, hard- and software requirements. The evaluation method, EVADIS.II, we present in this paper overcomes characteristic deficiencies of previous evaluation methods. In particular, it takes the tasks, the user, and the organizational context into consideration during the evaluation process, and provides computer support for the use of the evaluation procedure.     

Cooking personas: Goal-directed design requirements in the kitchen

Technology to assist people in the kitchen has become a major research topic as technology becomes more ubiquitous in the home. Research is being carried out in numerous areas to assist with cooking, from solving difficulties in the kitchen to more recently enhancing already good experiences related to cooking. To design solutions that are not only usable, but useful in all related aspects of cooking, we must get a good understanding of the needs of the user. It is important that real needs are identified, so that products or systems designed are adopted and are sustainable. There are various methods that researchers and designers use to gain user insights and there is much debate on different approaches and their effectiveness. Whilst there have been a number of ethnographic style studies in people's homes, there has been little in the way of understanding user goals in the kitchen which could lead to more effective design solutions. In this paper, we present goal-directed research of cooking needs in Singapore. We produce three primary design personas and three secondary personas, describing their goals and needs and where they can be helped through technology. An online survey was carried out to validate our personas, comparing our qualitative and quantitative findings. We provide a general holistic overview of kitchen requirements for these personas, whether in helping correct problems or enhancing positive experiences and how these needs can be connected. The user requirements provided can help guide researchers and help designers produce more meaningful, complete and acceptable technology solutions in the kitchen.     

Systems scenarios: a tool for facilitating the socio-technical design of work systems

Abstract

The socio-technical systems approach to design is well documented. Recognising the benefits of this approach, organisations are increasingly trying to work with systems, rather than their component parts. However, few tools attempt to analyse the complexity inherent in such systems, in ways that generate useful, practical outputs. In this paper, we outline the ‘System Scenarios Tool’ (SST), which is a novel, applied methodology that can be used by designers, end-users, consultants or researchers to help design or re-design work systems. The paper introduces the SST using examples of its application, and describes the potential benefits of its use, before reflecting on its limitations. Finally, we discuss potential opportunities for the tool, and describe sets of circumstances in which it might be used.

Practitioner Summary: The paper presents a novel, applied methodological tool, named the ‘Systems Scenarios Tool’. We believe this tool can be used as a point of reference by designers, end-users, consultants or researchers, to help design or re-design work systems. Included in the paper are two worked examples, demonstrating the tool’s application.     

A Qualitative Design Approach for Exploring the Use of Medication and Health Care Devices among Elderly Persons

An academia–industry collaborative research project regarding design for elderly persons was initiated to investigate their living needs and potential design opportunities for new technologies and products. In the first year, a qualitative design approach for exploring their use of medication and health care devices was proposed. First, a user study was conducted using self‐reporting, observation, and interview methods. Four personas representing different lifestyle patterns of elderly persons were then derived from the user study data. Finally, four designers were invited to present and synthesize their design ideas for those personas. Results showed that the user study could reveal considerable information and that the persona method was effective for designers to communicate their ideas and concentrate on user requirements. The study findings suggested that, for design for elderly persons, social and affective factors can be considered with decline in age‐related abilities. © 2012 Wiley Periodicals, Inc.     

A methodological approach for user interface development of collaborative applications: A case study

In the last few years, the production of systems which support learning and group work has been high. However, the design and development of these types of systems are difficult, mainly due to the multidisciplinarity involved. Furthermore, the Graphic User Interface (GUI) of an application is receiving greater attention, since it can be decisive in determining if the application is accepted or rejected by users. Model-based design is a widespread technique in the user interface development process. While reviewing approaches that deal with the modeling and design of user interfaces supporting collaborative tasks, we have detected that there is no proposal that links interactive and collaborative issues. We have introduced a methodological approach to solve this shortcoming. This approach is called CIAM (Collaborative Interactive Application Methodology) and it is composed of several stages in which conceptual models are created using CIAN (Collaborative Interactive Application Notation). These models start by modeling the organization in which the application will be used, as well as the tasks that must be supported. In the initial stages, the organization and the collaborative tasks are modeled using high-level specifications. In the following stages, the level of detail increases and, finally, the interaction between the individual users and the application is modeled using ConcurTaskTrees (CTT) notation. The interaction model acts as a bridge between the design and the implementation of the Graphic User Interface. In this paper we present our methodological approach and an example of applying this method for user interface design of collaborative and interactive applications.     

The design and realization of CoViD: a system for collaborative virtual 3D design

Many important decisions in the design process are made during fairly early on, after designers have presented initial concepts. In many domains, these concepts are already realized as 3D digital models. Then, in a meeting, the stakeholders for the project get together and evaluate these potential solutions. Frequently, the participants in this meeting want to interactively modify the proposed 3D designs to explore the design space better. Today’s systems and tools do not support this, as computer systems typically support only a single user and computer-aided design tools require significant training. This paper presents the design of a new system to facilitate a collaborative 3D design process. First, we discuss a set of guidelines which have been introduced by others and that are relevant to collaborative 3D design systems. Then, we introduce the new system, which consists of two main parts. The first part is an easy-to-use conceptual 3D design tool that can be used productively even by naive users. The tool provides novel interaction techniques that support important properties of conceptual design. The user interface is non-obtrusive, easy-to-learn, and supports rapid creation and modification of 3D models. The second part is a novel infrastructure for collaborative work, which offers an interactive table and several large interactive displays in a semi-immersive setup. It is designed to support multiple users working together. This infrastructure also includes novel pointing devices that work both as a stylus and a remote pointing device. The combination of the (modified) design tool with the collaborative infrastructure forms a new platform for collaborative virtual 3D design. Then, we present an evaluation of the system against the guidelines for collaborative 3D design. Finally, we present results of a preliminary user study, which asked naive users to collaborate in a 3D design task on the new system.     

Component-based tailorability: Enabling highly flexible software applications

Component technologies are perceived as an important means to keep software architectures flexible. Flexibility offered by component technologies typically addresses software developers at design time. However, the design of software which should support social systems, such as work groups or communities, also demands ‘use-time’, or technically spoken, ‘run-time’ flexibility. In this paper, we summarize a decade of research efforts on component-based approaches to flexibilize groupware applications at run-time. We address the user as a ‘casual programmer’ who develops and individualizes software for his work context. To deal with the challenges of run-time flexibility, we developed a design approach which covers three levels: software architecture, user interface, and collaboration support. With regard to the software architecture, a component model, called FlexiBeans, has been developed. The FreEvolve platform serves as an environment in which component-based applications can be tailored at run-time. Additionally, we have developed three different types of graphical user interfaces, enabling users to tailor their applications by recomposing components. To enable collaborative tailoring activities, we have integrated functions that allow sharing component structures among users. We also present different types of support techniques which are integrated into the user interface in order to enable users’ individual and collaborative tailoring activities. We conclude by elaborating on the notion of ‘software infrastructure’ which offers a holistic approach to support design activities of professional and non-professional programmers.     

Reflections on interdisciplinary team development for the design of a platform for computer supported collaborative play

The Constructed Narratives project has been designed for use in public spaces where there is the opportunity for individuals and groups of people, who are not acquainted with each other, to encounter the game and subsequently each other. The goal is to provide a platform that supports discourse in environments where ‘keeping comfortable distance’ between oneself and others is the norm. The system framework developed for this project can be applied for use in computer supported collaborative learning (CSCL), and collaborative design activities in the tradition of computer supported collaborative work (CSCW). The current domain explored in the Constructed Narratives project is computer systems designed to enable shared experience through play, or computer supported collaborative play (CSCP). This paper examines the learner-centred design methodologies used for the development of the physical artifacts and underlying software and hardware system architecture for the Constructed Narratives project. The design methodology demonstrates how an interdisciplinary team of artists, designers, and technologists can exploit the opportunities inherent in this ‘symmetry of ignorance’ to find solutions for multiple wicked design problems that can arise during the development of an integrated software and hardware system.     

A Review of:“Fundamentals of Psychology”. By FRANK A. GELDARD. ( New York,John Wiley and Sons. ) [Pp.437.] 57s.

Abstract

Industrial Design is a discipline which, despite an early pre-occupation with styling and making technology saleable, is now centrally concerned with user needs. This is reflected in Coventry's unique degree course in the Industrial Design/Transport Department at Lanchester Polytechnic. The course contains a significant commitment to Human Factors and this is evident in the range of design projects which are undertaken either by students or as sponsored research. The emphasis in these projects is on practical designs which meet user requirements.

This paper describes the following typical course projects:

1. The competition winning design for a dinghy which can be used as an emergency life-raft.

2. An adaptation of a standard front-wheel-drive car with a transverse slide seat to allow easy access for a wheel chair user.

3. The design for a bus seat which can be used either as a conventional seat or as a standee ‘lean’ seat.

4. Design of a vehicle for use by spina bifida children.

5. Design for a pedal driven power unit for transport and machine uses in Developing Countries.

6. The design of adult tricycles for the transport of mothers plus children under 5 yr.

It is concluded that greater collaboration between industrial designers and ergonomists would be possible if the latter published their data in a more accessible form, and that designers must place greater emphasis on the product evaluation aspect of their work.     

Design process excludes users: the co-creation activities between user and designer

User needs are a fundamental element of design. If the design process does not properly reflect user needs, the design will be severely compromised. Therefore, it is worthwhile to investigate how the user is, and user needs are, understood in the design process. In this article, three accepted linear process models for web site and interactive media design are reviewed in terms of the designer and user participation. The article then proposes a user-evolving collaborative design process which is built on co-creation activities between designer and user. Co-creation activities across the entire design process structurally and ontologically reposition the users, and user needs, centrally, which allows the designers to holistically approach to the user needs through building a partnership with the users. Co-creation creates an equal evolving participatory process between user and designer towards sharing values and knowledge and creating new domains of collective creativity.     

An evaluation methodology for crowdsourced design

In recent years, the “power of the crowd” has been repeatedly demonstrated and various Internet platforms have been used to support applications of collaborative intelligence in tasks ranging from open innovation to image analysis. However, crowdsourcing applications in the fields of design research and creative innovation have been much slower to emerge. So, although there have been reports of systems and researchers using Internet crowdsourcing to carry out generative design, there are still many gaps in knowledge about the capability and limitations of the technology. Indeed the process models developed to support traditional commercial design (e.g. Pugh’s Total Design, Agile, Double-Diamond etc.) have yet to be established for Crowdsourced Design (cDesign). As a contribution to the development of such a general model this paper proposes a cDesign framework to support the creation of crowdsourced design activities. Within the cDesign framework the effective evaluation of design quality is identified as a key component that not only enables the leveraging of a large, virtual workforce’s creative activities but is also fundamental to almost all iterative optimisation processes. This paper reports an experimental investigation into two different Crowdsourced design evaluation approaches; free evaluation and ‘Crowdsourced Design Evaluation Criteria’ (cDEC). The results are benchmarked against a ‘manual’ evaluation carried out by a panel of experienced designers. The results suggest that the cDEC approach produces design rankings that correlate strongly with the judgements of an “expert panel”. The paper concludes that cDEC assessment methodology demonstrates how Crowdsourcing can be effectively used to evaluate, as well as generate, new design solutions.     

Affective judgment in creative design: A method of fitness evaluation to the problem/solution spaces

A method of fitness evaluation to the problem/solution spaces is proposed based on the affective judgment in creative design. Six experienced designers and one-hundred-one users participated in a set of exploratory experiments on affective judgement of automobile design. From the experimental data, we identify the differences in designer's affective appeal in a design and user's affective response to the design. From data analysis, we confirm the general validity of that the ‘creative’ aspect of design can be described by introducing the notions of ‘default’ and ‘surprise’ problem/solution spaces. The affective judgments based on the expectations of designers and users are shown to be an effective alternative to quantitative metrics in design evaluation and concept selection. The presented findings suggested that the success and creativity are not the same, the goal of design practice is to find solutions that are both creative and successful.     

User-centred design: for users or by users?

Abstract

When ergonomists contribute to the design of products and services they aim to be user-centred. This paper explores two possible meanings of user-centredness; the ergonomist may use theories and findings about human behaviour to act for the user or may help the user to participate in design. Both approaches are well known in ergonomics and they can point in conflicting directions. This paper examines the rationale for the two approaches and presents the results of a survey, which found that the most successful strategy was to mix the two approaches. It offers a classification to support the identification of the appropriate approach to adopt in different situations. The paper proposes, for example, that ‘design by users’ is the appropriate strategy when significant value judgements have to be taken in a local or bespoke design setting. By contrast a ‘design for users’ approach is appropriate for the design of generic products. An additional approach ‘design for users with users’ is introduced for settings that require knowledge about human characteristics and that need users to make value judgements.     

Introducing ergonomics requirements in the eco-design of energy-related products from users’ behaviour approach

Ergonomics has been a very important activity in the design process. However, ergonomics rarely includes the environmental requirements into the design of products. The article proposes and presents the Eco-Ergo model through its application to a real-world product, a washing machine, to allow designers and ergonomists to establish product design requirements in order to minimise environmental impacts related to user–product interaction during the use stage. This model uses a visual language of representation, Blueprinting-based, that helps designers explore problems they have not previously considered during the market research when a wide variety of products with different interaction elements is analysed. The application of this model allows direct efforts and attention on the user analysis phase in the most influential user’s actions on the environmental performance of energy-related products during use, establishing ergonomics requirements related to users behaviour at the initial design phase.

Practitioner summary: This study provides a proposal to incorporate ergonomics into the practice of eco-design through the use of human factors in the establishment of initial eco-design requirements. This blueprint-based model combines an empirical and theoretical approach, based on the product test developed by designers, ergonomists and environmentalists.

Abbreviations: CO2: carbon dioxide; DBIM: design behaviour intervention model; DfSB: design for sustainable behaviour; DwI Method: design with intent method; LCA: life cycle assessment; LCD: liquid cristal display; MJ: megajoule; MTM: methods-time measurement; PSS: product-service system; RPM: revolutions per minute     

Cognitive processes in engineering design: a longitudinal study

Abstract

In this paper we report an empirical study of the cognitive processes of semi-expert electronics engineers pursuing real-world design projects. Extensive diary and interview data were analysed so as to reveal the organization of on-going design activity and the goals that were motivating behaviour. Our analyses indicated that subjects were implementing a highly systematic design strategy which deviated only a small amount (12%) from a top-down, depth-first procedure. Some of this deviation could be accounted for by social demands impinging on the individual designer, and whilst there was also evidence of ‘opportunistic’ deviation from a structured design approach, this did not appear to be a significant feature of our subjects' behaviour. We present a model of the design process as a set of production rules which describe an abstract ‘design schema’ for electronic engineering. This design schema embodies processes which control and co-ordinate problem-understanding and problem-structuring activity as well as the essentially top-down, depth-first pursuit of design solutions. The schema also allows for flexibility in the design process, permitting subjects to cope with contingencies arising through social influences and performance breakdowns. A further aspect of the design schema is that it encapsulates a ‘satisficing’ procedure which reflects a principle that dominated our engineers' solution-search and evaluation activity—they characteristically focused exclusively on initial, satisfactory (but usually sub-optimal) solution concepts rather than comparing alternatives with the aim of optimizing choices. We conclude our discussion by recommending facilities that we feel computer-based design aids should offer users if they are to be genuinely useful adjuncts to design activity.     

Bridging the Information Transfer Gap: Measuring Goodness of Information Fit

The ability to determine, a priori, how well an interface design meets the information needs of a user—that is, to ‘bridge the information transfer gap’—is a critical component of any intelligent interface design system. Yet most current approaches either sidestep the problem of computationally measuring ‘goodness of fit’ by compiling design knowledge into a set of pattern matching rules or, at best, use a very limited set of parameters in scoring. We describe a formal representation for capturing the information needs associated with user tasks and the information conveying capabilities of interface elements. A multifaceted scoring technique, using this representation, is presented for assessing a design's goodness of fit against current user requirements. The hallmarks of this approach are the explicit representation of the goals of good user interface design, evaluation of a proposed design against these goals and a computational approach to performing tradeoffs among these goals when incompatible. Finally, we report on the implementation and user acceptance of this design in an adaptive interface application for military attack/scout helicopters.     

A ‘speculative pasts’ pedagogy: where speculative design meets historical thinking

ABSTRACT

This paper provides a pedagogic context for the authors’ concept of ‘Speculative Pasts,’ framed within an auto-ethnographic account of their co-taught course ‘How the Computer Became Personal.’ Blending disciplinary methodologies from historical practice and speculative and critical design, the Speculative Pasts assignment asks students to create primary documents from a hypothetical historical scenario related to an aspect of American personal computing history. This paper lays out the disciplinary contexts and development process for working across design and history disciplines, curricular organization, assignment process, and offers analysis of examples from student work. Additionally, this paper details how ‘Speculative Pasts’ offer a critique of the narrowness and problematic futurism of ‘speculative futures.’ Altogether, the authors offer this course and its primary project as a model for making history essential, rather than supplementary, to design and for leveraging practice-based production as a valued mode of historical inquiry.     

The influence of a knowledge-based system on designers' cognitive activities: a study involving professional web designers

Over the last few decades, several usability knowledge-based systems have been developed to provide user interface designers with usability knowledge (e.g. heuristics, usability guidelines, standards). Such systems are intended to assist designers during the design process, and to improve the usability of the user interface being designed. However, the assumption that such systems actually improve the usability of the resulting user interface remains to be demonstrated: virtually no systems have been empirically tested by designers who create products. In order to confirm this assumption, we conducted an experimental study in which professional web designers had to create webpages, either using a knowledge-based system – MetroWeb – or without it. This study was intended to determine the influence of MetroWeb on the professional web designers' cognitive activity, and to find out whether MetroWeb actually assists them to develop a user-centred design. The results show that the web designers did not very often use MetroWeb while designing webpages; however, rather surprisingly, the designers who did use MetroWeb more often exhibited a user-centred activity than those working without MetroWeb. We conclude this paper by discussing these findings, and suggesting future possible ways of research intended to assist designers to adopt a user-centred approach to their activity.     

WISDOM: A website design method based on reusing design and software solutions

ContextWebsites are increasingly important for advertising and announcing information and they have become virtual business operations support tools. Thus, website designers have to take an increasing number of criteria (cost, delay, quality, security, maintenance) into account during design process to satisfy the needs.ObjectiveThe objective of this paper was to present our WISDOM method that: guides the designer through the website design process, proposes design solutions based on already existing solutions for online website design, facilitates the choice of software components to implement specific services, and speeds up website construction.MethodThe originality of our method is that it links the design process to formalized experience and a software component characterization that allows both functional and non-functional aspects to be considered.ResultsThis method relies on the state-of-the-art strengths in the website design process, modeling dimensions, Model-Driven Engineering and the patterns approach. We propose an implementation of our method as a dedicated website which helps website design and provides a website analysis catalog and a software component analysis catalog.ConclusionOur analysis of the method’s use highlights that formalizing the steps of the design process helps designers, especially novice designers, to design a website; our analysis of the tool’s use highlights its efficiency for rapid website development and its use of the “website family” concept. The results are so very encouraging for both method and tool; both facilitate website design by reusing existing solutions and components.     

Who are the users? Who are the developers? Webs of users and developers in the development process of a technical standard

The paper presents an empirical study of user involvement in developing a technical standard for a scientific community's information system project. The case illustrates how multiple perspectives are involved when considering the user role in practice. The case presents a situation where both developers and users were pre‐defined in the design and development phases of the standard as homogeneous groups of actors. Groups of actors changed to become more heterogeneous and ‘fluid’ in the deployment and implementation phases, thus forming ‘webs of developers’ and ‘webs of users’. Detailed analysis of the process in its entirety shows the blurredness of boundaries between ‘developer’ and ‘user’ categories and roles, and reveals challenges at social and organizational levels. Three models pertaining to the system development process are presented in order to illuminate differing perspectives on the user and on the development process itself. The paper draws theoretically from information systems, social informatics, and science and technology studies. The research contributes to a deeper, interdisciplinary understanding of ‘the’ user, of multiple roles in systems development, and of dynamic sets of user–developer relations.