Prioritizing usability problems: Data-driven and judgement-driven severity estimates

Software-ergonomic system analysis often reveals numerous usability problems. Given that system design suffers from limited resources, the prioritization of usability problems seems inevitable. Surprisingly enough, prioritization is not in the focus of scientific interest. Within this paper, approaches to prioritization relying on severity estimates will be presented. Two of the approaches, namely priorities based on data about the impact of a problem (data-driven) and priorities based on judgements of interest group members (judgement-driven) will be further explored. In the data-driven approach total problem-handling time caused by a usability problem is presented as a measure of severity. The major disadvantage of the data-driven approach is its costs. A possible alternative are severity estimates based on judgements by members of involved interest groups. The first of two studies shows how to obtain judgement driven severity estimates and reveals a fundamental lack of correspondence between data-driven and judgement-driven severity estimates. The second study supports the notion that the lack of correspondence may stem from a difference between assumptions of the data-driven approach and the naive judgement model of interest group members in the judgement-driven approach. A hypothetical model for severity estimates by interest group members is presented.     

Some issues on properties of the extended IOWA operators in fuzzy group decision making

Chiclana et al. [Chiclana, F., Herrera-Viedma, E., Herrera, F., Alonso, S. (2007). European Journal of Operational Research 182, 383–399] provided some IOWA operators for aggregating the individual fuzzy preference elations. The aim of this work is further to study their desired properties under group decision making problem with fuzzy preference relations. First, it is proved that the collective preference relations obtained by these cases of IOWA operators verified the reciprocity and the consistency properties. Next, the aggregation of individual judgements (AIJ) and the aggregation of individual priorities (AIP) provide the same priorities of alternatives by utilizing the IOWA operators as aggregation procedure and the row arithmetic mean method (RAMM) as prioritization procedure. Additionally, the Consistency IOWA (C-IOWA) operator guarantees that the group consistency degree is no less than the arithmetic mean of all the individual consistency degree. Finally, two illustrative numerical examples are used to verify the developed approaches.     

A robust multi-objective approach to balance severity and importance of refactoring opportunities

Refactoring large systems involves several sources of uncertainty related to the severity levels of code smells to be corrected and the importance of the classes in which the smells are located. Both severity and importance of identified refactoring opportunities (e.g. code smells) are difficult to estimate. In fact, due to the dynamic nature of software development, these values cannot be accurately determined in practice, leading to refactoring sequences that lack robustness. In addition, some code fragments can contain severe quality issues but they are not playing an important role in the system. To address this problem, we introduced a multi-objective robust model, based on NSGA-II, for the software refactoring problem that tries to find the best trade-off between three objectives to maximize: quality improvements, severity and importance of refactoring opportunities to be fixed. We evaluated our approach using 8 open source systems and one industrial project, and demonstrated that it is significantly better than state-of-the-art refactoring approaches in terms of robustness in all the experiments based on a variety of real-world scenarios. Our suggested refactoring solutions were found to be comparable in terms of quality to those suggested by existing approaches, better prioritization of refactoring opportunities and to carry an acceptable robustness price.     

Combining different prioritization methods in the analytic hierarchy process synthesis

A multicriteria approach for combining prioritization methods within the analytic hierarchy process (AHP) is proposed. The leading assumption is that for each particular decision problem and related hierarchy, AHP must not necessarily employ only one prioritization method (e.g. eigenvector method). If more available methods are used to identify the best estimates of local priorities for each comparison matrix in the hierarchy, then the estimate of final alternatives’ priorities should also be the best possible, which is in natural concordance with an additive compensatory structure of the AHP synthesis. The most popular methods for deriving priorities from comparison matrices are identified as candidates (alternatives) to participate in AHP synthesis: additive normalization, eigenvector, weighted least-squares, logarithmic least-squares, logarithmic goal programming and fuzzy preference programming. Which method will be used depends on the result of multicriteria evaluation of their priority vectors’ performance with regard to suggested deviation and rank reversal measures. Two hierarchies with matrices of size 3–6 are used to illustrate an approach.     

The quality of expert probability judgement: issues and analysis

Abstract: Many decision-aiding technologies require valid probability judgements to be elicited from domain experts. But how valid are experts' probability judgements? We describe two approaches to the assessment of quality of probability judgement—calibration and coherence—and review the research findings following from these two approaches. In many cases, expert probability judgement has been found to lack validity and this sub-optimality has largely been attributed to computational errors on the part of the expert. The preferred solution to poor validity in probability judgement has therefore been to reduce the amount of computation performed by the expert. Complex probabilities can be calculated mechanically from simple probability judgements elicited from the expert. We present evidence which suggests that this recomposition technique doesn't guarantee valid probabilities. Our explanation for this finding is that there are various problems concerned with eliciting even the simple probabilities which are necessary for subsequent recomposition. We conclude by proposing some solutions to these elicitation problems which should help ensure that probability judgements of increased validity are available to those attempting to capture subjective assessments for input into decision support systems.     

Judgement aggregation and distributed thinking

In recent years, judgement aggregation has emerged as an important area of social choice theory. Judgement aggregation is concerned with aggregating sets of individual judgements over logically connected propositions into a set of collective judgements. It has been shown that even seemingly weak conditions on the aggregation function make it impossible to find functions that produce rational collective judgements from all possible rational individual judgements. This implies that the step from individual judgements to collective judgements requires trade-offs between different desiderata, such as universal domain, rationality, epistemological quality, and unbiasedness. The typical application for judgement aggregation is the problem of group decision making. Juries and expert committees are the stock examples. However, the relevance of judgement aggregation goes beyond these cases. In this survey, I review some core results in the field of judgement aggregation and social epistemology and discuss their implications for the analysis of distributed thinking.     

Problem Prioritization in Usability Evaluation: From Severity Assessments Toward Impact on Design

Severity assessments enable prioritization of problems encountered during usability evaluations, and thereby provide a device for guiding the utilization of design resources. However, designers' response to usability evaluations is also influenced by other factors, which may overshadow severity. With the purpose of enhancing the impact of severity assessments, this study combines a field study of factors that influence the impact of evaluations with an experimental study of severity assessments made during usability inspections. The results show that even in a project receptive to input from evaluations, their impact was highly dependent on conducting evaluations early. This accorded with an informal method that blended elements of usability evaluation and participatory design and could be extended with user-made severity assessments. The major cost associated with the evaluations was not finding but fixing problems, emphasizing that, to be effective, severity assessments must be reliable, valid, and sufficiently persuasive to justify the cost of fixing problems. For the usability inspections, evaluators' ratings of problem impact and persistence were weakly correlated with the number of evaluators reporting a problem, indicating that different evaluators represent different subgroups of users or alternatively that evaluator-made severity assessments are of questionable reliability. To call designers' attention to the severe problems, the halving of the severity sum is proposed as a means of visualizing the large payoff of fixing a high-severity problem and, conversely, the modest potential of spending resources on low-severity problems.     

Comparison of evaluation methods using structured usability problem reports

. Recent HCI research has produced analytic evaluation techniques which claim to predict potential usability problems for an interactive system.Validation of these methods has involved matching predicted problems against usability problems found during empirical user testing. This paper shows that the matching of predicted and actual problems requires careful attention, and that current approaches lack rigour or generality. Requirements for more rigorous and general matching procedures are presented. A solution to one key requirement is presented: a new report structure for usability problems.It is designed to improve the quality of matches made between usability problems found during empirical user testing and problems predicted by analytic methods. The use of this report format is placed within its design research context, an ongoing project on domain-specific methods for software visualizations.     

Synthesis of individual best local priority vectors in AHP-group decision making

An assessment of the individual judgments and AHP-produced priority vectors for involved decision-makers indicates that the individual consistencies of decision makers may vary significantly, thus making the final group decision less reliable. In this paper, an approach is proposed as to how to combine decision makers’ local priority vectors in AHP synthesis and reduce so-called group inconsistency. Instead of aggregating individual judgments (AIJ), or aggregating individually derived final priorities (AIP), we propose to perform an AHP synthesis of the best local priority vectors taken from the most consistent decision makers. The approach and related algorithm we label as MGPS after the key terms ‘multicriteria group prioritization synthesis.’ The concept is analogous to the one proposed by Srdjevic [1] for individual AHP applications where the best local priority vectors are selected based on the consistency performance of several of the most popular prioritization methods. Here, decision makers are combined instead of prioritization methods, and group context is fully implemented. After completing an evaluation of the decision makers inconsistencies in each node of the hierarchy, the selected best local priority vectors are synthesized in a standard manner, and the final solution is declared to be an AHP-group decision. Two numerical examples indicate that the developed approach and algorithm generate the final priorities of alternatives with the lowest overall inconsistency (in the multicriteria sense).     

Usability guideline for banking software design

As the diversity of services in the financial market increases, it is critical to design usable banking software in order to overcome the complex structure of the system. The current study presents a usability guideline based on heuristics and their corresponding criteria that could be used during the early stages of banking software design process. In the design of a usability guideline, the heuristics and their criteria are categorized in terms of their effectiveness in solving usability problems grouped and ranging from usability catastrophe to cosmetic problems. The current study comprises of three main steps: First, actual usability problems from three banking software development projects are categorized according to their severity level. Secondly, usability criteria are rated for how well they explain the usability problems encountered. Finally, usability heuristics are categorized according to the severity level of usability problems through two analytical models; corresponding and cluster analyses. As the result, designers and project managers may give more importance to the heuristics related with the following usability problem categories: Usability catastrophe and then major usability problems. Furthermore, the proposed guideline can be used to understand which usability criteria would be helpful in explaining usability problems as well as preventing banking system catastrophes, by highlighting the critical parts in system design of banking software.     

Priorities on defaults with prerequisites,and their application in treating specificity in terminological default logic

In a recent paper we have proposed terminological default logic as a formalism that combines means both for structured representation of classes and objects and for default inheritance of properties. The major drawback that terminological default logic inherits from general default logic is that it does not take precedence of more specific defaults over more general ones into account. This behavior has already been criticized in the general context of default logic, but it is all the more problematic in the terminological case where the emphasis lies on the hierarchical organization of concepts.The present paper addresses the problem of modifying terminological default logic such that more specific defaults are preferred. We assume that the specificity ordering is induced by the hierarchical organization of concepts, which means that default information is not taken into account when computing priorities. It turns out that the existing approaches for expressing priorities between defaults do not seem to be appropriate for defaults with prerequisites. Therefore we shall consider an alternative approach for dealing with prioritization in the framework of Reiter's default logic. The formalism is presented in the general setting of default logic where priorities are given by an arbitrary partial ordering on the defaults. We shall exhibit some interesting properties of the new formalism, compare it with existing approaches, and describe an algorithm for computing extensions. In the terminological case, we thus obtain an automated default reasoning procedure that takes specificity into account.This is an extended version of a paper presented at the13th International Joint Conference on Artificial Intelligence, August 1993, Chambery, France.     

数据驱动的保证收敛速率最优输出调节

针对具有外部系统扰动的线性离散时间系统的输出调节问题,提出了可保证收敛速率的数据驱动最优输出调节方法,包括状态可在线测量系统的基于状态反馈的算法,与状态不可在线测量系统的基于输出反馈的算法.首先,该问题被分解为输出调节方程求解问题与反馈控制律设计问题,基于输出调节方程的解,通过引入收敛速率参数,建立了可保证收敛速率的最...     

The usability analysis with heuristic evaluation and analytic hierarchy process

Today people increasingly expect more from the functionality of a web site, so the web usability has emerged as an important topic. It is generally hard to renew the existing web sites to meet the changing demands of users. Therefore, the present study is aimed at detailing the usability problems of web sites. To this end, Heuristic Evaluation (HE) is used to identify the usability problems, and Analytic Hierarchy Process (AHP) is used to rate their severity. Finally, for a more user-friendly web site, a new approach to judging severity of usability problems is developed by integrating AHP into HE.

Relevance to industry

There is an increasing importance for higher usability in the web development industry and communities. Different usability evaluation techniques have been developed and incorporated into the process of web site design and development. This study proposes a new approach to reveal usability problems on a web site and to define solution priority of these problems.     

Consistency control for semistructured data

Semistructured data models provide an important tool for studying methods for representing, searching, and processing heterogeneous data in distributed systems with autonomous components (especially, on the WWW). However, the problem of data consistency remains little investigated. In this paper, various approaches to this problem are discussed, and the usability of various mechanisms suggested in the context of databases, including real-time databases, is analyzed. An approach based on the concept of multilevel transactions is presented in detail. In the framework of this approach, commutativity predicates for basic operations on semistructured data are constructed.     

A model-based framework for classifying and diagnosing usability problems

A great deal of study has been devoted to the problem of how to identify and categorize usability problems; however, there is still a lack of studies dealing with the problem of how to diagnose the causes of usability problems and how to feed them back into design process. The value of classifying usability problems can be enhanced when they are interpreted in connection with design process and activities. Thus, it is necessary to develop a systematic way of diagnosing usability problems in terms of design aspects and applying diagnosis results to improve design process and activities. With this issue in mind, this paper proposes a conceptual framework that supports a systematic classification and diagnosis of usability problems. This paper firstly reviews seven approaches to classifying usability problems. Then, we point out the needs of adopting a model-based approach to classifying and diagnosing usability problems and of developing a comprehensive framework guiding the use of model-based approaches. We then propose a conceptual framework that specifies how a model-based classification and diagnosis of usability problems should be conducted and suggests the combined use of three different types of models, each of which addresses context of use, design knowledge and design activities. Last, we explain how a sound classification scheme of usability problems can be systematically developed, and how the classification of usability problems can be connected to design process and activities on the basis of the framework.     

Aesthetics and preferences of web pages

The first impressions of web pages presented to users was investigated by using 13 different web pages, three types of scales and 18 participants. Multidimensional analysis of similarity and preference judgements found four important dimensions: beauty, mostly illustrations versus mostly text, overview and structure. Category scales indicated the existence of two factors related to formal aspects and to appeal of the objects, respectively. The best predictor for the overall judgement of the category scales was beauty. Property vector fitting of the multidimensional solutions with the category scales further indicated the importance of beauty for the preference space. Aspects of usability, product design and aesthetics are discussed.     

A usability study of iPhone built-in applications

The intuitiveness of gestural devices offers great potential for more physical ways of interacting, but it is imperative that such devices comply with existing, platform-independent usability principles that have proven their effect in heterogeneous settings over the last decades. This study focuses on iPhone, one of the most widely used haptic devices and reviews the usability problems of five built-in iPhone applications documented in related sources. The study investigates whether these problems are actually reported during testing with novice users, which is their severity and whether testing reveals additional issues. Overall, the study confirmed the list of usability problems documented and stressed the importance of conforming to established usability principles.     

WaveLines: towards effective visualization and analysis of stability in power grid simulation

Closely related to the safety and stability of power grids, stability analysis has long been a core topic in the electric industry. Conventional approaches employ computational simulation to make the quantitative judgement of the grid stability under distinctive conditions. The lack of in-depth data analysis tools has led to the difficulty in analytical tasks such as situation-aware analysis, instability reasoning and pattern recognition. To facilitate visual exploration and reasoning on the simulation data, we introduce WaveLines, a visual analysis approach which supports the supervisory control of multivariate simulation time series of power grids. We design and implement an interactive system that supports a set of analytical tasks proposed by domain experts and experienced operators. Experiments have been conducted with domain experts to illustrate the usability and effectiveness of WaveLines.     

A statistical approach to knowledge acquisition for expert systems

Classical expert systems are rule based, depending on predicates expressed over attributes and their values. In the process of building expert systems, the attributes and constants used to interpret their values need to be specified. Standard techniques for doing this are drawn from psychology, for instance, interviewing and protocol analysis. This paper describes a statistical approach to deriving interpreting constants for given attributes. It is also possible to suggest the need for attributes beyond those given.The approach for selecting an interpreting constant is demonstrated by an example. The data to be fitted are first generated by selecting a representative collection of instances of the narrow decision addressed by a rule, then making a judgement for each instance, and defining an initial set of potentially explanatory attributes. A decision rule graph plots the judgements made against pairs of attributes. It reveals rules and key instances directly. It also shows when no rule is possible, thus suggesting the need for additional attributes. A study of a collection of seven rule based models shows that the attributes defined during the fitting process improved the fit of the final models to the judgements by twenty percent over models built with only the initial attributes.