Quantitative analysis in a heuristic evaluation experiment

In usability context of interactive systems the heuristic evaluation method is widespread. In most applications the results tend to be qualitative, describing such aspects that require some improvement for the benefit of usability. However, these qualitative results do not allow us to determine how usable it is or how it becomes an interactive system. Hence the need for quantitative results may also be very necessary in order to determine the effort that would be needed to get a sufficiently usable system.This article describes, following the idea of the UsabAIPO Project, a new experiment to obtain quantitative results after a heuristic evaluation. This new experimentation has required some variation on the original idea, working with a set of different heuristic categories, while considering the use of the score depending on severity and frequency parameters.     

The usability analysis with heuristic evaluation and analytic hierarchy process

Today people increasingly expect more from the functionality of a web site, so the web usability has emerged as an important topic. It is generally hard to renew the existing web sites to meet the changing demands of users. Therefore, the present study is aimed at detailing the usability problems of web sites. To this end, Heuristic Evaluation (HE) is used to identify the usability problems, and Analytic Hierarchy Process (AHP) is used to rate their severity. Finally, for a more user-friendly web site, a new approach to judging severity of usability problems is developed by integrating AHP into HE.

Relevance to industry

There is an increasing importance for higher usability in the web development industry and communities. Different usability evaluation techniques have been developed and incorporated into the process of web site design and development. This study proposes a new approach to reveal usability problems on a web site and to define solution priority of these problems.     

A survey of usability capability maturity models: implications for practice and research

Improving the position and effectiveness of user-centred design (UCD) in software and product development is a challenge in many companies. One step towards improvements is to carry out a usability capability maturity (UCM) assessment to evaluate the strengths and weaknesses of a development organisation in UCD. While several diverse UCM models have been introduced, little research has been published in the public domain on these models. The paper aims to fill this gap by analysing the main features of the models. The results show that most models represent different approaches – although some of them have the same roots – meaning that understanding one model is not enough to understand the basics of another model. All models can be used for examining the status of UCD in individual development projects. In addition, models provide various means for assessment of the status of UCD in other organisational areas. The level of documentation of models varies a lot, and very few empirical research results exist. Based on the results, implications for practice and research are suggested.     

Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis

The use of e-learning technology is incontestably recognized as an important and integral part of the educational process. Considerable research studies are carried out in order to apprehend how effective and usable e-learning systems. In this paper, an empirical-based study is conducted to explore how lecturers interact with an e-learning environment based on a predefined task model describing low-level interactions. Client-side log data is collected from university lecturers from the Electrical and Computer Science departments. Subsequently, data analysis is conducted to infer the usability degree from the estimated usage metrics together with further exploratory analysis from user feedback via System Usability Scale. Experimental results reveal that the System Usability Scale score is not a sufficient measure to express the true acceptance and satisfaction level of lecturers for using the e-learning systems. The evaluation must be fulfilled in tandem with analyzing the usage metrics derived from interaction traces in a non-intrusive fashion. The proposed approach is a milestone towards usability evaluation to improve the acceptance and user experience for academic staff and students.     

A methodology for evaluating the usability of audiovisual consumer electronic products

Usability evaluation is now considered an essential procedure in consumer product development. Many studies have been conducted to develop various techniques and methods of usability evaluation hoping to help the evaluators choose appropriate methods. However, planning and conducting usability evaluation requires considerations of a number of factors surrounding the evaluation process including the product, user, activity, and environmental characteristics. In this perspective, this study suggested a new methodology of usability evaluation through a simple, structured framework. The framework was outlined by three major components: the interface features of a product as design variables, the evaluation context consisting of user, product, activity, and environment as context variables, and the usability measures as dependent variables. Based on this framework, this study established methods to specify the product interface features, to define evaluation context, and to measure usability. The effectiveness of this methodology was demonstrated through case studies in which the usability of audiovisual products was evaluated by using the methods developed in this study. This study is expected to help the usability practitioners in consumer electronics industry in various ways. Most directly, it supports the evaluators' plan and conduct usability evaluation sessions in a systematic and structured manner. In addition, it can be applied to other categories of consumer products (such as appliances, automobiles, communication devices, etc.) with minor modifications as necessary.     

A methodology for designing information security feedback based on User Interface Patterns

A methodology is provided here to assist in the design of secure interactive applications. In particular, this methodology helps design an adequate security information feedback based on User Interface Patterns, the resulting feedback is then evaluated against a set of design/evaluation criteria called Human–Computer Interaction for Security (HCI-S). In case of a security issue the security information feedback is generally presented using the visual and auditory channels required to achieve an effective notifications, and it is explicitly specified in the design of user interfaces for secure web system.     

新冠疫情相关社交媒体谣言传播量化分析

新冠肺炎疫情的爆发伴随着大量的谣言在社交媒体平台传播,对网络秩序和社会稳定产生了不良影响.已有的疫情相关社交媒体谣言传播量化分析研究仅对谣言内容等单一传播要素展开分析,而忽略了构成信息传播的其他基础要素,包括传播者、受众以及传播效果等.同时,这些研究的谣言数据与真实的社交媒体谣言数据也存在分布偏差和信息缺失.因此,基于新浪微博平台对新冠疫情相关社交媒体谣言的传播展开更加全面的量化分析.具体而言,首先对谣言传播内容进行分析,包括其主题分析、涉及地区分析、事件倾向性分析以及情感分析;进一步对谣言参与用户进行分析,将参与用户分为3类：造谣者、传谣者和辟谣者,并分别对其基础属性、关注主题、个体情绪以及自网络属性进行探究;最后对谣言引发舆情进行分析,探究其情感的整体分布、与主题、关键词和地区的关系、以及情感的演变规律.该研究首次从信息传播的各个基础要素层面对疫情相关的社交媒体谣言传播展开量化分析,不仅对新冠肺炎疫情相关谣言传播有了更全面深刻的认识,同时对突发公共事件的谣言研究和谣言治理也具有十分重要的价值.     

Too good to be bad: Favorable product expectations boost subjective usability ratings

In an experiment conducted to study the effects of product expectations on subjective usability ratings, participants (N = 36) read a positive or a negative product review for a novel mobile device before a usability test, while the control group read nothing. In the test, half of the users performed easy tasks, and the other half hard ones, with the device. A standard usability test procedure was utilized in which objective task performance measurements as well as subjective post-task and post-experiment usability questionnaires were deployed. The study revealed a surprisingly strong effect of positive expectations on subjective post-experiment ratings: the participants who had read the positive review gave the device significantly better post-experiment ratings than did the negative-prime and no-prime groups. This boosting effect of the positive prime held even in the hard task condition where the users failed in most of the tasks. This finding highlights the importance of understanding: (1) what kinds of product expectations participants bring with them to the test, (2) how well these expectations represent those of the intended user population, and (3) how the test situation itself influences and may bias these expectations.     

基于粗糙集的教育网站综合评价研究

教育网站的综合评价对教育信息化资源建设与发展具有重要的意义.依据粗糙集理论的对象分类能力,通过属性约简,删除冗余属性,有效简化评价指标体系.以知识的信息量概念为基础对属性的重要性进行定义,得到了指标权重确定方法.在此基础上建立了教育网站的综合评价模型,由此对评价对象进行排序选优,从而得到教育网站综合评价的新方法.最后通过实例验证了该方法的实用性和可行性.     

Predictive use error analysis – Development of AEA,SHERPA and PHEA to better predict,identify and present use errors

In health care, the use of technical equipment plays an integral part. To achieve a high level of patient safety, it is important to avoid use errors when handling equipment. Use errors can be mitigated by performing analyses of potential use errors during the design process. One proactive analytical method for use error analysis is Predictive Use Error Analysis (PUEA), which is a further development of the methods Action Error Analysis (AEA), Systematic Human Error Reduction and Prediction Approach (SHERPA) and Predictive Human Error Analysis (PHEA). PUEA employs a detailed process for breaking down the user's tasks into steps and then identifying and investigating potential errors of use for each step. Compared with other methods, it is significant in its use of two question levels, greater inclusion of human cognition theory and that the results of the analysis are presented in matrixes.     

An approach to usability evaluation of e-learning applications

Despite recent advances of electronic technologies in e-learning, a consolidated evaluation methodology for e-learning applications is not available. The evaluation of educational software must consider its usability and more in general its accessibility, as well as its didactic effectiveness. This work is a first step towards the definition of a methodology for evaluating e-learning applications. Specific usability attributes capturing the peculiar features of these applications are identified. A preliminary user study involving a group of e-students, observed during their interaction with an e-learning application in a real situation, is reported. Then, the proposal is put forward to adapt to the e-learning domain a methodology for systematic usability evaluation, called SUE. Specifically, evaluation patterns are proposed that are able to drive the evaluators in the analysis of an e-learning application.     

Evaluation of user interfaces: EVADIS II— a comprehensive evaluation approach

As a result of the importance of the usability approach in system development and the EC's 'Directive concerning the minimum safety and health requirements for VDT workers' (EWG 1990), there is an accepted need for practical evaluation methods for user interfaces. The usability approach and the EC Directive are not restricted to user interface design, as they include the design of appropriate hardware and software, as well as organization, job, and task design. Therefore system designers are faced with many, often conflicting, requirements and need to address the question, 'How can usability requirements comprehensively be considered and evaluated in system development?' Customers buying hardware and software and introducing them into their organization ask, (How can I select easy-to-use hardware and software?' Both designers and customers need an evaluation procedure that covers all the organizational, user, hard- and software requirements. The evaluation method, EVADIS.II, we present in this paper overcomes characteristic deficiencies of previous evaluation methods. In particular, it takes the tasks, the user, and the organizational context into consideration during the evaluation process, and provides computer support for the use of the evaluation procedure.     

Work-domain knowledge in usability evaluation: Experiences with Cooperative Usability Testing

Usability evaluation helps to determine whether interactive systems support users in their work tasks. However, knowledge about those tasks and, more generally, about the work-domain is difficult to bring to bear on the processes and outcome of usability evaluation. One way to include such work-domain knowledge might be Cooperative Usability Testing, an evaluation method that consists of (a) interaction phases, similar to classic usability testing, and (b) interpretation phases, where the test participant and the moderator discuss incidents and experiences from the interaction phases. We have studied whether such interpretation phases improve the relevance of usability evaluations in the development of work-domain specific systems. The study included two development cases. We conclude that the interpretation phases generate additional insight and redesign suggestions related to observed usability problems. Also, the interpretation phases generate a substantial proportion of new usability issues, thereby providing a richer evaluation output. Feedback from the developers of the evaluated systems indicates that the usability issues that are generated in the interpretation phases have substantial impact on the software development process. The benefits of the interpretation phases may be explained by the access these provide both to the test participants’ work-domain knowledge and to their experiences as users.