A usability evaluation of medical software at an expert conference setting

Introduction

A usability test was employed to evaluate two medical software applications at an expert conference setting. One software application is a medical diagnostic tool (electrocardiogram [ECG] viewer) and the other is a medical research tool (electrode misplacement simulator [EMS]). These novel applications have yet to be adopted by the healthcare domain, thus, (1) we wanted to determine the potential user acceptance of these applications and (2) we wanted to determine the feasibility of evaluating medical diagnostic and medical research software at a conference setting as opposed to the conventional laboratory setting.

Methods

The medical diagnostic tool (ECG viewer) was evaluated using seven delegates and the medical research tool (EMS) was evaluated using 17 delegates that were recruited at the 2010 International Conference on Computing in Cardiology. Each delegate/participant was required to use the software and undertake a set of predefined tasks during the session breaks at the conference. User interactions with the software were recorded using screen-recording software. The ‘think-aloud’ protocol was also used to elicit verbal feedback from the participants whilst they attempted the pre-defined tasks. Before and after each session, participants completed a pre-test and a post-test questionnaire respectively.

Results

The average duration of a usability session at the conference was 34.69 min (SD = 10.28). However, taking into account that 10 min was dedicated to the pre-test and post-test questionnaires, the average time dedication to user interaction of the medical software was 24.69 min (SD = 10.28). Given we have shown that usability data can be collected at conferences, this paper details the advantages of conference-based usability studies over the laboratory-based approach. For example, given delegates gather at one geographical location, a conference-based usability evaluation facilitates recruitment of a convenient sample of international subject experts. This would otherwise be very expensive to arrange. A conference-based approach also allows for data to be collected over a few days as opposed to months by avoiding administration duties normally involved in laboratory based approach, e.g. mailing invitation letters as part of a recruitment campaign.Following analysis of the user video recordings, 41 (previously unknown) use errors were identified in the advanced ECG viewer and 29 were identified in the EMS application. All use errors were given a consensus severity rating from two independent usability experts. Out of a rating scale of 4 (where 1 = cosmetic and 4 = critical), the average severity rating for the ECG viewer was 2.24 (SD = 1.09) and the average severity rating for the EMS application was 2.34 (SD = 0.97). We were also able to extract task completion rates and times from the video recordings to determine the effectiveness of the software applications. For example, six out of seven tasks were completed by all participants when using both applications. This statistic alone suggests both applications already have a high degree of usability. As well as extracting data from the video recordings, we were also able to extract data from the questionnaires. Using a semantic differential scale (where 1 = poor and 5 = excellent), delegates highly rated the ‘responsiveness’, ‘usefulness’, ‘learnability’ and the ‘look and feel’ of both applications.

Conclusion

This study has shown the potential user acceptance and user-friendliness of the novel EMS and the ECG viewer applications within the healthcare domain. It has also shown that both medical diagnostic software and medical research software can be evaluated for their usability at an expert conference setting. The primary advantage of a conference-based usability evaluation over a laboratory-based evaluation is the high concentration of experts at one location, which is convenient, less time consuming and less expensive.     

On the reliability of mapping studies in software engineering

Background

Systematic literature reviews and systematic mapping studies are becoming increasingly common in software engineering, and hence it becomes even more important to better understand the reliability of such studies.

Objective

This paper presents a study of two systematic mapping studies to evaluate the reliability of mapping studies and point out some challenges related to this type of study in software engineering.

Method

The research is based on an in-depth case study of two published mapping studies on software product line testing.

Results

We found that despite the fact that the two studies are addressing the same topic, there are quite a number of differences when it comes to papers included and in terms of classification of the papers included in the two mapping studies.

Conclusions

From this we conclude that although mapping studies are important, their reliability cannot simply be taken for granted. Based on the findings we also provide four conjectures that further research has to address to make secondary studies (systematic mapping studies and systematic literature reviews) even more valuable to both researchers and practitioners.     

A study of cyclic dependencies on defect profile of software components

Background

Empirical evidence shows that dependency cycles among software components are pervasive in real-life software systems, although such cycles are known to be detrimental to software quality attributes such as understandability, testability, reusability, build-ability and maintainability.

Research goals

Can the use of extended object-oriented metrics make us better understand the relationships among cyclic related components and their defect-proneness?

Approach

First, we extend such metrics to mine and classify software components into two groups – the cyclic and the non-cyclic ones. Next, we have performed an empirical study of six software applications. Using standard statistical tests on four different hypotheses, we have determined the significance of the defect profiles of both groups.

Results

Our results show that most defects and defective components are concentrated in cyclic-dependent components, either directly or indirectly.

Discussion and conclusion

These results have important implications for software maintenance and system testing. By identifying the most defect-prone set in a software system, it is possible to effectively allocate testing resources in a cost efficient manner. Based on these results, we demonstrate how additional structural properties could be collected to understand component's defect proneness and aid decision process in refactoring defect-prone cyclic related components.     

Reliability analysis and optimal version-updating for open source software

Context

Although reliability is a major concern of most open source projects, research on this problem has attracted attention only recently. In addition, the optimal version-dating for open source software considering its special properties is not yet discussed.

Objective

In this paper, the reliability analysis and optimal version-updating for open source software are studied.

Method

A modified non-homogeneous Poisson process model is developed for open source software reliability modeling and analysis. Based on this model, optimal version-updating for open source software is investigated as well. In the decision process, the rapid release strategy and the level of reliability are the two most important factors. However, they are essentially contradicting with each other. In order to consider these two conflicting factors simultaneously, a new decision model based on multi-attribute utility theory is proposed.

Results

Our models are tested on the real world data sets from two famous open source projects: Apache and GNOME. It is found that traditional software reliability models provide overestimations of the reliability of open source software. In addition, the proposed decision model can help management to make a rational decision on the optimal version-updating for open source software.

Conclusion

Empirical results reveal that the proposed model for open source software reliability can describe the failure process more accurately. Furthermore, it can be seen that the proposed decision model can assist management to appropriately determine the optimal version-update time for open source software.     

Empowering village doctors and enhancing rural healthcare using cloud computing in a rural area of mainland China

Background

China's healthcare system often struggles to meet the needs of its 900 million people living in rural areas due to major challenges in preventive medicine and management of chronic diseases. Here we address some of these challenges by equipping village doctors (ViDs) with Health Information Technology and developing an electronic health record (EHR) system which collects individual patient information electronically to aid with implementation of chronic disease management programs.

Methods

An EHR system based on a cloud-computing architecture was developed and deployed in Xilingol county of Inner Mongolia using various computing resources (hardware and software) to deliver services over the health network using Internet when available. The system supports the work at all levels of the healthcare system, including the work of ViDs in rural areas. An analysis done on 291,087 EHRs created from November 2008 to June 2011 evaluated the impact the EHR system has on preventive medicine and chronic disease management programs in rural China.

Results

From 2008 to 2011 health records were created for 291,087 (26.25%) from 1,108,951 total Xilingol residents with 10,240 cases of hypertension and 1152 cases of diabetes diagnosed and registered. Furthermore, 2945 hypertensive and 305 diabetic patients enrolled in follow-up. Implementing the EHR system revealed a high rate of cholecystectomies leading to investigations and findings of drinking water contaminated with metals. Measures were taken to inform the population and clean drinking water was supplied.

Conclusions

The cloud-based EHR approach improved the care provision for ViDs in rural China and increased the efficiency of the healthcare system to monitor the health status of the population and to manage preventive care efforts. It also helped discover contaminated water in one of the project areas revealing further benefits if the system is expanded and improved.     

A systematic review of research on open source software in commercial software product development

Context

The popularity of the open source software development in the last decade, has brought about an increased interest from the industry on how to use open source components, participate in the open source community, build business models around this type of software development, and learn more about open source development methodologies. There is a need to understand the results of research in this area.

Objective

Since there is a need to understand conducted research, the aim of this study is to summarize the findings of research that has ben carried out on usage of open source components and development methodologies by the industry, as well as companies’ participation in the open source community.

Method

Systematic review through searches in library databases and manual identification of articles from the open source conference. The search was first carried out in May 2009 and then once again in May 2010.

Results

In 2009, 237 articles were first found, from which 19 were selected based on content and quality, and in 2010, 76 new articles were found from which four were selected. Twenty three articles were identified in total.

Conclusions

The articles could be divided into four categories: open source as part of component based software engineering, business models with open source in commercial organization, company participation in open source development communities, and usage of open source processes within a company.     

Development of online diary and self-management system on e-Healthcare for asthmatic children in Taiwan

Objective

Many regional programs of the countries educate asthmatic children and their families to manage healthcare data. This study aims to establish a Web-based self-management system, eAsthmaCare, to promote the electronic healthcare (e-Healthcare) services for the asthmatic children in Taiwan. The platform can perform real time online functionality based upon a five-tier infrastructure with mutually supportive components to acquire asthma diaries, quality of life assessments and health educations.

Methods

We have designed five multi-disciplinary portions on the interactive interface functioned with the analytical diagrams: (1) online asthma diary, (2) remote asthma assessment, (3) instantaneous asthma alert, (4) diagrammatical clinic support, and (5) asthma health education. The Internet-based asthma diary and assessment program was developed for patients to process self-management healthcare at home. In addition, the online analytical charts can help healthcare professionals to evaluate multi-domain health information of patients immediately.

Results

eAsthmaCare was developed by Java™ Servlet/JSP technology upon Apache Tomcat™ web server and Oracle™ database. Forty-one voluntary asthmatic children (and their parents) were intervened to examine the proposed system. Seven domains of satisfiability assessment by using the system were applied for approving the development. The average scores were scaled in the acceptable range for each domain to ensure feasibility of the proposed system.

Conclusion

The study revealed the details of system infrastructure and developed functions that can help asthmatic children in self-management for healthcare to enhance communications between patients and hospital professionals.     

Code smells as system-level indicators of maintainability: An empirical study

Context

Code smells are manifestations of design flaws that can degrade code maintainability. So far, no research has investigated if these indicators are useful for conducting system-level maintainability evaluations.

Aim

The research in this paper investigates the potential of code smells to reflect system-level indicators of maintainability.

Method

We evaluated four medium-sized Java systems using code smells and compared the results against previous evaluations on the same systems based on expert judgment and the Chidamber and Kemerer suite of metrics. The systems were maintained over a period of up to 4 weeks. During maintenance, effort (person-hours) and number of defects were measured to validate the different evaluation approaches.

Results

Most code smells are strongly influenced by size; consequently code smells are not good indicators for comparing the maintainability of systems differing greatly in size. Also, from the comparison of the different evaluation approaches, expert judgment was found as the most accurate and flexible since it considered effects due to the system's size and complexity and could adapt to different maintenance scenarios.

Conclusion

Code smell approaches show promise as indicators of the need for maintenance in a way that other purely metric-based approaches lack.     

Is software “green”? Application development environments and energy efficiency in open source applications

Context

The energy efficiency of IT systems, also referred to as Green IT, is attracting more and more attention. While several researchers have focused on the energy efficiency of hardware and embedded systems, the role of application software in IT energy consumption still needs investigation.

Objective

This paper aims to define a methodology for measuring software energy efficiency and to understand the consequences of abstraction layers and application development environments for the energy efficiency of software applications.

Method

We first develop a measure of energy efficiency that is appropriate for software applications. We then examine how the use of application development environments relates to this measure of energy efficiency for a sample of 63 open source software applications.

Results

Our findings indicate that a greater use of application development environments - specifically, frameworks and external libraries - is more detrimental in terms of energy efficiency for larger applications than for smaller applications. We also find that different functional application types have distinctly different levels of energy efficiency, with text and image editing and gaming applications being the most energy inefficient due to their intense use of the processor.

Conclusion

We conclude that different designs can have a significant impact on the energy efficiency of software applications. We have related the use of software application development environments to software energy efficiency suggesting that there may be a trade-off between development efficiency and energy efficiency. We propose new research to further investigate this topic.     

Exploring physical exposures and identifying high-risk work tasks within the floor layer trade

Introduction

Floor layers have high rates of musculoskeletal disorders yet few studies have examined their work exposures. This study used observational methods to describe physical exposures within floor laying tasks.

Methods

We analyzed 45 videos from 32 floor layers using Multimedia-Video Task Analysis software to determine the time in task, forces, postures, and repetitive hand movements for installation of four common flooring materials. We used the WISHA checklists to define exposure thresholds.

Results

Most workers (91%) met the caution threshold for one or more exposures. Workers showed high exposures in multiple body parts with variability in exposures across tasks and for different materials. Prolonged exposures were seen for kneeling, poor neck and low back postures, and intermittent but frequent hand grip forces.

Conclusions

Floor layers experience prolonged awkward postures and high force physical exposures in multiple body parts, which probably contribute to their high rates of musculoskeletal disorders.     

Kinematic and electromyographic differences between mouse and touchpad use on laptop computers

Background

Numerous studies have compared the postures and muscular activities induced by the various input devices available. To the authors' knowledge, no studies have yet compared upper body posture and movement, sEMG activity and muscle co-contractions induced by use of a mouse or of a touchpad.

Objective

The aim of this study was to compare posture, joint excursion and sEMG activity when using a laptop equipped with a mouse or a touchpad.

Methods

Trunk and upper arm posture, ranges of motion and muscle activities were measured in ten subjects during two standardized tasks.

Results

Mouse use induced larger shoulder abduction than touchpad use. On the other hand, when a touchpad was used, the upper arm joints were bound and fixed to a greater extent than during mouse use. Touchpad users are forced to maintain a more static posture. The upper arm electromyography results clearly indicate that touchpad use requires more stabilization than mouse use.

Conclusion

Motor tasks executed by means of the mouse allow a greater range of motions and reduce the biomechanical stress thanks to the greater postural mobility, even though the posture is less neutral. The results of our study tend to suggest that an external mouse should be preferred to the touchpad by frequent users of laptops.

Relevance to industry

This study is a suggestion for IT companies to give customers information of possible advantages using an external mouse. Furthermore our data highlight the importance of providing a mouse to all laptop users in order to reduce biomechanical risks.     

Learning performance with interactive simulations in medical education: Lessons learned from results of learning complex physiological models with the HAEMOdynamics SIMulator

Objective

Since simulations are often accepted uncritically, with excessive emphasis being placed on technological sophistication at the expense of underlying psychological and educational theories, we evaluated the learning performance of simulation software, in order to gain insight into the proper use of simulations for application in medical education.

Design

The authors designed and evaluated a software packet, following of user-centered development, which they call Haemodynamics Simulator (HAEMOSIM), for the simulation of complex physiological models, e.g., the modeling of arterial blood flow dependent on the pressure gradient, radius and bifurcations; shear–stress and blood flow profiles depending on viscosity and radius.

Measurements

In a quasi-experimental real-life setup, the authors compared the learning performance of 96 medical students for three conditions: (1) conventional text-based lesson; (2) HAEMOSIM alone and (3) HAEMOSIM with a combination of additional material and support, found necessary during user-centered development. The individual student’s learning time was unvarying in all three conditions.

Results

While the first two settings produced equivalent results, the combination of additional support and HAEMOSIM yielded a significantly higher learning performance. These results are discussed regarding Mayer’s multimedia learning theory, Sweller’s cognitive load theory, and claims of prior research on utilizing interactive simulations for learning.

Conclusion

The results showed that simulations can be beneficial for learning complex concepts, however, interacting with sophisticated simulations strain the limitation of cognitive processes; therefore successful application of simulations require careful additional guidance from medical professionals and a certain amount of previous knowledge on the part of the learners. The inclusion of pedagogical and psychological expertise into the design and development of educational software is essential.     

Mutation testing on an object-oriented framework: An experience report

Context

The increasing presence of Object-Oriented (OO) programs in industrial systems is progressively drawing the attention of mutation researchers toward this paradigm. However, while the number of research contributions in this topic is plentiful, the number of empirical results is still marginal and mostly provided by researchers rather than practitioners.

Objective

This article reports our experience using mutation testing to measure the effectiveness of an automated test data generator from a user perspective.

Method

In our study, we applied both traditional and class-level mutation operators to FaMa, an open source Java framework currently being used for research and commercial purposes. We also compared and contrasted our results with the data obtained from some motivating faults found in the literature and two real tools for the analysis of feature models, FaMa and SPLOT.

Results

Our results are summarized in a number of lessons learned supporting previous isolated results as well as new findings that hopefully will motivate further research in the field.

Conclusion

We conclude that mutation testing is an effective and affordable technique to measure the effectiveness of test mechanisms in OO systems. We found, however, several practical limitations in current tool support that should be addressed to facilitate the work of testers. We also missed specific techniques and tools to apply mutation testing at the system level.     

Usability evaluation methods for the web: A systematic mapping study

Context

In recent years, many usability evaluation methods (UEMs) have been employed to evaluate Web applications. However, many of these applications still do not meet most customers’ usability expectations and many companies have folded as a result of not considering Web usability issues. No studies currently exist with regard to either the use of usability evaluation methods for the Web or the benefits they bring.

Objective

The objective of this paper is to summarize the current knowledge that is available as regards the usability evaluation methods (UEMs) that have been employed to evaluate Web applications over the last 14 years.

Method

A systematic mapping study was performed to assess the UEMs that have been used by researchers to evaluate Web applications and their relation to the Web development process. Systematic mapping studies are useful for categorizing and summarizing the existing information concerning a research question in an unbiased manner.

Results

The results show that around 39% of the papers reviewed reported the use of evaluation methods that had been specifically crafted for the Web. The results also show that the type of method most widely used was that of User Testing. The results identify several research gaps, such as the fact that around 90% of the studies applied evaluations during the implementation phase of the Web application development, which is the most costly phase in which to perform changes. A list of the UEMs that were found is also provided in order to guide novice usability practitioners.

Conclusions

From an initial set of 2703 papers, a total of 206 research papers were selected for the mapping study. The results obtained allowed us to reach conclusions concerning the state-of-the-art of UEMs for evaluating Web applications. This allowed us to identify several research gaps, which subsequently provided us with a framework in which new research activities can be more appropriately positioned, and from which useful information for novice usability practitioners can be extracted.     

Reconciling software development models: A quasi-systematic review

Purpose

The purpose of this paper is to characterize reconciliation among the plan-driven, agile, and free/open source software models of software development.

Design/methodology/approach

An automated quasi-systematic review identified 42 papers, which were then analyzed.

Findings

The main findings are: there exist distinct - organization, group and process - levels of reconciliation; few studies deal with reconciliation among the three models of development; a significant amount of work addresses reconciliation between plan-driven and agile development; several large organizations (such as Microsoft, Motorola, and Philips) are interested in trying to combine these models; and reconciliation among software development models is still an open issue, since it is an emerging area and research on most proposals is at an early stage.

Research limitations

Automated searches may not capture relevant papers in publications that are not indexed. Other data sources not amenable to execution of the protocol were not used. Data extraction was performed by only one researcher, which may increase the risk of threats to internal validity.

Implications

This characterization is important for practitioners wanting to be current with the state of research. This review will also assist the scientific community working with software development processes to build a common understanding of the challenges that must be faced, and to identify areas where research is lacking. Finally, the results will be useful to software industry that is calling for solutions in this area.

Originality/value

There is no other systematic review on this subject, and reconciliation among software development models is an emerging area. This study helps to identify and consolidate the work done so far and to guide future research. The conclusions are an important step towards expanding the body of knowledge in the field.     

A systematic review of evaluation of variability management approaches in software product lines

Context

Variability management (VM) is one of the most important activities of software product-line engineering (SPLE), which intends to develop software-intensive systems using platforms and mass customization. VM encompasses the activities of eliciting and representing variability in software artefacts, establishing and managing dependencies among different variabilities, and supporting the exploitation of the variabilities for building and evolving a family of software systems. Software product line (SPL) community has allocated huge amount of effort to develop various approaches to dealing with variability related challenges during the last two decade. Several dozens of VM approaches have been reported. However, there has been no systematic effort to study how the reported VM approaches have been evaluated.

Objective

The objectives of this research are to review the status of evaluation of reported VM approaches and to synthesize the available evidence about the effects of the reported approaches.

Method

We carried out a systematic literature review of the VM approaches in SPLE reported from 1990s until December 2007.

Results

We selected 97 papers according to our inclusion and exclusion criteria. The selected papers appeared in 56 publication venues. We found that only a small number of the reviewed approaches had been evaluated using rigorous scientific methods. A detailed investigation of the reviewed studies employing empirical research methods revealed significant quality deficiencies in various aspects of the used quality assessment criteria. The synthesis of the available evidence showed that all studies, except one, reported only positive effects.

Conclusion

The findings from this systematic review show that a large majority of the reported VM approaches have not been sufficiently evaluated using scientifically rigorous methods. The available evidence is sparse and the quality of the presented evidence is quite low. The findings highlight the areas in need of improvement, i.e., rigorous evaluation of VM approaches. However, the reported evidence is quite consistent across different studies. That means the proposed approaches may be very beneficial when they are applied properly in appropriate situations. Hence, it can be concluded that further investigations need to pay more attention to the contexts under which different approaches can be more beneficial.     

Who benefits from simulator training: Personality and heart rate variability in relation to situation awareness during navigation training

Objective

This study tested the relationship between individual differences and Situation Awareness (SA) during training in a navigation simulator.

Background

Simulators have become an important tool in the training and education of maritime personnel with a view to improving decisions and performance. There are a few, inconclusive empirical studies that have focused on mapping personality characteristics and their link to the ability to generate and maintain SA during simulator training.

Method

Thirty-six first-year students from the Royal Norwegian Navy Officer Candidate School participated in the study. SA was measured using both subjective and observer ratings.

Results

The results indicated that low scores on Neuroticism and high scores on Extraversion and Conscientiousness (resilient personality type) predicted both subjective and observer-rated SA. Furthermore, participants with high SA were able to modulate their Heart Rate Variability (HRV), with suppression of HRV during navigation training as well showing recovery of HRV.

Application

The potential applications of this research include the assessment of personality differences as a tool in selecting navigators, and the use of HRV as an objective index of adaptability to environmental demands.     

Evaluation of a haptics-based virtual reality temporal bone simulator for anatomy and surgery training

Introduction

Virtual reality simulation training may improve knowledge of anatomy and surgical skills. We evaluated a 3-dimensional, haptic, virtual reality temporal bone simulator for dissection training.

Methods

The subjects were 7 otolaryngology residents (3 training sessions each) and 7 medical students (1 training session each). The virtual reality temporal bone simulation station included a computer with software that was linked to a force-feedback hand stylus, and the system recorded performance and collisions with vital anatomic structures. Subjects performed virtual reality dissections and completed questionnaires after the training sessions.

Results

Residents and students had favorable responses to most questions of the technology acceptance model (TAM) questionnaire. The average TAM scores were above neutral for residents and medical students in all domains, and the average TAM score for residents was significantly higher for the usefulness domain and lower for the playful domain than students. The average satisfaction questionnaire for residents showed that residents had greater overall satisfaction with cadaver temporal bone dissection training than training with the virtual reality simulator or plastic temporal bone. For medical students, the average comprehension score was significantly increased from before to after training for all anatomic structures. Medical students had significantly more collisions with the dura than residents. The residents had similar mean performance scores after the first and third training sessions for all dissection procedures.

Discussion

The virtual reality temporal bone simulator provided satisfactory training for otolaryngology residents and medical students.