Context checklist for industrial software engineering research and practice

The relevance of context is particularly stressed in case studies, where it is said that “case study is an empirical method aimed at investigating contemporary phenomena in their context”. In this research, we classify context information and provide a context checklist for industrial software engineering. The checklist serves the purpose of (a) supporting researchers and practitioners in characterizing the context in which they are working; (b) supporting researchers with a checklist to identify relevant contextual information to elicit and report during primary and secondary studies. We utilized a systematic approach for constructing the classification of context information and provided a detailed definition for each item. We collected feedback from researchers as well as practitioners. The usefulness of the checklist was perceived more positively by researchers than practitioners, though they highlighted benefits (raising awareness of the importance of context and usefulness for management). The understandability was perceived positively by both practitioners and researchers. The checklist may serve as a “meta-model”, forming the basis for specific adaptations for different research areas, and as input for researchers deciding which context information to extract in systematic reviews. The checklist may also help researchers in reporting context in research papers.     

A framework for epistemological analysis in empirical (laboratory and field) studies

In their search for generalizable behavioral patterns and design principles, cognitive field researchers should reflect on the epistemological limitations of empirical studies. In this paper we describe a framework for epistemological analysis that can help serve this purpose and discuss its application to two prototypical cases of cognitive engineering research: laboratory experiments and field studies. The framework examines two, often implicit, processes in empirical research: the abstraction from empirical data and the substantiation of theoretical constructs and principles. By explicitly considering these two processes in several systematic steps, we can gain appreciation for the epistemological contribution of empirical studies to cognitive engineering research. The framework and its application also provide guidance to such important issues as generalizability of results and external validity. Possible applications of this research include providing guidance to researchers and practitioners in evaluating design principles or conducting field studies.     

Preliminary guidelines for empirical research in software engineering

Empirical software engineering research needs research guidelines to improve the research and reporting processes. We propose a preliminary set of research guidelines aimed at stimulating discussion among software researchers. They are based on a review of research guidelines developed for medical researchers and on our own experience in doing and reviewing software engineering research. The guidelines are intended to assist researchers, reviewers, and meta-analysts in designing, conducting, and evaluating empirical studies. Editorial boards of software engineering journals may wish to use our recommendations as a basis for developing guidelines for reviewers and for framing policies for dealing with the design, data collection, and analysis and reporting of empirical studies.     

Doing online research involving university students with disabilities: Methodological issues

Changes in how disability is understood have determined an increase in the number of people with disabilities who study at university. Several studies have aimed to investigate the experiences of students with disabilities. However, one difficulty in these studies is that very different types of disability (e.g., reading disabilities as compared to motor disability) are grouped together under the broad label of “students with disability”. Recently, the increase in access to computers and technology has made it possible to distribute questionnaires through the Internet and reach a higher number of participants; however, the use of online questionnaires for conducting research on the experiences of students with disabilities poses theoretical, methodological and ethical challenges to the researcher. Some of these issues are common to questionnaires conducted both online and offline; others, however, are typical of online studies. Here we will review these problems and their consequences in terms of research validity, along with some possible solutions that may minimize the risks of harming the privacy of participants, losing responses, and biasing data.     

Three empirical studies on the agreement of reviewers about the quality of software engineering experiments

ContextDuring systematic literature reviews it is necessary to assess the quality of empirical papers. Current guidelines suggest that two researchers should independently apply a quality checklist and any disagreements must be resolved. However, there is little empirical evidence concerning the effectiveness of these guidelines.AimsThis paper investigates the three techniques that can be used to improve the reliability (i.e. the consensus among reviewers) of quality assessments, specifically, the number of reviewers, the use of a set of evaluation criteria and consultation among reviewers. We undertook a series of studies to investigate these factors.MethodTwo studies involved four research papers and eight reviewers using a quality checklist with nine questions. The first study was based on individual assessments, the second study on joint assessments with a period of inter-rater discussion. A third more formal randomised block experiment involved 48 reviewers assessing two of the papers used previously in teams of one, two and three persons to assess the impact of discussion among teams of different size using the evaluations of the “teams” of one person as a control.ResultsFor the first two studies, the inter-rater reliability was poor for individual assessments, but better for joint evaluations. However, the results of the third study contradicted the results of Study 2. Inter-rater reliability was poor for all groups but worse for teams of two or three than for individuals.ConclusionsWhen performing quality assessments for systematic literature reviews, we recommend using three independent reviewers and adopting the median assessment. A quality checklist seems useful but it is difficult to ensure that the checklist is both appropriate and understood by reviewers. Furthermore, future experiments should ensure participants are given more time to understand the quality checklist and to evaluate the research papers.     

The impact of learning design on student behaviour,satisfaction and performance: A cross-institutional comparison across 151 modules

Pedagogically informed designs of learning are increasingly of interest to researchers in blended and online learning, as learning design is shown to have an impact on student behaviour and outcomes. Although learning design is widely studied, often these studies are individual courses or programmes and few empirical studies have connected learning designs of a substantial number of courses with learning behaviour. In this study we linked 151 modules and 111.256 students with students' behaviour (<400 million minutes of online behaviour), satisfaction and performance at the Open University UK using multiple regression models. Our findings strongly indicate the importance of learning design in predicting and understanding Virtual Learning Environment behaviour and performance of students in blended and online environments. In line with proponents of social learning theories, our primary predictor for academic retention was the time learners spent on communication activities, controlling for various institutional and disciplinary factors. Where possible, appropriate and well designed communication tasks that align with the learning objectives of the course may be a way forward to enhance academic retention.     

Sessional,weekly and diurnal patterns of computer lab usage by students attending a regional University in Australia

Most universities have invested in extensive infrastructure in the form on computer laboratories and computer kiosks. However, is this investment justified when it is suggested that students work predominantly from home using their own computers? This paper provides an empirical study investigating how students at a regional university multi-campus university use computer laboratories. The findings suggest that universities need to reassess the traditional driving force and expectation for extensive on-campus computing facilities as there is a huge underutilisation of these facilities. Instead it is recommended that alternative computing facilities in the form of student-centred facilities such as wireless hotspots could be provided.     

Technical efficiency of business administration courses: a simultaneous analysis using DEA and SFA

The aim of this study is to contribute to the existing literature on the efficiency of higher education institutions. The technical efficiency of higher education has been mostly studied using methods such as data envelopment analysis and stochastic frontier analysis. Many researchers compare either the efficiency among public education institutions or among departments in a university by using variables that give equal importance to teaching and research. In this paper, both these methodologies are applied to measure the efficiency of higher education courses, especially the business administration courses offered by private for‐profit institutions that focus just on education and are located in the same geographical region. The variables selected covered the specific aspects of these courses and the results showed the complementarities of these two approaches.     

对软件工程中经验研究的调查

为了描述、理解、评估、预测、控制、管理或者改善与软件相关的内容,研究者常常使用经验研究的方法.经验研究在软件工程领域已经得到广泛的应用并备受关注.为了了解近年来软件工程中经验研究的特点,并希望经验研究方法为更多研究者所了解,我们通过系统映射的方法,对软件工程中经验研究的典型期刊ESE（Empirical Software Engineering）近5年的论文做了一个调研,搜集了从2013年1月至2017年6月发表在该期刊的250篇论文.通过定性和定量的分析,给出了软件工程领域采用经验研究的主要目的,常用的经验研究方法、以及这些方法在软件工程各个领域中的使用情况和呈现的一些新特征.之后,分析了经验研究的主要数据来源、采集手段、常用的数理统计方法以及开源项目在经验研究中的使用情况等,给出了研究者对有效性和可重现性问题的关心程度.最后进行了本文的有效性分析,并进一步探讨了经验研究的发展方向和大数据时代下经验研究面临的机遇及一些开放性问题.     

Can interactive visualization tools engage and support pre-university students in exploring non-trivial mathematical concepts?

Many students find it difficult to engage with mathematical concepts. As a relatively new class of learning tools, visualization tools may be able to promote higher levels of engagement with mathematical concepts. Often, development of new tools may outpace empirical evaluations of the effectiveness of these tools, especially in educational contexts. This seems to be the case with educational visualization tools. Much evidence about the effectiveness of these tools appears to be more suggestive than based on empirical evaluations. In this paper, we attempt to fill this gap and provide empirical evidence for the use of visualization tools in supporting exploratory and other learning-related activities. In particular, we aim to investigate whether visualization tools can be used to engage pre-university students in exploring non-trivial mathematical concepts. We focus particularly on this age group and content domain because of the difficulty these students may encounter when trying to investigate more challenging mathematical concepts. Also, it is during their formative years before university that students’ predisposition and likeness towards mathematical ideas are formed. We report in this paper a study assessing whether a visualization tool, whose design was informed explicitly by research from information visualization and human–computer interaction, could engage pre-university students in their exploration and learning of more advanced mathematical concepts. Students who participated in this study came from multiple grade levels and have diverse cognitive and language skills as well as preferences towards mathematics. The results of this study indicate that visualization tools can effectively engage these students and support their exploration of non-trivial mathematical concepts, only if the tool is designed such that it can cater the diverse needs of these students.     

RSS — a primer for publishers and content providers

The key observation about institutional research repositories is that they are under used. This paper looks at why take-up has been slow, and what might encourage researchers to use these facilities. This paper surveys the evolution of practice and rationale for the institutional research repository, looks at existing use studies and opinion surveys of users, and examines ways in which university libraries, the main proponents, have changed their approaches in response to their experience. The experience of Swinburne University of Technology, a partner in the Australian ARROW project, is drawn upon.     

Gravitating towards mobile phone (GoToMP) during lecture periods by students: Why are they using it? and how can it be measured?

With the growing amount of research on the use of technology, especially the mobile phones, the paucity of research in the area of mobile phone use that focus on the reasons why students use their mobile phones during lecture periods is somewhat surprising. Findings from the previous studies have documented that students continue to use their mobile phones during lecture periods even in classes where its use is banned. Thus, the motives that determine students' continued intention to use their mobile phones during lecture periods becomes an important issue that deserves urgent attention from researchers and education policymakers. In the present investigation, we tried to fill this gap by examining the determinants of students' continuance use of mobile phones during lecture periods. To achieve this, three stages of instrument development (item generation, scale development, and instrument testing) were undertaken through two studies. The results of the exploratory and confirmatory factor analysis confirmed the existence of six dimensions in the proposed measurement instrument (which we referred to as gravitating towards mobile phone (GoToMP)); the dimensions are boredom, class-related use, social connection, emergency, addiction, and perceived behavioral control. The combined findings from the two studies reported here provide initial evidence that GoToMP exhibits excellent psychometric properties and can be used in future empirical research on mobile phone use behaviors of students.     

DP-Share: Privacy-Preserving Software Defect Prediction Model Sharing Through Differential Privacy

In current software defect prediction (SDP) research, most previous empirical studies only use datasets provided by PROMISE repository and this may cause a threat to the external validity of previous empirical results. Instead of SDP dataset sharing, SDP model sharing is a potential solution to alleviate this problem and can encourage researchers in the research community and practitioners in the industrial community to share more models. However, directly sharing models may result in privacy disclosure, such as model inversion attack. To the best of our knowledge, we are the first to apply differential privacy (DP) to privacy-preserving SDP model sharing and then propose a novel method DP-Share, since DP mechanisms can prevent this attack when the privacy budget is carefully selected. In particular, DP-Share first performs data preprocessing for the dataset, such as over-sampling for minority instances (i.e., defective modules) and conducting discretization for continuous features to optimize privacy budget allocation. Then, it uses a novel sampling strategy to create a set of training sets. Finally it constructs decision trees based on these training sets and these decision trees can form a random forest (i.e., model). The last phase of DP-Share uses Laplace and exponential mechanisms to satisfy the requirements of DP. In our empirical studies, we choose nine experimental subjects from real software projects. Then, we use AUC (area under ROC curve) as the performance measure and holdout as our model validation technique. After privacy and utility analysis, we find that DP-Share can achieve better performance than a baseline method DF-Enhance in most cases when using the same privacy budget. Moreover, we also provide guidelines to effectively use our proposed method. Our work attempts to fill the research gap in terms of differential privacy for SDP, which can encourage researchers and practitioners to share more SDP models and then effectively advance the state of the art of SDP.     

Information Visualization Evaluation Using Crowdsourcing

Visualization researchers have been increasingly leveraging crowdsourcing approaches to overcome a number of limitations of controlled laboratory experiments, including small participant sample sizes and narrow demographic backgrounds of study participants. However, as a community, we have little understanding on when, where, and how researchers use crowdsourcing approaches for visualization research. In this paper, we review the use of crowdsourcing for evaluation in visualization research. We analyzed 190 crowdsourcing experiments, reported in 82 papers that were published in major visualization conferences and journals between 2006 and 2017. We tagged each experiment along 36 dimensions that we identified for crowdsourcing experiments. We grouped our dimensions into six important aspects: study design & procedure, task type, participants, measures & metrics, quality assurance, and reproducibility. We report on the main findings of our review and discuss challenges and opportunities for improvements in conducting crowdsourcing studies for visualization research.     

Data harmonisation for information fusion in digital healthcare: A state-of-the-art systematic review,meta-analysis and future research directions

Removing the bias and variance of multicentre data has always been a challenge in large scale digital healthcare studies, which requires the ability to integrate clinical features extracted from data acquired by different scanners and protocols to improve stability and robustness. Previous studies have described various computational approaches to fuse single modality multicentre datasets. However, these surveys rarely focused on evaluation metrics and lacked a checklist for computational data harmonisation studies. In this systematic review, we summarise the computational data harmonisation approaches for multi-modality data in the digital healthcare field, including harmonisation strategies and evaluation metrics based on different theories. In addition, a comprehensive checklist that summarises common practices for data harmonisation studies is proposed to guide researchers to report their research findings more effectively. Last but not least, flowcharts presenting possible ways for methodology and metric selection are proposed and the limitations of different methods have been surveyed for future research.     

The hybrid course: Merging on-line instruction and the traditional classroom

The efficacy of on-line learning environments has received considerable attention in recent years. Generally, researchers have found little impact of virtual learning environments on learning outcomes. With a few exceptions, however, these studies have focused upon distance learning courses where students enter the course without expectations of significant amounts of face-to-face contact with the instructor(s). Information technology, and particularly Web-based multimedia, increasingly offers opportunities to transfer content delivery and other associated activities from face-to-face settings to on-line venues. However, in order for this transference of content and process to be effective in courses recognized as face-to-face courses, students must find the virtual environment appealing. They must also achieve learning outcomes at least as high as those achieved in traditional class settings. The research reported upon herein demonstrates the utility of on-line learning environments in traditional classes both as an efficient means for executing activities previously tethered to the classroom setting and as a means to allow the pursuit of higher levels of learning. This revised version was published online in July 2006 with corrections to the Cover Date.     

Mediating the impact of technology usage on perceived ease of use by anxiety

Computerphobic adults including first-year university students have been reported to range from 25% to 50%. Although self-reported computer anxiety has reduced in the past decade, it continues to be a significant issue for many. This is especially true for students of today where the stakes are high when using computers for their course work. Anxiety becomes even more critical when students are taking online courses. Past research has shown that computer experience is a strong predictor of whether or not a user will suffer anxiety symptoms while using a computer. Moreover, a substantial amount of work has been done to study the effect of previous computer experience on anxiety and on ease of use of information technologies using the technology acceptance model. However, few, if any, have investigated the role of anxiety in mediating technology usage experience on perceived ease of use. In this study, technology usage is viewed from two perspectives, the computer and the internet. We study the influence of anxiety in mediating the impact of computer and internet experiences on perceived ease of use. Questionnaire data from 114 university students were analyzed. The context was the use of a quiz tool to practice multiple choice questions. Contrary to most related studies, results indicated that anxiety has no mediating role on the impact of computer experience and perceived ease of use. However, anxiety was shown to present some moderating influence on perceived ease of use.     

Empirical research methodologies and studies in Requirements Engineering: How far did we come?

Since the inception of the RE conference series (1992), both researchers and practitioners in the RE community have acknowledged the significance of empirical evaluation as an instrument to gain knowledge about various aspects of RE phenomena and the validity of our research results. A significant number of empirical studies have been conducted in the search for knowledge about RE problems as well as evidence of successful and less successful application of proposed solutions. This editorial presents the progress empirical RE research has made since 1992. Based on a search in the Scopus digital library, we report from an analysis of peer-reviewed systematic literature reviews and mapping studies to showcase major areas of RE research that use methods from the Empirical Software Engineering paradigm. We summarize prior empirical research in RE and introduce the contributors to this special issue on empirical research methodologies and studies in RE.     

Impact of using interactive devices in Spanish early childhood education public schools

The adoption of Information and Communication Technologies in early childhood education is crucial for adapting traditional classrooms to the digital era. Over time, young children are increasingly using touch screen technologies such as tablets at home and in early childhood settings. However, the literature shows that there is a significant gap in knowledge of using this technology in early childhood education. Most researchers have focused on the pedagogical theory behind using touch screen devices, but there have not been many empirical studies about how these technologies affect students' learning processes. This paper presents three learning experiences where early childhood students perform educational activities using tablet computers, interactive whiteboards, and paper cards. The results show that students who used the technology were more motivated and achieved better results that those who used paper cards.