Theoretical and practical issues in evaluating the quality of conceptual models: current state and future directions

An international standard has now been established for evaluating the quality of software products. However there is no equivalent standard for evaluating the quality of conceptual models. While a range of quality frameworks have been proposed in the literature, none of these have been widely accepted in practice and none has emerged as a potential standard. As a result, conceptual models continue to be evaluated in practice in an ad hoc way, based on common sense, subjective opinions and experience. For conceptual modelling to progress from an “art” to an engineering discipline, quality standards need to be defined, agreed and applied in practice. This paper conducts a review of research in conceptual model quality and identifies the major theoretical and practical issues which need to be addressed. We consider how conceptual model quality frameworks can be structured, how they can be developed, how they can be empirically validated and how to achieve acceptance in practice. We argue that the current proliferation of quality frameworks is counterproductive to the progress of the field, and that researchers and practitioners should work together to establish a common standard (or standards) for conceptual model quality. Finally, we describe some initial efforts towards developing a common standard for data model quality, which may provide a model for future standardisation efforts.     

Work analysis in French language ergonomics: origins and current research trends

This paper is not so much a synthesis as a reflection on work analysis in French language ergonomics. It is structured around two questions. First, how is francophone analysis positioned in relation to other developments which themselves have attempted to describe work—like anglophone taxonomy? Second, how has francophone analysis evolved? The author suggests that the evolution reveals a shift. Gradually, over the years, questions of meaning and analysis have taken on a greater importance. This shift appears clearly, in work analysis and verbalization, two topics which are discussed in the paper.     

Research and trends in the field of e-learning from 2001 to 2005: A content analysis of cognitive studies in selected journals

This paper provided a content analysis of studies in the field of cognition in e-learning that were published in five Social Sciences Citation Index (SSCI) journals (i.e. Computers and Education, British Journal of Educational Technology, Innovations in Education and Teaching International, Educational Technology Research & Development, and Journal of Computer Assisted Learning) from 2001 to 2005. Among the 1027 articles published in these journals from 2001 to 2005, 444 articles were identified as being related to the topic of cognition in e-learning. These articles were cross analyzed by published years, journal, research topic, and citation count. Furthermore, 16 highly-cited articles across different topics were chosen for further analysis according to their research settings, participants, research design types, and research methods. It was found from the analysis of the 444 articles that “Instructional Approaches,” “Learning Environment,” and “Metacognition” were the three most popular research topics, but the analysis of the citation counts suggested that the studies related to “Instructional Approaches,” “Information Processing” and “Motivation” might have a greater impact on subsequent research. Although the use of questionnaires might still be the main method of gathering research data in e-learning cognitive studies, a clear trend was observed that more and more studies were utilizing learners’ log files or online messages as data sources for analysis. The results of the analysis provided insights for educators and researchers into research trends and patterns of cognition in e-learning.     

Evaluating intertwined effects in e-learning programs: A novel hybrid MCDM model based on factor analysis and DEMATEL

Internet evolution has affected all industrial and commercial activity and accelerated e-learning growth. Due to cost, time, or flexibility for designer courses and learners, e-learning has been adopted by corporations as an alternative training method. E-learning effectiveness evaluation is vital, and evaluation criteria are diverse. A large effort has been made regarding e-learning effectiveness evaluation; however, a generalized quantitative evaluation model, which considers both the interaffected relation between criteria and the fuzziness of subjective perception concurrently, is lacking. In this paper, the proposed new novel hybrid MCDM model addresses the independent relations of evaluation criteria with the aid of factor analysis and the dependent relations of evaluation criteria with the aid of DEMATEL. The AHP and the fuzzy integral methods are used for synthetic utility in accordance with subjective perception environment. Empirical experimental results show the proposed model is capable of producing effective evaluation of e-learning programs with adequate criteria that fit with respondent’s perception patterns, especially when the evaluation criteria are numerous and intertwined.     

Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power

Experimental analysis of the performance of a proposed method is a crucial and necessary task in an investigation. In this paper, we focus on the use of nonparametric statistical inference for analyzing the results obtained in an experiment design in the field of computational intelligence. We present a case study which involves a set of techniques in classification tasks and we study a set of nonparametric procedures useful to analyze the behavior of a method with respect to a set of algorithms, such as the framework in which a new proposal is developed.Particularly, we discuss some basic and advanced nonparametric approaches which improve the results offered by the Friedman test in some circumstances. A set of post hoc procedures for multiple comparisons is presented together with the computation of adjusted p-values. We also perform an experimental analysis for comparing their power, with the objective of detecting the advantages and disadvantages of the statistical tests described. We found that some aspects such as the number of algorithms, number of data sets and differences in performance offered by the control method are very influential in the statistical tests studied. Our final goal is to offer a complete guideline for the use of nonparametric statistical procedures for performing multiple comparisons in experimental studies.