Performance analysis of GAME: A generic automated marking environment

This paper describes the Generic Automated Marking Environment (GAME) and provides a detailed analysis of its performance in assessing student programming projects and exercises. GAME has been designed to automatically assess programming assignments written in a variety of languages based on the “structure” of the source code and the correctness of the program’s output. Currently, the system is able to mark programs written in Java, C++ and the C language. To use the system, instructors are required to provide a simple “marking schema” for each given assessment item, which includes pertinent information such as the location of files and the model solution. In this research, GAME has been tested on a number of student programming exercises and assignments and its performance has been compared against that of a human marker. An in-depth statistical analysis of the comparison is presented, providing encouraging results and directions for employing GAME as a tool for teaching and learning.     

Statistical profiles of highly-rated learning objects

The continuously growth of learning resources available in on-line repositories has raised the concern for the development of automated methods for quality assessment. The current existence of on-line evaluations in such repositories has opened the possibility of searching for statistical profiles of highly-rated resources that can be used as priori indicators of quality. In this paper, we analyzed 35 metrics in learning objects refereed inside the MERLOT repository and elaborated profiles for these resources regarding the different categories of disciplines and material types available. We found that some of the intrinsic metrics presented significant differences between highly rated and poorly-rated resources and that those differences are dependent on the category of discipline to which the resource belongs and on the type of the resource. Moreover, we found that different profiles should be identified according to the type of rating (peer-review or user) under evaluation. At last, we developed an initial model using linear discriminant analysis to evaluate the strength of relevant metrics when performing an automated quality classification task. The initial results of this work are promising and will be used as the foundations for the further development of an automated tool for contextualized quality assessment of learning objects inside repositories.     

Computer support for learning mathematics: A learning environment based on recreational learning objects

In this paper, we introduce an electronic collaborative learning environment based on Interactive Instructors of Recreational Mathematics (IIRM), establishing an alternative approach for motivating students towards mathematics. The IIRM are educational software components, specializing in mathematical concepts, presented through recreational mathematics, conceived as interactive, recreation-oriented learning objects, integrated within the environment. We present the architecture of the learning environment which integrates communication services that support the interaction processes of the learning community, through instant messaging, chat rooms, and multi-player math games. Through the environment’s interface of their personal workspace, students have access to several easy-to-use mechanisms that allows them to customize its content, its layout, and its appearance. At internal levels, the functionality of IIRM is enhanced with features supported by the environment infrastructure. We evaluated different aspects of the learning environment in three short, motivation-oriented math courses given to Mexican high-school students. The results indicate that the use of the IIRM-based electronic learning environment, positively affects student attitudes towards mathematics. We believe that this approach has the potential to promote the mathematics learning process, basically on its motivational aspects.     

Toward an integrated model for designing assessment systems: An analysis of the current status of computer-based assessments in science

Drawing upon an integrated model proposed by Bennett and Bejar (1998), this review study examined how 66 computer-based science assessments (CBSAs) in basic science and medicine took advantage of advanced technologies. The model regarded a CBSA as an integrated system, included several assessment components (e.g., assessment purpose, measured construct, test and task design, examinee interface, and scoring procedure), and emphasized the interplay among these components. Accordingly, this study systematically analyzed the item presentations of interactive multimedia, the constructs measured, the response formats in formative and summative assessments, the scoring procedures, the adaptive test activities administrated based on the algorithms related to the item response theory (IRT) and other rules beyond IRT, and the strategies for the automatic provision of informative hints and feedbacks in the CBSAs. Our analysis revealed that although only 19 out of 66 assessments took advantage of dynamic and interactive media for item presentations, CBSAs with these media could measure integrated understanding of science phenomena and complex problem-solving skills. However, we also found that limitations in automated scoring may lead to infrequent use of the automated provision of hints and feedbacks with open-ended and extended responses. These findings suggest the interrelatedness of the assessment components, and thus we argue that designers should repeatedly consider the relationships between components of CBSAs to ensure the validity of the assessments. Finally, we also indicate the issues for future research in computer-based assessments.     

The formation of learners’ semiosphere by authentic inquiry with an integrated learning object “Young Scientist”

An integrated learning object, a web-based inquiry environment “Young Scientist” for basic school level is introduced by applying the semiosphere conception for explaining learning processes. The study focused on the development of students’ (n = 30) awareness of the affordances of learning objects (LO) during the 3 inquiry tasks, and their ability of dynamically reconstructing meanings in the inquiry subtasks through exploiting these LO affordances in “Young Scientist”. The problem-solving data recorded by the inquiry system and the awareness questionnaire served as the data-collection methods.It was demonstrated that learners obtain complete awareness of the LO affordances in an integrated learning environment only after several problem-solving tasks. It was assumed that the perceived task-related properties and functions of LOs depend on students’ interrelations with LOs in specific learning contexts. Learners’ overall awareness of certain LO affordances, available in the inquiry system “Young Scientist”, developed with three kinds of patterns, describing the hierarchical development of the semiosphere model for learners. The better understanding of the LO affordances, characteristic to the formation of the functioning semiosphere, was significantly related to the advanced knowledge construction during these inquiry subtasks that presumed translation of information from one semiotic system to another. The implications of the research are discussed in the frames of the development of new contextual gateways for learning with virtual objects. It is assumed that effective LO-based learning has to be organized through pedagogically constrained gateways by manifesting certain LO affordances in the context in order to build up the dynamic semiosphere model for learners.     

Toward interactive mobile synchronous learning environment with context-awareness service

Mobile synchronous learning is a new challenge in the e-learning domain. While popular mobile communication devices, such as cell phones, cannot directly accommodate traditional synchronous content due to the major limitation of display size, other constraints also restrict convenient interactions while using mobile devices in a synchronous learning environment. These problems have motivated the authors of this study to design a context-awareness synchronous learning system and to develop a corresponding pedagogical framework. Different than existing synchronous learning strategies, the proposed system enhances the feedback mechanism and implements an enhanced model for achieving mobile interaction in a synchronous learning environment. The enhanced model is named Interactive Service Module, which enables interactions between teachers and students via short message delivery. In the proposed synchronous learning environment, different kinds of learning devices are used to access the same source of synchronous content simultaneously. To accommodate the diversity of devices, several content styles have been developed and an appropriate style can be selected to a learner via a decision mechanism. This mechanism is based on fuzzy weighted average technique to measure the average computational power for each device. Finally, questionnaires were used to evaluate the usability of the proposed synchronous learning environment, and the results indicate that our system can facilitate synchronous learning by enabling students to access lessons conveniently and efficiently from a wide variety of locations, using common mobile communication devices.     

An adaptive testing system for supporting versatile educational assessment

With the rapid growth of computer and mobile technology, it is a challenge to integrate computer based test (CBT) with mobile learning (m-learning) especially for formative assessment and self-assessment. In terms of self-assessment, computer adaptive test (CAT) is a proper way to enable students to evaluate themselves. In CAT, students are assessed through a process that uses item response theory (IRT), a well-founded psychometric theory. Furthermore, a large item bank is indispensable to a test, but when a CAT system has a large item bank, the test item selection of IRT becomes more tedious. Besides the large item bank, item exposure mechanism is also essential to a testing system. However, IRT all lack the above-mentioned points. These reasons have motivated the authors to carry out this study. This paper describes a design issue aimed at the development and implementation of an adaptive testing system. The system can support several assessment functions and different devices. Moreover, the researchers apply a novel approach, particle swarm optimization (PSO) to alleviate the computational complexity and resolve the problem of item exposure. Throughout the development of the system, a formative evaluation was embedded into an integral part of the design methodology that was used for improving the system. After the system was formally released onto the web, some questionnaires and experiments were conducted to evaluate the usability, precision, and efficiency of the system. The results of these evaluations indicated that the system provides an adaptive testing for different devices and supports versatile assessment functions. Moreover, the system can estimate students’ ability reliably and validly and conduct an adaptive test efficiently. Furthermore, the computational complexity of the system was alleviated by the PSO approach. By the approach, the test item selection procedure becomes efficient and the average best fitness values are very close to the optimal solutions.     

Generic service integration in adaptive learning experiences using IMS learning design

IMS Learning Design is a specification to capture the orchestration taking place in a learning scenario. This paper presents an extension called Generic Service Integration. This paradigm allows a bidirectional communication between the course engine in charge of the orchestration and conventional Web 2.0 tools. This communication allows the engine to configure external services so as to adjust their behaviour, and also retrieve information about their activity and use it to influence the orchestration process. This extension impacts all the phases of the life cycle of a learning activity: authoring, deployment and enactment. The paper analyses this impact and proposes appropriate adjustments. A pilot test was conducted and the obtained results show that learning experiences that adapt their flow-based on third party tools while maintaining interoperability, reusability and self-containment requirements are feasible.     

The effect of computer science on physics learning in a computational science environment

College and high-school students face many difficulties when dealing with physics formulas, such as a lack of understanding of their components or of the physical relationships between the two sides of a formula. To overcome these difficulties some instructors suggest combining simulations' design while learning physics, claiming that the programming process forces the students to understand the physical mechanism activating the simulation. This study took place in a computational-science course where high-school students programmed simulations of physical systems, thus combining computer science (CS) and mathematics with physics learning. The study explored the ways in which CS affected the students' conceptual understanding of the physics behind formulas. The major part of the analysis process was qualitative, although some quantitative analysis was applied as well. Findings revealed that a great amount of the time was invested by the students on representing their physics knowledge in terms of computer science. Three knowledge domains were found to be applied: structural, procedural and systemic. A fourth domain which enabled reflection on the knowledge was found as well, the domain of execution. Each of the domains was found to promote the emergence of knowledge integration processes (Linn & Eylon, 2006, 2011), thus promoting students’ physics conceptual understanding. Based on these findings, some instructional implications are discussed.     

Mobile learning: A framework and evaluation

Wireless data communications in form of Short Message Service (SMS) and Wireless Access Protocols (WAP) browsers have gained global popularity, yet, not much has been done to extend the usage of these devices in electronic learning (e-learning). This project explores the extension of e-learning into wireless/handheld (W/H) computing devices with the help of a mobile learning (m-learning) framework. This framework provides the requirements to develop m-learning applications that can be used to complement classroom or distance learning. A prototype application was developed to link W/H devices to three course websites. The m-learning applications were pilot-tested for two semesters with a total of 63 students from undergraduate and graduate courses at our university. The students used the m-learning environment with a variety of W/H devices and reported their experiences through a survey and interviews at the end of the semester. The results from this exploratory study provide a better understanding on the role of mobile technology in higher education.     

Design and evaluation of a learning environment to effectively provide network security skills

Information system security and network security are topics of increasing importance in the information society. They are also topics where the adequate education of professionals requires the use of specific laboratory environments where the practical aspects of the discipline may be addressed. However, most approaches currently used are excessively static and lack the flexibility that the education requirements of security professionals demand. In this paper we present NEMESIS, a scenario generation framework for education on system and network security, which is based on virtualization technologies and has been designed to be open, distributed, modular, scalable and flexible. Finally, an example scenario is described and some results validating the benefits of its use in undergraduate computer security courses are shown.     

Reusable,lifelike virtual humans for mentoring and role-playing

Lifelike, interactive digital characters, serving as mentors and role-playing actors, have been shown to significantly improve learner motivation and retention. However, the cost of modeling such characters, authoring and editing their interactions, and delivering them over limited-bandwidth connections can be prohibitive. This paper describes a framework, authoring tools, and Web-based run-time environment that support the creation of training scenarios using digital virtual humans and other reusable 3D components. By conforming to the Humanoid Animation (H-Anim), Extensible 3D, and ADL Shareable Content Object Reference Model specifications, these 3D components are designed to promote reuse and interoperability at several levels. Recently, these software tools were used to develop prototype lessons in foreign language and cultural familiarization for use at the Defense Language Institute Foreign Language Center. These lessons include simulations in which the student, taking the role of a US soldier, interacts with the local population of a foreign culture in authentic situations, using both English and the local dialect. Digital virtual humans are used to represent not only the soldier, and the role-playing actors; but also a mentor who reviews language and cultural learning points, and provides remediation. Using these prototype lessons as examples, we review the potential advantages of the technology in other training applications.     

Modelling experts' behavior with e-valUAM to measure computer science skills

In this work we present an evaluation method that focuses on experts' behavior instead of the traditional scores based just on the number of correct answers. The method presented here is especially suitable to measure the skills in Computer Science since this is a wide discipline very difficult to evaluate due to the many facts publicly available on the Internet. By using traditional evaluation tools, it is very difficult to measure the real knowledge of the users since they can correctly answer even without having acquired formal academic knowledge. To use this method, we have developed a test that can detect significant differences between standard users and experts in Computer Science. The test is applied by the e-valUAM application, which has been modified to store several parameters from the users' answers. By optimizing the parameters by a linear model, we have developed an equation that can be used to quantitatively compare the results of a single user with the results from the reference group of experts. This optimization is only possible because this group shows good stability and gives statistically different results compared to the other groups. The scores achieved with our method can be used to predict the formal knowledge of the users and modify their training when needed.     

Construction of a learning environment supporting learners’ reflection: A case of information seeking on the Web

In this study, we design a learning environment that supports reflective activities for information seeking on the Web and evaluate its educational effects. The features of this design are: (1) to visualize the learners’ search processes as described, based on a cognitive schema, (2) to support two types of reflective activities, such as “reflection-in-action” and “reflection-on-action”, and (3) to facilitate reflective activities by comparing their own search processes to other learners’ search processes. We have conducted an experiment to investigate the effects of our design. The experimental results confirm that: (1) the participants’ search performance in the instructional group supported by our instructional design improved effectively than in the control group; (2) they changed their ideas about important activities when seeking information on the Web, and (3) they activated their search cycles more than the control group did.     

A distributed system for learning programming on-line

Several Web-based on-line judges or on-line programming trainers have been developed in order to allow students to train their programming skills. However, their pedagogical functionalities in the learning of programming have not been clearly defined. EduJudge is a project which aims to integrate the “UVA On-line Judge”, an existing on-line programming trainer with an important number of problems and users, into an effective educational environment consisting of the e-learning platform Moodle and the competitive learning tool QUESTOURnament. The result is the EduJudge system which allows teachers to apply different pedagogical approaches using a proven e-learning platform, makes problems easy to search through an effective search engine, and provides an automated evaluation of the solutions submitted to these problems. The final objective is to provide new learning strategies to motivate students and present programming as an easy and attractive challenge. EduJudge has been tried and tested in three algorithms and programming courses in three different Engineering degrees. The students’ motivation and satisfaction levels were analysed alongside the effects of the EduJudge system on students’ academic outcomes. Results indicate that both students and teachers found that among other multiple benefits the EduJudge system facilitates the learning process. Furthermore, the experiment also showed an improvement in students’ academic outcomes. It must be noted that the students’ level of satisfaction did not depend on their computer skills or their gender.     

Environmental design for a structured network learning society

Social interactions profoundly impact the learning processes of learners in traditional societies. The rapid rise of the Internet using population has been the establishment of numerous different styles of network communities. Network societies form when more Internet communities are established, but the basic form of a network society, especially a network learning society, remains unclear. In 1998, a group of Taiwanese researchers created a network learning society, named “EduCities”. Based on the experience of building this network learning society, the authors found that a structured network learning society architecture helps participants to coordinate and manage interaction processes. This study describes 10 basic elements involved in establishing a structured network learning society, i.e., participants, shared visions, devices, services, rules, relations, manners, learning domains, learning goals and learning activities. A structured network learning society environment, “EduCities”, was then implemented based on the structured network learning society concept. EduCities included numerous structural designs, including EduCity, EduTown, EduSisterTown, EduVillage, EduSisterVillage, EduCitizen, and EduHome were practiced. Participants in EduCities own their roles, managing power, learning goals, and social relations. The structured network learning society concept represents a prototype of an online learning society.     

Distance learning in advanced military education: Analysis of joint operations course in the Taiwan military

High-ranking officers require advanced military education in war tactics for future combat. However, line officers rarely have time to take such courses on campus. The conventional solution to this problem used to take the inefficient correspondence courses. Whereas Internet technologies progress, online course is the current trend for military training. However, the question is what distance learning methodology best suits such a proprietary learning purpose.This study presents a sequential process of developing distance learning courses in advanced military education. Further, the Petri-Net analytical approach is adopted to discover the essential interaction requirements of advanced military education delivered via Internet. This study developed a systematic method for designing e-learning systems according to specific requirements of target courses. The proposed approach starts by comparing on-campus programs with the existing e-learning systems to identify the steps required to transform the program into an e-learning system. After first outlining the pedagogy of the on-campus program, its proposed teaching flow through the Internet is then sketched. Finally, the Petri-Net model was used for in-depth analysis of the stages affecting the learning curve of the line office taking e-learning courses. The example of a “Joint Operations” AME course elaborated the presented approach. An e-learning system prototype was also designed accordingly. Lastly, an experiment was conducted to verify the efficiency of the presented approach.     

The toolbox for local and global plagiarism detection

Digital plagiarism is a problem for educators all over the world. There are many software tools on the market for uncovering digital plagiarism. Most of them can work only with text submissions. In this paper, we present a new architecture for a plagiarism detection tool that can work with many different kinds of digital submissions, from plain or formatted texts to audio podcasts. The open architecture is based on converting the digital submission into text form for processing by a plagiarism detection algorithm. To process non-text submissions, the system is extended with the appropriate converter. Such an open architecture makes the anti-plagiarism toolbox universal and easily adaptable for processing virtually any kind of digital submissions. This paper describes a software prototype based on the proposed architecture and presents the results of its implementation on a large archive of student papers.     

Using a concept map knowledge management system to enhance the learning of biology

This study was undertaken with the goal of developing a Concept Map Knowledge Management System (CMKMS) for use as a tool in observing change in a student's understanding of biology concepts over time. The CMKMS should be useful in assessing the extent of a student's knowledge and in revealing their unique thought processes. The study was concerned not only with a student's self-evaluation of learning but also with their level of satisfaction after using the CMKMS. The CMKMS combines the diagnosis of concept mapping with the style of thought processing, and promotes teaching activities step-by-step, in order to promote effective student learning. In analyzing the factors that influence the effect of teaching, the teacher can determine a student's knowledge structure and highlight misconceptions by inspecting the concept maps and logs. The results of this study show that knowledge management involved in computer-aided instruction in the teaching of biology had a positive influence on learning effectiveness. The CMKMS also was useful in promoting the student's thought processing, creativity, and ability to judge.