From paper-and-pen annotations to artefact-based mobile learning

Making annotations on books and learning materials is part of students' everyday life. Although there are many computer-based annotation systems, many people prefer to print digital documents and make annotations on physical paper. We introduce a classification scheme for computer-based annotation systems and provide an overview of different systems using this scheme. During our investigations, we identified several shortcomings in existing annotation systems: limitation to single document formats, no capturing of relations to reflect lateral reading, loss of context information an annotation was created in and limited sharing capabilities among collaborative users. To overcome these shortcomings, we introduce the concepts of Anchor Points , Artefacts and Spaces . Anchor Points define locations in arbitrary documents. Artefacts capture annotations on multiple Anchor Points in several documents, allowing students to map their current working context that typically contains several open documents and a relationship between them. Spaces are repositories for the learning materials and annotations that are automatically replicated among collaborative mobile users.     

Towards the semantic and context-aware management of mobile multimedia

Users of mobile devices can nowadays easily create large quantities of mobile multimedia documents tracing significant events attended, places visited or, simply, moments of their everyday life. However, they face the challenge of organizing these documents in order to facilitate searching through them at a later time and sharing them with other users. We propose using context awareness and semantic technologies in order to improve and facilitate the organization, annotation, retrieval and sharing of personal mobile multimedia documents. Our approach combines metadata extracted and enriched automatically from the users’ context with annotations provided manually by the users and with annotations inferred by applying user-defined rules to context features. These new contextual metadata are integrated into the processes of annotation, sharing and keyword-based retrieval.     

The LEMO annotation framework: weaving multimedia annotations with the web

Cultural institutions and museums have realized that annotations contribute valuable metadata for search and retrieval, which in turn can increase the visibility of the digital items they expose via their digital library systems. By exploiting annotations created by others, visitors can discover content they would not have found otherwise, which implies that annotations must be accessible and processable for humans and machines. Currently, however, there exists no widely adopted annotation standard that goes beyond specific media types. Most institutions build their own in-house annotation solution and employ proprietary annotation models, which are not interoperable with those of other systems. As a result, annotation data are usually stored in closed data silos and visible and processable only within the scope of a certain annotation system. As the main contribution of this paper, we present the LEMO Annotation Framework. It (1) provides a uniform annotation model for multimedia contents and various types of annotations, (2) can address fragments of various content-types in a uniform, interoperable manner and (3) pulls annotations out of closed data silos and makes them available as interoperable, dereferencable Web resources. With the LEMO Annotation Framework annotations become part of the Web and can be processed, linked, and referenced by other services. This in turn leads to even higher visibility and increases the potential value of annotations.     

How Software Developers Use Tagging to Support Reminding and Refinding

Developers frequently add annotations to source code to help them remember pertinent information and mark locations of interest for future investigation. Finding and refinding these notes is a form of navigation that is integral to software maintenance. Although there is some tool support in modern development environments for authoring and navigating these comments, we have observed that these annotations often fail to remind and are sometimes difficult to find by the programmer. To address these shortcomings, we have designed a new approach for software navigation called Tags for Software Engineering Activities (TagSEA). TagSEA combines the notion of waypointing (a mechanism for marking locations in spatial navigation) with social tagging to support programmers in defining semantically rich annotations to source code comments. The tool provides support for creating, editing, navigating, and managing these annotations. We present the results from two empirical studies, where we observed and then analyzed how professional programmers used source code annotations to support their development activities over 24 months. Our findings indicate that the addition of semantic information to annotations can improve their value. We also provide suggestions on how annotation tools in general may be improved.     

A historical and contemporary study on annotations to derive key features for systems design

This paper provides a comprehensive study on annotations by defining their contours and complexity. This work adds a new complementary approach to the usual case and user studies, and also investigates history in order to benefit from previous knowledge and our cultural heritage. This study emphasizes an aspect which has never previously been taken into account: the temporal dimension involved in annotations. Moreover, it discusses both the notion of hypertext between documents and annotations and the idea of annotations as context for documents. The study gives the necessary historical and cultural background to derive a set of key features of annotations that must be taken into account when designing systems that have to support the management of digital annotations on digital contents.     

Extended 3D annotations as a new mechanism to explicitly communicate geometric design intent and increase CAD model reusability

A successful implementation of the Model-Based Enterprise concept (MBE) requires maximizing the potential benefits of annotated 3D models. The foundations of the MBE model are established by digital product definition data practices, which are currently regulated by standards such as ASME Y14.41-2003 and ISO 16792:2006. At the center of the MBE concept is the notion of CAD model reusability, which relies on the idea that 3D CAD models can be reused both throughout the entire product lifecycle and as a starting point for future development of new products. In this context, a critical aspect of CAD model reuse is the proper identification and understanding of the geometric design intent that is usually expressed implicitly within the CAD model.In this work, we present a method to communicate geometric design intent explicitly by overloading and extending the scope of the current annotation instruments available in the MBE approach. We propose a new broader type of model annotation that we call “extended annotation”, where design information is represented both internally within the 3D model and externally, on a separate repository. This structure naturally demands additional mechanisms to support the interaction of users with the information. In order to manage the information stored in these extended annotations effectively, we implemented an annotation manager that automatically synchronizes the dual representation of the annotations. To reduce the visual clutter in the 3D model, the software provides powerful filtering, editing, and visualization capabilities, giving users complete control of the information stored in the model. Finally, a study was conducted with 60 participants to evaluate the performance of the proposed model and the usability of the annotation manager. Results show a statistically significant benefit of using the extended annotation system, suggesting the use of this model as a valuable approach to improve design intent communication.     

Scalable search-based image annotation

With the popularity of digital cameras, more and more people have accumulated considerable digital images on their personal devices. As a result, there are increasing needs to effectively search these personal images. Automatic image annotation may serve the goal, for the annotated keywords could facilitate the search processes. Although many image annotation methods have been proposed in recent years, their effectiveness on arbitrary personal images is constrained by their limited scalability, i.e. limited lexicon of small-scale training set. To be scalable, we propose a search-based image annotation algorithm that is analogous to information retrieval. First, content-based image retrieval technology is used to retrieve a set of visually similar images from a large-scale Web image set. Second, a text-based keyword search technique is used to obtain a ranked list of candidate annotations for each retrieved image. Third, a fusion algorithm is used to combine the ranked lists into a final candidate annotation list. Finally, the candidate annotations are re-ranked using Random Walk with Restarts and only the top ones are reserved as the final annotations. The application of both efficient search techniques and Web-scale image set guarantees the scalability of the proposed algorithm. Moreover, we provide an annotation rejection scheme to point out the images that our annotation system cannot handle well. Experimental results on U. Washington dataset show not only the effectiveness and efficiency of the proposed algorithm but also the advantage of image retrieval using annotation results over that using visual features.     

CASAM: collaborative human-machine annotation of multimedia

The CASAM multimedia annotation system implements a model of cooperative annotation between a human annotator and automated components. The aim is that they work asynchronously but together. The system focuses upon the areas where automated recognition and reasoning are most effective and the user is able to work in the areas where their unique skills are required. The system’s reasoning is influenced by the annotations provided by the user and, similarly, the user can see the system’s work and modify and, implicitly, direct it. The CASAM system interacts with the user by providing a window onto the current state of annotation, and by generating requests for information which are important for the final annotation or to constrain its reasoning. The user can modify the annotation, respond to requests and also add their own annotations. The objective is that the human annotator’s time is used more effectively and that the result is an annotation that is both of higher quality and produced more quickly. This can be especially important in circumstances where the annotator has a very restricted amount of time in which to annotate the document. In this paper we describe our prototype system. We expand upon the techniques used for automatically analysing the multimedia document, for reasoning over the annotations generated and for the generation of an effective interaction with the end-user. We also present the results of evaluations undertaken with media professionals in order to validate the approach and gain feedback to drive further research.     

Towards effective video annotation: An approach to automatically link notes with video content

The characteristics of annotations, such as highlighting, context-based notes, and organization are difficult to translate from the traditional paper-based medium to the digital format. An added challenge is how to facilitate annotations on a digital video in a collaborative distance learning environment. To explore issues in video annotation, we developed a tool called Interactive Shared Education Environment (ISEE). ISEE automatically generates hyperlinked timestamps, which we called Smartlinks, to associate the notes with their video contents. A usability study with 59 participants, following up by a small-scale eye-tracking study, was conducted to explore users’ video note-taking behaviors and to examine the effect of the new Smartlink design. Our results showed that participants with Smartlink took fewer notes, focused less on video controls and more on video content than those without Smartlink. We believe the main benefit of Smartlink is that it may offload non-learning related cognitive loads and allow users to take better notes. Findings from this study on users’ video annotation behaviors shed light on the future design of video annotation systems in both individual and collaborative environments.     

Decorating the cloud: enabling annotation management in MapReduce

Data curation and annotation are indispensable mechanisms to a wide range of applications for capturing various types of metadata information. This metadata not only increases the data’s credibility and merit, and allows end users and applications to make more informed decisions, but also enables advanced processing over the data that is not feasible otherwise. That is why annotation management has been extensively studied in the context of scientific repositories, web documents, and relational database systems. In this paper, we make the case that cloud-based applications that rely on the emerging Hadoop infrastructure are also in need for data curation and annotation and that the presence of such mechanisms in Hadoop would bring value-added capabilities to these applications. We propose the “CloudNotes” system, a full-fledged MapReduce-based annotation management engine. CloudNotes addresses several new challenges to annotation management including: (1) scalable and distributed processing of annotations over large clusters, (2) propagation of annotations under the MapReduce’s blackbox execution model, and (3) annotation-driven optimizations ranging from proactive prefetching and colocation of annotations, annotation-aware task scheduling, novel shared execution strategies among the annotation jobs, and concurrency control mechanisms for annotation management. These challenges have not been addressed or explored before by the state-of-art technologies. CloudNotes is built on top of the open-source Hadoop/HDFS infrastructure and experimentally evaluated to demonstrate the practicality and scalability of its features, and the effectiveness of its optimizations under large workloads.     

Annotation technologies: A software and research review

This article describes a range of currently available and developing technologies for creating and presenting annotations, glosses, and other comments on digital documents. The potential applications of these tools for providing feedback to student writers, supporting extended group discussions around digital texts, and facilitating research and reading-to-write tasks are discussed. Different software programs are compared and evaluated and composition researchers are urged to engage in research that will influence the design of future annotation technologies.     

Enhancement of digital reading performance by using a novel web-based collaborative reading annotation system with two quality annotation filtering mechanisms

Collaboratively annotating digital texts allows learners to add valued information, share ideas, and create knowledge. However, excessive annotations and poor-quality annotations in a digital text may cause information overload and divert attention from the main content. The increased cognitive load ultimately reduces the effectiveness of collaborative annotations in promoting reading comprehension. Thus, this work develops a web-based collaborative reading annotation system (WCRAS-TQAFM) with two quality annotation filtering mechanisms—high-grade and master annotation filters—to promote the reading performance of learners. Ninety-seven students from three classes of a senior high school in Taiwan were invited to participate in an 80-min reading activity in which individual readers use WCRAS with or without annotation filters. Analytical results indicate that digital reading performance is significantly better in readers who use the high-grade annotation filter compared to those who read all annotations. Moreover, the high-grade annotation filter can enhance the reading comprehension of learners in all considered question types (i.e., recall, main idea, inference, and application). Also, the Cohen’s kappa statistics was used for assessing whether the annotation selected by the high-grade annotation filter is in agreement with the annotations selected by a domain expert. The statistic results indicate that the proposed high-grade annotation filter is valid to some degree. Finally, neither of the proposed quality annotation filtering approaches significantly reduces cognitive load.     

Automatic ontology-based knowledge extraction from Web documents

To bring the Semantic Web to life and provide advanced knowledge services, we need efficient ways to access and extract knowledge from Web documents. Although Web page annotations could facilitate such knowledge gathering, annotations are rare and will probably never be rich or detailed enough to cover all the knowledge these documents contain. Manual annotation is impractical and unscalable, and automatic annotation tools remain largely undeveloped. Specialized knowledge services therefore require tools that can search and extract specific knowledge directly from unstructured text on the Web, guided by an ontology that details what type of knowledge to harvest. An ontology uses concepts and relations to classify domain knowledge. Other researchers have used ontologies to support knowledge extraction, but few have explored their full potential in this domain. The paper considers the Artequakt project which links a knowledge extraction tool with an ontology to achieve continuous knowledge support and guide information extraction. The extraction tool searches online documents and extracts knowledge that matches the given classification structure. It provides this knowledge in a machine-readable format that will be automatically maintained in a knowledge base (KB). Knowledge extraction is further enhanced using a lexicon-based term expansion mechanism that provides extended ontology terminology.     

A Web 2.0-based collaborative annotation system for enhancing knowledge sharing in collaborative learning environments

A limitation of current Web-based collaborative learning is the restricted ability of students to create and share individual annotations with annotated documents. Applying Web 2.0 collaborative annotation systems and analyzing students’ annotation behavior has attracted attention to improve collaborative learning. This study designed a personalized annotation management system 2.0 (PAMS 2.0) for managing, sharing, and reusing individual and collaborative annotations as well as providing a shared mechanism for discussion about shared annotations among multiple users.     

Tailored semantic annotation for semantic search

This paper presents a novel method for semantic annotation and search of a target corpus using several knowledge resources (KRs). This method relies on a formal statistical framework in which KR concepts and corpus documents are homogeneously represented using statistical language models. Under this framework, we can perform all the necessary operations for an efficient and effective semantic annotation of the corpus. Firstly, we propose a coarse tailoring of the KRs w.r.t the target corpus with the main goal of reducing the ambiguity of the annotations and their computational overhead. Then, we propose the generation of concept profiles, which allow measuring the semantic overlap of the KRs as well as performing a finer tailoring of them. Finally, we propose how to semantically represent documents and queries in terms of the KRs concepts and the statistical framework to perform semantic search. Experiments have been carried out with a corpus about web resources which includes several Life Sciences catalogs and Wikipedia pages related to web resources in general (e.g., databases, tools, services, etc.). Results demonstrate that the proposed method is more effective and efficient than state-of-the-art methods relying on either context-free annotation or keyword-based search.     

A Sequence-Based Object-Oriented Model for Video Databases

Structuration, annotation and composition are amidst the most crucial modeling issues that video editing and querying in the context of a database entail. In this paper, we propose a sequence-based, object-oriented data model that addresses them in an unified, yet orthogonal way. Thanks to this orthogonality, the interactions between these three aspects are properly captured, i.e., annotations may be attached to any level of video structuration, and all the composition operators preserve the structurations and annotations of the argument videos. We also propose to query both the structuration and the annotations of videos using an extension of ODMG's OQL which integrates a set of algebraic operators on sequences. The overall proposal is formalized and implemented on top of an object-oriented DBMS.     

An annotation management system for relational databases

We present an annotation management system for relational databases. In this system, every piece of data in a relation is assumed to have zero or more annotations associated with it and annotations are propagated along, from the source to the output, as data is being transformed through a query. Such an annotation management system could be used for understanding the provenance (aka lineage) of data, who has seen or edited a piece of data or the quality of data, which are useful functionalities for applications that deal with integration of scientific and biological data. We present an extension, pSQL, of a fragment of SQL that has three different types of annotation propagation schemes, each useful for different purposes. The default scheme propagates annotations according to where data is copied from. The default-all scheme propagates annotations according to where data is copied from among all equivalent formulations of a given query. The custom scheme allows a user to specify how annotations should propagate. We present a storage scheme for the annotations and describe algorithms for translating a pSQL query under each propagation scheme into one or more SQL queries that would correctly retrieve the relevant annotations according to the specified propagation scheme. For the default-all scheme, we also show how we generate finitely many queries that can simulate the annotation propagation behavior of the set of all equivalent queries, which is possibly infinite. The algorithms are implemented and the feasibility of the system is demonstrated by a set of experiments that we have conducted.     

The perceived utility of standard ontologies in document management for specialized domains

A user-based document management system has been developed for small communities on the Web. The system is based on the free annotation of documents by users. A number of annotation support tools are used to suggest possible annotations, including suggesting terms from external ontologies. This paper outlines some evaluation data on how users actually interact with the system in annotating their document especially on the use of standard ontologies. Results indicate that although an established external taxonomy can be useful in proposing annotation terms, users appear to be very selective in their use of the terms proposed and to have little interest in adhering to the particular hierarchical structure provided.