From paper-and-pen annotations to artefact-based mobile learning

Making annotations on books and learning materials is part of students' everyday life. Although there are many computer-based annotation systems, many people prefer to print digital documents and make annotations on physical paper. We introduce a classification scheme for computer-based annotation systems and provide an overview of different systems using this scheme. During our investigations, we identified several shortcomings in existing annotation systems: limitation to single document formats, no capturing of relations to reflect lateral reading, loss of context information an annotation was created in and limited sharing capabilities among collaborative users. To overcome these shortcomings, we introduce the concepts of Anchor Points , Artefacts and Spaces . Anchor Points define locations in arbitrary documents. Artefacts capture annotations on multiple Anchor Points in several documents, allowing students to map their current working context that typically contains several open documents and a relationship between them. Spaces are repositories for the learning materials and annotations that are automatically replicated among collaborative mobile users.     

Semi-automatic photo annotation strategies using event based clustering and clothing based person recognition

Managing a large number of digital photos is a challenging task for casual users. Personal photos often don’t have rich metadata, or additional information associated with them. However, available metadata can play a crucial role in managing photos. Labeling the semantic content of photos (i.e., annotating them), can increase the amount of metadata and facilitate efficient management. However, manual annotation is tedious and labor intensive while automatic metadata extraction techniques often generate inaccurate and irrelevant results. This paper describes a semi-automatic annotation strategy that takes advantage of human and computer strengths. The semi-automatic approach enables users to efficiently update automatically obtained metadata interactively and incrementally. Even though automatically identified metadata are compromised with inaccurate recognition errors, the process of correcting inaccurate information can be faster and easier than manually adding new metadata from scratch. In this paper, we introduce two photo clustering algorithms for generating meaningful photo groups: (1) Hierarchical event clustering; and (2) Clothing based person recognition, which assumes that people who wear similar clothing and appear in photos taken in one day are very likely to be the same person. To explore our semi-automatic strategies, we designed and implemented a prototype called SAPHARI (Semi-Automatic PHoto Annotation and Recognition Interface). The prototype provides an annotation framework which focuses on making bulk annotations on automatically identified photo groups. The prototype automatically creates photo clusters based on events, people, and file metadata so that users can easily bulk annotation photos. We performed a series of user studies to investigate the effectiveness and usability of the semi-automatic annotation techniques when applied to personal photo collections. The results show that users were able to make annotations significantly faster with event clustering using SAPHARI. We also found that users clearly preferred the semi-automatic approaches.     

Bridging the gap between tagging and querying vocabularies: Analyses and applications for enhancing multimedia IR

Collaborative tagging has become an increasingly popular means for sharing and organizing Web resources, leading to a huge amount of user-generated metadata. These annotations represent quite a few different aspects of the resources they are attached to, but it is not obvious which characteristics of the objects are predominantly described. The usefulness of these tags for finding/re-finding the annotated resources is also not completely clear. Several studies have started to investigate these issues, however only by focusing on a single type of tagging system or resource. We study this problem across multiple domains and resource types and identify the gaps between the tag space and the querying vocabulary. Based on the findings of this analysis, we then try to bridge the identified gaps, focusing in particular on multimedia resources. We focus on the two scenarios of music and picture resources and develop algorithms, which identify usage (theme) and opinion (mood) characteristics of the items. The mood and theme labels our algorithms infer are recommended to the users, in order to support them during the annotation process. The evaluation of the proposed methods against user judgements, as well as against expert ground truth reveal the high quality of our recommended annotations and provide insights into possible extensions for music and picture tagging systems to support retrieval.     

Semantic representation of multimedia content: Knowledge representation and semantic indexing

In this paper we present a framework for unified, personalized access to heterogeneous multimedia content in distributed repositories. Focusing on semantic analysis of multimedia documents, metadata, user queries and user profiles, it contributes to the bridging of the gap between the semantic nature of user queries and raw multimedia documents. The proposed approach utilizes as input visual content analysis results, as well as analyzes and exploits associated textual annotation, in order to extract the underlying semantics, construct a semantic index and classify documents to topics, based on a unified knowledge and semantics representation model. It may then accept user queries, and, carrying out semantic interpretation and expansion, retrieve documents from the index and rank them according to user preferences, similarly to text retrieval. All processes are based on a novel semantic processing methodology, employing fuzzy algebra and principles of taxonomic knowledge representation. The first part of this work presented in this paper deals with data and knowledge models, manipulation of multimedia content annotations and semantic indexing, while the second part will continue on the use of the extracted semantic information for personalized retrieval.

The LEMO annotation framework: weaving multimedia annotations with the web

Cultural institutions and museums have realized that annotations contribute valuable metadata for search and retrieval, which in turn can increase the visibility of the digital items they expose via their digital library systems. By exploiting annotations created by others, visitors can discover content they would not have found otherwise, which implies that annotations must be accessible and processable for humans and machines. Currently, however, there exists no widely adopted annotation standard that goes beyond specific media types. Most institutions build their own in-house annotation solution and employ proprietary annotation models, which are not interoperable with those of other systems. As a result, annotation data are usually stored in closed data silos and visible and processable only within the scope of a certain annotation system. As the main contribution of this paper, we present the LEMO Annotation Framework. It (1) provides a uniform annotation model for multimedia contents and various types of annotations, (2) can address fragments of various content-types in a uniform, interoperable manner and (3) pulls annotations out of closed data silos and makes them available as interoperable, dereferencable Web resources. With the LEMO Annotation Framework annotations become part of the Web and can be processed, linked, and referenced by other services. This in turn leads to even higher visibility and increases the potential value of annotations.     

A Web 2.0-based collaborative annotation system for enhancing knowledge sharing in collaborative learning environments

A limitation of current Web-based collaborative learning is the restricted ability of students to create and share individual annotations with annotated documents. Applying Web 2.0 collaborative annotation systems and analyzing students’ annotation behavior has attracted attention to improve collaborative learning. This study designed a personalized annotation management system 2.0 (PAMS 2.0) for managing, sharing, and reusing individual and collaborative annotations as well as providing a shared mechanism for discussion about shared annotations among multiple users.     

Linked Data and multimedia: the state of affairs

Linked Data is a way of exposing and sharing data as resources on the Web and interlinking them with semantically related resources. In the last three years significant amounts of data have been generated, increasingly forming a globally connected, distributed data space. For multimedia content, metadata are a key factor for efficient management, organization, and retrieval. However, the relationship between multimedia and Linked Data has been rarely studied, leading to a lack of mutual awareness and, as a consequence thereof, technological deficiencies. This article introduces the basic concepts of Linked Data in the context of multimedia metadata, and discusses techniques to generate, expose, discover, and consume Linked Data. It shows that a large amount of data sources exist, which are ready to be exploited by multimedia applications. The benefit of Linked Data in two multimedia-related applications is discussed and open research issues are outlined with the goal of bringing the research fields of multimedia and Linked Data closer together in order to facilitate mutual benefit.     

Decorating the cloud: enabling annotation management in MapReduce

Data curation and annotation are indispensable mechanisms to a wide range of applications for capturing various types of metadata information. This metadata not only increases the data’s credibility and merit, and allows end users and applications to make more informed decisions, but also enables advanced processing over the data that is not feasible otherwise. That is why annotation management has been extensively studied in the context of scientific repositories, web documents, and relational database systems. In this paper, we make the case that cloud-based applications that rely on the emerging Hadoop infrastructure are also in need for data curation and annotation and that the presence of such mechanisms in Hadoop would bring value-added capabilities to these applications. We propose the “CloudNotes” system, a full-fledged MapReduce-based annotation management engine. CloudNotes addresses several new challenges to annotation management including: (1) scalable and distributed processing of annotations over large clusters, (2) propagation of annotations under the MapReduce’s blackbox execution model, and (3) annotation-driven optimizations ranging from proactive prefetching and colocation of annotations, annotation-aware task scheduling, novel shared execution strategies among the annotation jobs, and concurrency control mechanisms for annotation management. These challenges have not been addressed or explored before by the state-of-art technologies. CloudNotes is built on top of the open-source Hadoop/HDFS infrastructure and experimentally evaluated to demonstrate the practicality and scalability of its features, and the effectiveness of its optimizations under large workloads.     

Brahma: Browsing and Retrieval Architecture for Hierarchical Multimedia Annotation

Traditional browsing of large multimedia documents (e.g., video, audio) is primarily sequential. In the absence of an index structure browsing and searching for relevant information in a long video, audio or other multimedia document becomes difficult. Manual annotation can be used to mark various segments of such documents. Different segments can be combined to create new annotated segments, thus creating hierarchical annotation structures. Given the lack of structure in media data, it is natural for different users to have different views on the same media data. Therefore, different users can create different annotation structures. Users may also share some or all of each other's annotation structures. The annotation structure can be browsed or used to playback as a composed video consisting of different segments. Finally, the annotation structures can be manipulated dynamically by different users to alter views on a document. BRAHMA is a multimedia environment for browsing and retrieval of multimedia documents based on such hierarchical annotation structures.     

DanVideo: an MPEG-7 authoring and retrieval system for dance videos

A well-annotated dance media is an essential part of a nation’s identity, transcending cultural and language barriers. Many dance video archives suffer from problems concerning authoring and access, because of the complex spatio-temporal relationships that exist between the dancers in terms of movements of their body parts and the emotions expressed by them in a dance. This paper presents a system named DanVideo for semi-automatic authoring and access to dance archives. DanVideo provides methods of annotation and authoring and retrieval tools for choreographers, dancers, and students. We demonstrate how dance media can be semantically annotated and how this information can be used for the retrieval of the dance video semantics. In particular, DanVideo offers an MPEG-7 based semi-automatic authoring tool that takes dance video annotations generated by dance experts and produces MPEG-7 metadata. DanVideo also has a search engine that takes users’ queries and retrieves dance semantics from metadata arranged using tree-embedding technique and based on spatial, temporal and spatio-temporal features of dancers. The search engine also leverages a domain-specific ontology to process knowledge-based queries. We have assessed the dance-video queries and semantic annotations in terms of precision, recall, and fidelity.     

System architecture for semantic annotation and adaptation in content sharing environments

This paper describes a system architecture, which enables the automatic semantic annotation and adaptation of multimedia content in context-aware content sharing environments. The discussed architecture is the result of research done in the EU FP6 IST INTERMEDIA project. Generating a common vision on user-centric multimedia services in shared content environments to provide users with content personalized to their user preferences and usage environment is one of the objectives of the project. The work presented in this paper describes how media formats with their related metadata are automatically annotated and dynamically adapted. Based on the architecture, a full-featured demonstrator is built.     

Exploring Social Annotations with the Application to Web Page Recommendation

Collaborative social annotation systems allow users to record and share their original keywords or tag attachments to Web resources such as Web pages, photos, or videos. These annotations are a method for organizing and labeling information. They have the potential to help users navigate the Web and locate the needed resources. However, since annotations are posted by users under no central control, there exist problems such as spam and synonymous annotations. To efficiently use annotation information to facilitate knowledge discovery from the Web, it is advantageous if we organize social annotations from semantic perspective and embed them into algorithms for knowledge discovery. This inspires the Web page recommendation with annotations, in which users and Web pages are clustered so that semantically similar items can be related. In this paper we propose four graphic models which cluster users, Web pages and annotations and recommend Web pages for given users by assigning items to the right cluster first. The algorithms are then compared to the classical collaborative filtering recommendation method on a real-world data set. Our result indicates that the graphic models provide better recommendation performance and are robust to fit for the real applications.     

MCSOSA: multimedia content share using ontology and secure access agent in mobile cloud

Mobile cloud is not just a traditional cloud, but a concept of virtualization that has expanded into mobile technology. It provides access to the data created and used by a user and content service by cloud platform. A feature of mobile cloud is supported that is the convenience of multimedia content sharing by mobile devices. However, there is a problem of inaccuracy of information retrieval in the process of sharing as well as personal information leakage and service inability status due to the malicious access to the mobile terminal in the retrieval process. This paper suggests the model to which the protective technique of multimedia content retrieval & access in mobile cloud is applied. The model stores and manages the individually different forms of content, and constructs the multimedia ontology in order to enhance the reliability in mismatched problems occurring in the retrieval process, and also suggests the response technique to security vulnerability occurring in the content access.     

PhotoGeo: a photo digital library with spatial-temporal support and self-annotation

The recent popularity of digital cameras has posed a new problem: how to efficiently store and retrieve the very large number of digital photos captured and chaotically stored in multiple locations without any annotation. This paper proposes an infrastructure, called PhotoGeo, which aims at helping users with the people photo annotation, event photo annotation, storage and retrieval of personal digital photos. To achieve the desired objective, PhotoGeo uses new algorithms that make it possible to annotate photos with the key metadata to facilitate their retrieval, such as: the people who were shown in the photo (who); where it was captured (where); the date and time of capture (when); and the event that was captured. The paper concludes with a detailed evaluation of these algorithms.