Semantic annotation of narrative media objects

This paper addresses the annotation of the narrative features of media objects. Based on a relevant narratological and computational background, we introduce an ontology?Cbased model called Drammar, an annotation schema for the narrative features of media objects based on Drammar and a software tool, Cinematic, for annotating these objects and validating the annotation. Annotated media objects can also be automatically edited into sequences, with the twofold goal of testing the validity of the annotation??through the reconstruction of the baseline sequence??and exploring the possibility of alternative sequences. The software tool encodes both the narrative model and the annotation itself in ontological format, and relies on external ontologies for representing world knowledge and limit the arbitrariness of the annotation. The paper opens the way to the design of a general annotation schema for narrative multimedia with the long?Cterm goal of building large corpora of annotated video material and of bridging the gap between the low?Clevel signal analysis and the high?Clevel semantic representation of the narrative content of the media objects. Finally, the paper illustrates a few projects elaborated with the Drammar annotation and the Cinematic tool, with purposes of artistic research and cross?Cmedia analysis, that provide an empirical validation of the annotation process.     

Pointwise and pairwise clothing annotation: combining features from social media

In this paper, we present effective algorithms to automatically annotate clothes from social media data, such as Facebook and Instagram. Clothing annotation can be informally stated as recognizing, as accurately as possible, the garment items appearing in the query photo. This task brings huge opportunities for recommender and e-commerce systems, such as capturing new fashion trends based on which clothes have been used more recently. It also poses interesting challenges for existing vision and recognition algorithms, such as distinguishing between similar but different types of clothes or identifying a pattern of a cloth even if it has different colors and shapes. We formulate the annotation task as a multi-label and multi-modal classification problem: (i) both image and textual content (i.e., tags about the image) are available for learning classifiers, (ii) the classifiers must recognize a set of labels (i.e., a set of garment items), and (iii) the decision on which labels to assign to the query photo comes from a set of instances that is used to build a function, which separates labels that should be assigned to the query photo, from those that should not be assigned. Using this configuration, we propose two approaches: (i) the pointwise one, called MMCA, which receives a single image as input, and (ii) a multi-instance classification, called M3CA, also known as pairwise approach, which uses pair of images to create the classifiers. We conducted a systematic evaluation of the proposed algorithms using everyday photos collected from two major fashion-related social media, namely pose.com and chictopia.com. Our results show that the proposed approaches provide improvements when compared to popular first choice multi-label, multi-modal, multi-instance algorithms that range from 20 % to 30 % in terms of accuracy.     

The xLiMe system: Cross-lingual and cross-modal semantic annotation,search and recommendation over live-TV,news and social media streams

Modern Web search engines still have many limitations: search terms are not disambiguated, search terms in one query cannot be in different languages, the retrieved media items have to be in the same language as the search terms and search results are not integrated across a live stream of different media channels, including TV, online news and social media. The system described in this paper enables all of this by combining a media stream processing architecture with cross-lingual and cross-modal semantic annotation, search and recommendation. All those components were developed in the xLiMe project.     

Inter-operability and reusability: the science of annotation

Annotating linguistic data has become a major field of interest, both for supplying the necessary data for machine learning approaches to NLP applications, and as a research issue in its own right. This comprises issues of technical formats, tools, and methodologies of annotation. We provide a brief overview of these notions and then introduce the papers assembled in this special issue.     

Orthogonal moments based on exponent functions: Exponent-Fourier moments

In this paper, we propose a new set of orthogonal moments based on Exponent functions, named Exponent-Fourier moments (EFMs), which are suitable for image analysis and rotation invariant pattern recognition. Compared with Zernike polynomials of the same degree, the new radial functions have more zeros, and these zeros are evenly distributed, this property make EFMs have strong ability in describing image. Unlike Zernike moments, the kernel of computation of EFMs is extremely simple. Theoretical and experimental results show that Exponent-Fourier moments perform very well in terms of image reconstruction capability and invariant recognition accuracy in noise-free, noisy and smooth distortion conditions. The Exponent-Fourier moments can be thought of as generalized orthogonal complex moments.     

Manual annotation and automatic image processing of multimodal emotional behaviors: validating the annotation of TV interviews

There have been a lot of psychological researches on emotion and nonverbal communication. Yet, these studies were based mostly on acted basic emotions. This paper explores how manual annotation and image processing can cooperate towards the representation of spontaneous emotional behavior in low-resolution videos from TV. We describe a corpus of TV interviews and the manual annotations that have been defined. We explain the image-processing algorithms that have been designed for the automatic estimation of movement quantity. Finally, we explore how image processing can be used for the validation of manual annotations.     

Multiple annotation for biodiversity: developing an annotation framework among biology,linguistics and text technology

Language Resources and Evaluation - Biodiversity information is contained in countless digitized and unprocessed scholarly texts. Although automated extraction of these data has been gaining...     

CASAM: collaborative human-machine annotation of multimedia

The CASAM multimedia annotation system implements a model of cooperative annotation between a human annotator and automated components. The aim is that they work asynchronously but together. The system focuses upon the areas where automated recognition and reasoning are most effective and the user is able to work in the areas where their unique skills are required. The system’s reasoning is influenced by the annotations provided by the user and, similarly, the user can see the system’s work and modify and, implicitly, direct it. The CASAM system interacts with the user by providing a window onto the current state of annotation, and by generating requests for information which are important for the final annotation or to constrain its reasoning. The user can modify the annotation, respond to requests and also add their own annotations. The objective is that the human annotator’s time is used more effectively and that the result is an annotation that is both of higher quality and produced more quickly. This can be especially important in circumstances where the annotator has a very restricted amount of time in which to annotate the document. In this paper we describe our prototype system. We expand upon the techniques used for automatically analysing the multimedia document, for reasoning over the annotations generated and for the generation of an effective interaction with the end-user. We also present the results of evaluations undertaken with media professionals in order to validate the approach and gain feedback to drive further research.     

FARM: file annotation and retrieval on mobile devices

As the technology evolves, mobile phones and portable devices are equipped with advanced features like built-in cameras, audio/video recordings, and other applications. When a user starts utilizing such features and applications, usually a large number of files are generated, and as this number grows, it becomes a challenging task to find the requested files with limited user-interface capabilities. This article presents FARM, a framework for file annotation and retrieval on resource-limited mobile devices like mobile phones, PDAs, and other MIDP compliant devices. Once the annotation process is completed, user can easily retrieve the files on the device itself or on in a networked environment where a large number of mobile nodes are involved. The metadata for annotation is gathered from the underlying file system and stored locally, which gives this framework a two-fold gain. Firstly, this framework does not require any common repository and hence, does not require communication medium to store and retrieve metadata. Secondly, searches are performed in a distributed fashion when more than one device is searched. The proposed framework has been implemented with a full featured application, and experimental results show that files can be retrieved efficiently and accurately with annotated data.     

SpatialML: annotation scheme,resources, and evaluation

SpatialML is an annotation scheme for marking up references to places in natural language. It covers both named and nominal references to places, grounding them where possible with geo-coordinates, and characterizes relationships among places in terms of a region calculus. A freely available annotation editor has been developed for SpatialML, along with several annotated corpora. Inter-annotator agreement on SpatialML extents is 91.3 F-measure on a corpus of SpatialML-annotated ACE documents released by the Linguistic Data Consortium. Disambiguation agreement on geo-coordinates on ACE is 87.93 F-measure. An automatic tagger for SpatialML extents scores 86.9 F on ACE, while a disambiguator scores 93.0 F on it. Results are also presented for two other corpora. In adapting the extent tagger to new domains, merging the training data from the ACE corpus with annotated data in the new domain provides the best performance.     

Ultra-arithmetic II: intervals of polynomials

In Part 1 of this paper (Function Data Types) we developed ultra-arithmetic. a calculus for functions which is performable on a digital computer. Here in analogy with the notion of interval arithmetic for intervals of reals we begin the development of an interval arithmetic for functions. The operations of addition, subtraction, multiplication, division, integration and differentiation for intervals of polynomials are defined and studied. In certain cases simplified isotonal approximations to the resultant interval as well as error analyses are also given.     

Semantic annotation of soccer videos: automatic highlights identification

Automatic semantic annotation of video streams allows both to extract significant clips for production logging and to index video streams for posterity logging. Automatic annotation for production logging is particularly demanding, as it is applied to non-edited video streams and must rely only on visual information. Moreover, annotation must be computed in quasi real-time. In this paper, we present a system that performs automatic annotation of the principal highlights in soccer video, suited for both production and posterity logging. The knowledge of the soccer domain is encoded into a set of finite state machines, each of which models a specific highlight. Highlight detection exploits visual cues that are estimated from the video stream, and particularly, ball motion, the currently framed playfield zone, players’ positions and colors of players’ uniforms. The highlight models are checked against the current observations, using a model checking algorithm. The system has been developed within the EU ASSAVID project.     

Semantic hierarchies for image annotation: A survey

In this survey, we argue that using structured vocabularies is capital to the success of image annotation. We analyze literature on image annotation uses and user needs, and we stress the need for automatic annotation. We briefly expose the difficulties posed to machines for this task and how it relates to controlled vocabularies. We survey contributions in the field showing how structures are introduced. First we present studies that use unstructured vocabulary, focusing on those introducing links between categories or between features. Then we review work using structured vocabularies as an input and analyze how the structure is exploited.     

Research on Chinese negation and speculation: corpus annotation and identification

Identifying negative or speculative narrative fragments from facts is crucial for deep understanding on natural language processing (NLP). In this paper, we firstly construct a Chinese corpus which consists of three sub-corpora from different resources. We also present a general framework for Chinese negation and speculation identification. In our method, first, we propose a feature-based sequence labeling model to detect the negative or speculative cues. In addition, a cross-lingual cue expansion strategy is proposed to increase the coverage in cue detection. On this basis, this paper presents a new syntactic structure-based framework to identify the linguistic scope of a negative or speculative cue, instead of the traditional chunking-based framework. Experimental results justify the usefulness of our Chinese corpus and the appropriateness of our syntactic structure-based framework which has showed significant improvement over the state-of-the-art on Chinese negation and speculation identification.     

Automatic summarisation and annotation of microarray data

The study of biological processes within cells is based on the measurement of the activity of different molecules, in particular genes and proteins whose activities are strictly related. The activity of genes is measured through a systematic investigation carried out by microarrays. Such technology enables the investigation of all the genes of an organism in a single experiment, encoding meaningful biological information. Nevertheless, the preprocessing of raw microarray data needs automatic tools that standardise such phase in order to: (a) avoiding errors in analysis phases, and (b) making comparable the results of different laboratories. The preprocessing problem is as much relevant as considering results obtained from analysis platforms of different vendors. Nevertheless, there is currently a lack of tools that allow to manage and preprocess multivendor dataset. This paper presents a software platform (called GSAT, General-purpose Summarisation and Annotation Tool) able to manage and preprocess microarray data. The GSAT allows the summarisation, normalisation and annotation of multivendor microarray data, using web services technology. First experiments and results on Affymetrix data samples are also discussed. GSAT is available online at as a standalone application or as a plugin of the TMEV microarray data analysis platform.     

Online annotation – Research and practices

Annotation can be a valuable exercise when trying to understand new information. The technique can be used to create a ‘condensed’ version of the original information for later review and to add additional information into the existing document. The growth in web-based learning materials and information sources has created requirement for systems that allow annotations to be attached to these new sources and, potentially, shared with other learners. This paper discusses annotation in an educational context and introduces some of the web annotation systems currently available. It also provides an overview of the development of a new system, eLAWS, by the authors, based upon the Web Service architecture. Finally, the paper provides suggestions for the future development of e-Learning Annotation tools.