Receiver-side semantic reasoning for digital TV personalization in the absence of return channels

Experience has proved that interactive applications delivered through Digital TV must provide personalized information to the viewers in order to be perceived as a valuable service. Due to the limited computational power of DTV receivers (either domestic set-top boxes or mobile devices), most of the existing systems have opted to place the personalization engines in dedicated servers, assuming that a return channel is always available for bidirectional communication. However, in a domain where most of the information is transmitted through broadcast, there are still many cases of intermittent, sporadic or null access to a return channel. In such situations, it is impossible for the servers to learn who is watching TV at the moment, and so the personalization features become unavailable. To solve this problem without sacrificing much personalization quality, this paper introduces solutions to run a downsized semantic reasoning process in the DTV receivers, supported by a pre-selection of material driven by audience stereotypes in the head-end. Evaluation results are presented to prove the feasibility of this approach, and also to assess the quality it achieves in comparison with previous ones.

Analyzing the efficacy of using digital ink devices in a learning environment

There has been increased interest on the impact of mobile devices such as PDAs and Tablet PCs in introducing new pedagogical approaches and active learning experiences. We propose an intelligent system that efficiently addresses the inherent subjectivity in student perception of note taking and information retrieval. We employ the idea of cross indexing the digital ink notes with matching electronic documents in the repository. Latent Semantic Indexing is used to perform document and page level indexing. Thus for each retrieved document, the user can go over to the relevant pages that match the query. Techniques to handle problems such as polysemy (multiple meanings of a word) in large databases, document folding and no match for query are discussed. We tested our system for its performance, usability and effectiveness in the learning process. The results from the exploratory studies reveal that the proposed system provides a highly enhanced student learning experience, thereby facilitating high test scores.

Creating ambient music spaces in real and virtual worlds

Sound and, specifically, music is a medium that is used for a wide range of purposes in different situations in very different ways. Ways for music selection and consumption range from completely passive, almost unnoticed perception of background sound environments to the very specific selection of a particular recording of a piece of music with a specific orchestra and conductor at a certain event. Different systems and interfaces exist for the broad range of needs in music consumption. Locating a particular recording is well supported by traditional search interfaces via metadata. Other interfaces support the automatic creation of playlists via artist or album selection, up to more artistic installations of sound environments that users can navigate through. In this paper we present a set of systems that support the creation of as well as the navigation in musical spaces, both in the real world as well as in virtual environments. We show common principles and point out further directions for a more direct coupling of the various spaces and interaction methods, creating ambient sound environments and providing organic interaction with music for different purposes.

Metadata-driven interactive web video assembly

The recent expansion of broadband Internet access led to an exponential increase of potential consumers of video on the Web. The huge success of video upload websites shows that the online world, with its virtually unlimited possibilities of active user participation, is an ideal complement to traditional consumption-only media like TV and DVD. It is evident that users are willing to interact with content-providing systems in order to get the content they desire. In parallel to these developments, innovative tools for producing interactive, non-linear audio-visual content are being created. They support the authoring process alongside management of media and metadata, enabling on-demand assembly of videos based on the consumer’s wishes. The quality of such a dynamic video remixing system mainly depends on the expressiveness of associated metadata. Eliminating the need for manual input as far as possible, we aim at designing a system which is able to automatically enrich its own media and metadata repositories continuously. Currently, video content remixing is available on the Web mostly in very basic forms. Most platforms offer upload and simple modification of content. Although several implementations exist, to the best of our knowledge no solution uses metadata to its full extent to dynamically render a video stream based on consumers’ wishes. With the research presented in this paper, we propose a novel concept to interactive video assembly on the Web. In this approach, consumers may describe the desired content using a set of domain-specific parameters. Based on the metadata the video clips are annotated with, the system chooses clips fitting the user criteria. They are aligned in an aesthetically pleasing manner while the user furthermore is able to interactively influence content selection during playback at any time. We use a practical example to clarify the concept and further outline what it takes to implement a suchlike system.

Computational linguistics for metadata building (CLiMB): using text mining for the automatic identification,categorization, and disambiguation of subject terms for image metadata

In this paper, we present a system using computational linguistic techniques to extract metadata for image access. We discuss the implementation, functionality and evaluation of an image catalogers’ toolkit, developed in the Computational Linguistics for Metadata Building (CLiMB) research project. We have tested components of the system, including phrase finding for the art and architecture domain, functional semantic labeling using machine learning, and disambiguation of terms in domain-specific text vis a vis a rich thesaurus of subject terms, geographic and artist names. We present specific results on disambiguation techniques and on the nature of the ambiguity problem given the thesaurus, resources, and domain-specific text resource, with a comparison of domain-general resources and text. Our primary user group for evaluation has been the cataloger expert with specific expertise in the fields of painting, sculpture, and vernacular and landscape architecture.

Automated music video generation using multi-level feature-based segmentation

We show how to create a music video automatically, using computable characteristics of the video and music to promote coherent matching. We analyze the flow of both music and video, and then segment them into sequences of near-uniform flow. We extract features from the both video and music segments, and then find matching pairs. The granularity of the matching process can be adapted by extending the segmentation process to several levels. Our approach drastically reduces the skill required to make simple music videos.

Semantic image classification using statistical local spatial relations model

In this paper, a statistical model called statistical local spatial relations (SLSR) is presented as a novel technique of a learning model with spatial and statistical information for semantic image classification. The model is inspired by probabilistic Latent Semantic Analysis (PLSA) for text mining. In text analysis, PLSA is used to discover topics in a corpus using the bag-of-word document representation. In SLSR, we treat image categories as topics, therefore an image containing instances of multiple categories can be modeled as a mixture of topics. More significantly, SLSR introduces spatial relation information as a factor which is not present in PLSA. SLSR has rotation, scale, translation and affine invariant properties and can solve partial occlusion problems. Using the Dirichlet process and variational Expectation-Maximization learning algorithm, SLSR is developed as an implementation of an image classification algorithm. SLSR uses an unsupervised process which can capture both spatial relations and statistical information simultaneously. The experiments are demonstrated on some standard data sets and show that the SLSR model is a promising model for semantic image classification problems.

Personalized retrieval of sports video based on multi-modal analysis and user preference acquisition

In this paper, we present a novel framework on personalized retrieval of sports video, which includes two research tasks: semantic annotation and user preference acquisition. For semantic annotation, web-casting texts which are corresponding to sports videos are firstly captured from the webpages using data region segmentation and labeling. Incorporating the text, we detect events in the sports video and generate video event clips. These video clips are annotated by the semantics extracted from web-casting texts and indexed in a sports video database. Based on the annotation, these video clips can be retrieved from different semantic attributes according to the user preference. For user preference acquisition, we utilize click-through data as a feedback from the user. Relevance feedback is applied on text annotation and visual features to infer the intention and interested points of the user. A user preference model is learned to re-rank the initial results. Experiments are conducted on broadcast soccer and basketball videos and show an encouraging performance of the proposed method.

Parallel neural networks for multimodal video genre classification

Improvements in digital technology have made possible the production and distribution of huge quantities of digital multimedia data. Tools for high-level multimedia documentation are becoming indispensable to efficiently access and retrieve desired content from such data. In this context, automatic genre classification provides a simple and effective solution to describe multimedia contents in a structured and well understandable way. We propose in this article a methodology for classifying the genre of television programmes. Features are extracted from four informative sources, which include visual-perceptual information (colour, texture and motion), structural information (shot length, shot distribution, shot rhythm, shot clusters duration and saturation), cognitive information (face properties, such as number, positions and dimensions) and aural information (transcribed text, sound characteristics). These features are used for training a parallel neural network system able to distinguish between seven video genres: football, cartoons, music, weather forecast, newscast, talk show and commercials. Experiments conducted on more than 100 h of audiovisual material confirm the effectiveness of the proposed method, which reaches a classification accuracy rate of 95%.

Crossing textual and visual content in different application scenarios

This paper deals with multimedia information access. We propose two new approaches for hybrid text-image information processing that can be straightforwardly generalized to the more general multimodal scenario. Both approaches fall in the trans-media pseudo-relevance feedback category. Our first method proposes using a mixture model of the aggregate components, considering them as a single relevance concept. In our second approach, we define trans-media similarities as an aggregation of monomodal similarities between the elements of the aggregate and the new multimodal object. We also introduce the monomodal similarity measures for text and images that serve as basic components for both proposed trans-media similarities. We show how one can frame a large variety of problem in order to address them with the proposed techniques: image annotation or captioning, text illustration and multimedia retrieval and clustering. Finally, we present how these methods can be integrated in two applications: a travel blog assistant system and a tool for browsing the Wikipedia taking into account the multimedia nature of its content.

Attributing semantics to personal photographs

A major bottleneck for the efficient management of personal photographic collections is the large gap between low-level image features and high-level semantic contents of images. This paper proposes and evaluates two methodologies for making appropriate (re)use of natural language photographic annotations for extracting references to people, location and objects and propagating any location references encountered to previously unannotated images. The evaluation identifies the strengths of each approach and shows extraction and propagation results with promising accuracy.

An RTP/RTCP based approach for multimedia group and inter-stream synchronization

Most multimedia group and inter-stream synchronization techniques define or use proprietary protocols with new control messages. Many multimedia applications have been developed using RTP/RTCP as the standard for transmission of multimedia streams over IP networks. Instead of defining a new protocol, we propose the use of RTP/RTCP to provide synchronization. We take advantage of the feedback capabilities provided by RTCP and the ability to extend the protocol by extending and creating RTCP messages containing synchronization information. We have implemented our proposal and tested it in our University WAN. Our experiments have shown that network load resulting from synchronization is minimized and that asynchronies are within acceptable limits for multimedia applications.

Using visual and text features for direct marketing on multimedia messaging services domain

Traditionally, direct marketing companies have relied on pre-testing to select the best offers to send to their audience. Companies systematically dispatch the offers under consideration to a limited sample of potential buyers, rank them with respect to their performance and, based on this ranking, decide which offers to send to the wider population. Though this pre-testing process is simple and widely used, recently the industry has been under increased pressure to further optimize learning, in particular when facing severe time and learning space constraints. The main contribution of the present work is to demonstrate that direct marketing firms can exploit the information on visual content to optimize the learning phase. This paper proposes a two-phase learning strategy based on a cascade of regression methods that takes advantage of the visual and text features to improve and accelerate the learning process. Experiments in the domain of a commercial Multimedia Messaging Service (MMS) show the effectiveness of the proposed methods and a significant improvement over traditional learning techniques. The proposed approach can be used in any multimedia direct marketing domain in which offers comprise both a visual and text component.

Virtual assistant: enhancing content acquisition by eliciting information from humans

In this paper, we propose the “Virtual Assistant,” a novel framework for supporting knowledge capturing in videos. The Virtual Assistant is an artificial agent that simulates a human assistant shown in TV programs and prompts users to provide feedback by asking questions. This framework ensures that sufficient information is provided in the captured content while users interact in a natural and enjoyable way with the agent. We developed a prototype agent based on a chatbot-like approach and applied it to a daily cooking scene. Experimental results demonstrate the potential of the Virtual Assistant framework, as it allows a person to provide feedback easily with few interruptions and elicits a variety of useful information.

Theory of relative defect proneness

In this study, we investigated the functional form of the size-defect relationship for software modules through replicated studies conducted on ten open-source products. We consistently observed a power-law relationship where defect proneness increases at a slower rate compared to size. Therefore, smaller modules are proportionally more defect prone. We externally validated the application of our results for two commercial systems. Given limited and fixed resources for code inspections, there would be an impressive improvement in the cost-effectiveness, as much as 341% in one of the systems, if a smallest-first strategy were preferred over a largest-first one. The consistent results obtained in this study led us to state a theory of relative defect proneness (RDP): In large-scale software systems, smaller modules will be proportionally more defect-prone compared to larger ones. We suggest that practitioners consider our results and give higher priority to smaller modules in their focused quality assurance efforts.

An industrial case study in reconstructing requirements views

Requirements views, such as coverage and status views, are an important asset for monitoring and managing software development projects. We have developed a method that automates the process of reconstructing these views, and we have built a tool, ReqAnalyst, that supports this method. This paper presents an investigation as to which extent requirements views can be automatically generated in order to monitor requirements in industrial practice. The paper focuses on monitoring the requirements in test categories and test cases. In order to retrieve the necessary data, an information retrieval technique, called Latent Semantic Indexing, was used. The method was applied in an industrial study. A number of requirements views were defined and experiments were carried out with different reconstruction settings for generating these views. Finally, we explored how these views can help the developers during the software development process.

Macro-level software evolution: a case study of a large software compilation

Software evolution studies have traditionally focused on individual products. In this study we scale up the idea of software evolution by considering software compilations composed of a large quantity of independently developed products, engineered to work together. With the success of libre (free, open source) software, these compilations have become common in the form of ‘software distributions’, which group hundreds or thousands of software applications and libraries into an integrated system. We have performed an exploratory case study on one of them, Debian GNU/Linux, finding some significant results. First, Debian has been doubling in size every 2 years, totalling about 300 million lines of code as of 2007. Second, the mean size of packages has remained stable over time. Third, the number of dependencies between packages has been growing quickly. Finally, while C is still by far the most commonly used programming language for applications, use of the C++, Java, and Python languages have all significantly increased. The study helps not only to understand the evolution of Debian, but also yields insights into the evolution of mature libre software systems in general.

Presenting software engineering results using structured abstracts: a randomised experiment

When conducting a systematic literature review, researchers usually determine the relevance of primary studies on the basis of the title and abstract. However, experience indicates that the abstracts for many software engineering papers are of too poor a quality to be used for this purpose. A solution adopted in other domains is to employ structured abstracts to improve the quality of information provided. This study consists of a formal experiment to investigate whether structured abstracts are more complete and easier to understand than non-structured abstracts for papers that describe software engineering experiments. We constructed structured versions of the abstracts for a random selection of 25 papers describing software engineering experiments. The 64 participants were each presented with one abstract in its original unstructured form and one in a structured form, and for each one were asked to assess its clarity (measured on a scale of 1 to 10) and completeness (measured with a questionnaire that used 18 items). Based on a regression analysis that adjusted for participant, abstract, type of abstract seen first, knowledge of structured abstracts, software engineering role, and preference for conventional or structured abstracts, the use of structured abstracts increased the completeness score by 6.65 (SE 0.37, p < 0.001) and the clarity score by 2.98 (SE 0.23, p < 0.001). 57 participants reported their preferences regarding structured abstracts: 13 (23%) had no preference; 40 (70%) preferred structured abstracts; four preferred conventional abstracts. Many conventional software engineering abstracts omit important information. Our study is consistent with studies from other disciplines and confirms that structured abstracts can improve both information content and readability. Although care must be taken to develop appropriate structures for different types of article, we recommend that Software Engineering journals and conferences adopt structured abstracts.