Receiver-side semantic reasoning for digital TV personalization in the absence of return channels

Experience has proved that interactive applications delivered through Digital TV must provide personalized information to the viewers in order to be perceived as a valuable service. Due to the limited computational power of DTV receivers (either domestic set-top boxes or mobile devices), most of the existing systems have opted to place the personalization engines in dedicated servers, assuming that a return channel is always available for bidirectional communication. However, in a domain where most of the information is transmitted through broadcast, there are still many cases of intermittent, sporadic or null access to a return channel. In such situations, it is impossible for the servers to learn who is watching TV at the moment, and so the personalization features become unavailable. To solve this problem without sacrificing much personalization quality, this paper introduces solutions to run a downsized semantic reasoning process in the DTV receivers, supported by a pre-selection of material driven by audience stereotypes in the head-end. Evaluation results are presented to prove the feasibility of this approach, and also to assess the quality it achieves in comparison with previous ones.

Virtual assistant: enhancing content acquisition by eliciting information from humans

In this paper, we propose the “Virtual Assistant,” a novel framework for supporting knowledge capturing in videos. The Virtual Assistant is an artificial agent that simulates a human assistant shown in TV programs and prompts users to provide feedback by asking questions. This framework ensures that sufficient information is provided in the captured content while users interact in a natural and enjoyable way with the agent. We developed a prototype agent based on a chatbot-like approach and applied it to a daily cooking scene. Experimental results demonstrate the potential of the Virtual Assistant framework, as it allows a person to provide feedback easily with few interruptions and elicits a variety of useful information.

Automated music video generation using multi-level feature-based segmentation

We show how to create a music video automatically, using computable characteristics of the video and music to promote coherent matching. We analyze the flow of both music and video, and then segment them into sequences of near-uniform flow. We extract features from the both video and music segments, and then find matching pairs. The granularity of the matching process can be adapted by extending the segmentation process to several levels. Our approach drastically reduces the skill required to make simple music videos.

An RTP/RTCP based approach for multimedia group and inter-stream synchronization

Most multimedia group and inter-stream synchronization techniques define or use proprietary protocols with new control messages. Many multimedia applications have been developed using RTP/RTCP as the standard for transmission of multimedia streams over IP networks. Instead of defining a new protocol, we propose the use of RTP/RTCP to provide synchronization. We take advantage of the feedback capabilities provided by RTCP and the ability to extend the protocol by extending and creating RTCP messages containing synchronization information. We have implemented our proposal and tested it in our University WAN. Our experiments have shown that network load resulting from synchronization is minimized and that asynchronies are within acceptable limits for multimedia applications.

T-MAESTRO and its authoring tool: using adaptation to integrate entertainment into personalized t-learning

Interactive Digital TV opens new learning possibilities where new forms of education are needed. On the one hand, the combination of education and entertainment is essential to boost the participation of viewers in TV learning (t-learning), overcoming their typical passiveness. On the other hand, researchers broadly agree that in order to prevent the learner from abandoning the learning experience, it is necessary to take into account his/her particular needs and preferences by means of a personalized experience. Bearing this in mind, this paper introduces a new approach to the conception of personalized t-learning: edutainment and entercation experiences. These experiences combine TV programs and learning contents in a personalized way, with the aim of using the playful nature of TV to make learning more attractive and to engage TV viewers in learning. This paper brings together our work in constructing edutainment/entercation experiences by relating TV and learning contents. Taking personalization one step further, we propose the adaptation of learning contents by defining A-SCORM (Adaptive-SCORM), an extension of the ADL SCORM standard. Over and above the adaptive add-ons, this paper focuses on two fundamental entities for the proposal: (1) an Intelligent Tutoring System, called T-MAESTRO, which constructs the t-learning experiences by applying semantic knowledge about the t-learners; and (2) the authoring tool which allow teachers to create adaptive courses with a minimal technical background.

Semi-supervised discriminative classification with application to tumorous tissues segmentation of MR brain images

Due to the large data size of 3D MR brain images and the blurry boundary of the pathological tissues, tumor segmentation work is difficult. This paper introduces a discriminative classification algorithm for semi-automated segmentation of brain tumorous tissues. The classifier uses interactive hints to obtain models to classify normal and tumor tissues. A non-parametric Bayesian Gaussian random field in the semi-supervised mode is implemented. Our approach uses both labeled data and a subset of unlabeled data sampling from 2D/3D images for training the model. Fast algorithm is also developed. Experiments show that our approach produces satisfactory segmentation results comparing to the manually labeled results by experts.

Semantic image classification using statistical local spatial relations model

In this paper, a statistical model called statistical local spatial relations (SLSR) is presented as a novel technique of a learning model with spatial and statistical information for semantic image classification. The model is inspired by probabilistic Latent Semantic Analysis (PLSA) for text mining. In text analysis, PLSA is used to discover topics in a corpus using the bag-of-word document representation. In SLSR, we treat image categories as topics, therefore an image containing instances of multiple categories can be modeled as a mixture of topics. More significantly, SLSR introduces spatial relation information as a factor which is not present in PLSA. SLSR has rotation, scale, translation and affine invariant properties and can solve partial occlusion problems. Using the Dirichlet process and variational Expectation-Maximization learning algorithm, SLSR is developed as an implementation of an image classification algorithm. SLSR uses an unsupervised process which can capture both spatial relations and statistical information simultaneously. The experiments are demonstrated on some standard data sets and show that the SLSR model is a promising model for semantic image classification problems.

Attributing semantics to personal photographs

A major bottleneck for the efficient management of personal photographic collections is the large gap between low-level image features and high-level semantic contents of images. This paper proposes and evaluates two methodologies for making appropriate (re)use of natural language photographic annotations for extracting references to people, location and objects and propagating any location references encountered to previously unannotated images. The evaluation identifies the strengths of each approach and shows extraction and propagation results with promising accuracy.

Crossing textual and visual content in different application scenarios

This paper deals with multimedia information access. We propose two new approaches for hybrid text-image information processing that can be straightforwardly generalized to the more general multimodal scenario. Both approaches fall in the trans-media pseudo-relevance feedback category. Our first method proposes using a mixture model of the aggregate components, considering them as a single relevance concept. In our second approach, we define trans-media similarities as an aggregation of monomodal similarities between the elements of the aggregate and the new multimodal object. We also introduce the monomodal similarity measures for text and images that serve as basic components for both proposed trans-media similarities. We show how one can frame a large variety of problem in order to address them with the proposed techniques: image annotation or captioning, text illustration and multimedia retrieval and clustering. Finally, we present how these methods can be integrated in two applications: a travel blog assistant system and a tool for browsing the Wikipedia taking into account the multimedia nature of its content.

Using information retrieval based coupling measures for impact analysis

Coupling is an important property of software systems, which directly impacts program comprehension. In addition, the strength of coupling measured between modules in software is often used as a predictor of external software quality attributes such as changeability, ripple effects of changes and fault-proneness. This paper presents a new set of coupling measures for Object-Oriented (OO) software systems measuring conceptual coupling of classes. Conceptual coupling is based on measuring the degree to which the identifiers and comments from different classes relate to each other. This type of relationship, called conceptual coupling, is measured through the use of Information Retrieval (IR) techniques. The proposed measures are different from existing coupling measures and they capture new dimensions of coupling, which are not captured by the existing coupling measures. The paper investigates the use of the conceptual coupling measures during change impact analysis. The paper reports the findings of a case study in the source code of the Mozilla web browser, where the conceptual coupling metrics were compared to nine existing structural coupling metrics and proved to be better predictors for classes impacted by changes.

MBC-based shape retrieval: basics, optimizations, and open problems

Shape of an object is an important feature for image and multimedia similarity retrievals. In our previous studies we introduced a new - technique (-) for shape retrieval and compared its performance to other techniques. In this study, we describe in detail the basics of our - shape representation techniques, and we show how they support different query types. In addition, we describe two original optimization techniques that can further improve the performance of our - methods in several aspects, and show that they are also applicable to other applications (e.g., pattern recognition techniques). Finally, we define open problems in the area (e.g., partial similarity) and provide some hints on how to approach those problems.

Analyzing the efficacy of using digital ink devices in a learning environment

There has been increased interest on the impact of mobile devices such as PDAs and Tablet PCs in introducing new pedagogical approaches and active learning experiences. We propose an intelligent system that efficiently addresses the inherent subjectivity in student perception of note taking and information retrieval. We employ the idea of cross indexing the digital ink notes with matching electronic documents in the repository. Latent Semantic Indexing is used to perform document and page level indexing. Thus for each retrieved document, the user can go over to the relevant pages that match the query. Techniques to handle problems such as polysemy (multiple meanings of a word) in large databases, document folding and no match for query are discussed. We tested our system for its performance, usability and effectiveness in the learning process. The results from the exploratory studies reveal that the proposed system provides a highly enhanced student learning experience, thereby facilitating high test scores.

An index and retrieval framework integrating perceptive features and semantics for multimedia databases

Typically, in multimedia databases, there exist two kinds of clues for query: perceptive features and semantic classes. In this paper, we propose a novel framework for multimedia databases index and retrieval integrating the perceptive features and semantic classes to improve the speed and the precision of the content-based multimedia retrieval (CBMR). We develop a semantics supervised clustering based index approach (briefly as SSCI): the entire data set is divided hierarchically into many clusters until the objects within a cluster are not only close in the perceptive feature space but also within the same semantic class, and then an index term is built for each cluster. Especially, the perceptive feature vectors in a cluster are organized adjacently in disk. So the SSCI-based nearest-neighbor (NN) search can be divided into two phases: first, the indexes of all clusters are scanned sequentially to get the candidate clusters with the smallest distances from the query example; second, the original feature vectors within the candidate clusters are visited to get search results. Furthermore, if the results are not satisfied, the SSCI supports an effective relevance feedback (RF) search: users mark the positive and negative samples regarded a cluster as unit instead of a single object; then the Bayesian classifiers on perceptive features and that on semantics are used respectively to adjust retrieval similarity distance. Our experiments show that SSCI-based searching was faster than VA⁺-based searching; the quality of the search result based on SSCI was better than that of the sequential search in terms of semantics; and a few cycles of the RF by the proposed approach can improve the retrieval precision significantly.

Using visual and text features for direct marketing on multimedia messaging services domain

Traditionally, direct marketing companies have relied on pre-testing to select the best offers to send to their audience. Companies systematically dispatch the offers under consideration to a limited sample of potential buyers, rank them with respect to their performance and, based on this ranking, decide which offers to send to the wider population. Though this pre-testing process is simple and widely used, recently the industry has been under increased pressure to further optimize learning, in particular when facing severe time and learning space constraints. The main contribution of the present work is to demonstrate that direct marketing firms can exploit the information on visual content to optimize the learning phase. This paper proposes a two-phase learning strategy based on a cascade of regression methods that takes advantage of the visual and text features to improve and accelerate the learning process. Experiments in the domain of a commercial Multimedia Messaging Service (MMS) show the effectiveness of the proposed methods and a significant improvement over traditional learning techniques. The proposed approach can be used in any multimedia direct marketing domain in which offers comprise both a visual and text component.

On detecting faults for Boolean expressions

Fault based testing aims at detecting hypothesized faults based on specifications or program source. There are some fault based techniques for testing Boolean expressions which are commonly used to model conditions in specifications as well as logical decisions in program source. The MUMCUT strategy has been proposed to generate test cases from Boolean expressions. Moreover, it detects eight common types of hypothesized faults provided that the original expression is in irredundant disjunctive normal form, IDNF. Software practitioners are more likely to write the conditions and logical decisions in general form rather than IDNF. Hence, it is interesting to investigate the fault detecting capability of the MUMCUT strategy with respect to general form Boolean expressions. In this article, we perform empirical studies to investigate the fault detection capability of the MUMCUT strategy with respect to general form Boolean expressions as well as mutated expressions. A mutated expression can be obtained from the original given Boolean expression by making a syntactic change based on a particular type of fault.

Face recognition using an NNSRM classifier in LDA subspace

NNSRM is an implementation of the structural risk minimization (SRM) principle using the nearest neighbor (NN) rule, and linear discriminant analysis (LDA) is a dimension-reducing method, which is usually used in classifications. This paper combines the two methods for face recognition. We first project the face images into a PCA subspace, then project the results into a much lower-dimensional LDA subspace, and then use an NNSRM classifier to recognize them in the LDA subspace. Experimental results demonstrate that the combined method can achieve a better performance than NN by selecting different distances and a comparable performance with SVM but costing less computational time.

Metadata-driven interactive web video assembly

The recent expansion of broadband Internet access led to an exponential increase of potential consumers of video on the Web. The huge success of video upload websites shows that the online world, with its virtually unlimited possibilities of active user participation, is an ideal complement to traditional consumption-only media like TV and DVD. It is evident that users are willing to interact with content-providing systems in order to get the content they desire. In parallel to these developments, innovative tools for producing interactive, non-linear audio-visual content are being created. They support the authoring process alongside management of media and metadata, enabling on-demand assembly of videos based on the consumer’s wishes. The quality of such a dynamic video remixing system mainly depends on the expressiveness of associated metadata. Eliminating the need for manual input as far as possible, we aim at designing a system which is able to automatically enrich its own media and metadata repositories continuously. Currently, video content remixing is available on the Web mostly in very basic forms. Most platforms offer upload and simple modification of content. Although several implementations exist, to the best of our knowledge no solution uses metadata to its full extent to dynamically render a video stream based on consumers’ wishes. With the research presented in this paper, we propose a novel concept to interactive video assembly on the Web. In this approach, consumers may describe the desired content using a set of domain-specific parameters. Based on the metadata the video clips are annotated with, the system chooses clips fitting the user criteria. They are aligned in an aesthetically pleasing manner while the user furthermore is able to interactively influence content selection during playback at any time. We use a practical example to clarify the concept and further outline what it takes to implement a suchlike system.