Multi-Modal Dialog Scene Detection Using Hidden Markov Models for Content-Based Multimedia Indexing

A class of audio-visual data (fiction entertainment: movies, TV series) is segmented into scenes, which contain dialogs, using a novel hidden Markov model-based (HMM) method. Each shot is classified using both audio track (via classification of speech, silence and music) and visual content (face and location information). The result of this shot-based classification is an audio-visual token to be used by the HMM state diagram to achieve scene analysis. After simulations with circular and left-to-right HMM topologies, it is observed that both are performing very good with multi-modal inputs. Moreover, for circular topology, the comparisons between different training and observation sets show that audio and face information together gives the most consistent results among different observation sets.     

Analytical evaluation of the MoM matrix elements

Derivation of the closed-form Green's functions has eliminated the computationally expensive evaluation of the Sommerfeld integrals to obtain the Green's functions in the spatial domain. Therefore, using the closed-form Green's functions in conjunction with the method of moments (MoM) has improved the computational efficiency of the technique significantly. Further improvement can be achieved on the calculation of the matrix elements involved in the MoM, usually double integrals for planar geometries, by eliminating the numerical integration. The contribution of this paper is to present the analytical evaluation of the matrix elements when the closed-form Green's functions are used, and to demonstrate the amount of improvement in computation time     

Dominant sets based movie scene detection

Multimedia indexing and retrieval has become a challenging topic in organizing huge amount of multimedia data. This problem is not a trivial task for large visual databases; hence, segmentation into low- and high-level temporal video segments might improve the realization of this task. In this paper, we introduce a weighted undirected graph-based movie scene detection approach to detect semantically meaningful temporal video segments. The method is based on the idea of finding the dominant scene of the video according to the selected low-level feature. The proposed method starts from obtaining the most reliable solution first and exploit each solution in the subsequent steps recursively. The dominant movie scene boundary, which can be the highest probability to be the correct one, is determined and this scene boundary information is also exploited in the subsequent steps. We handle two partitioning strategies to determine the boundaries of the remaining scenes. One is a tree-based strategy and the other is an order-based strategy. The proposed dominant sets based movie scene detection method is compared with the graph-based video scene detection methods presented in literature.     

Unequal error protection of SPIHT encoded image bit streams

A derivative of the set partitioning into hierarchical trees (SPIHT) image coding method, which generates substreams with different error-resilience properties, is proposed. By dividing the image bit stream into three classes, substreams with different immunity properties are obtained. The unequal protection of these substreams with different channel coding rates improves the overall performance of the method against channel errors. Simulation results show the superiority of the proposed method over some of the state-of-the-art methods     

User assisted disparity remapping for stereo images

This study concentrates on user assisted disparity remapping for stereo image footage, i.e. the disparity of an object of interest is altered while leaving the remaining scene unattended. This application is useful in the sense that it provides a method for emphasizing/de-emphasizing an object on the scene by adjusting its depth with respect to the camera. The proposed technique can also be used as a post-processing step for retargeting stereoscopic footage on different display sizes and resolutions. The proposed technique involves an MRF-based energy minimization step for interactive stereo image segmentation, for which user assistance on only one of the stereo pairs is required for determining the location of stereo object pair. A key contribution of the proposed study is elimination of dense disparity estimation step from the pipeline. This step is realized through a sparse feature matching technique between the stereo pairs. Moreover, by the help of the proposed technique, novel disparity adjusted views are synthesized using the produced stereo object segments and background information for the images. Qualitative and quantitative evaluation of the generated segments and the disparity adjusted images prove the functionality and superiority of the proposed technique.     

Use of computationally efficient method of moments in theoptimization of printed antennas

Derivation of the closed-form Green's functions and analytical evaluation of the method of moments (MOM) matrix entries have improved the computational efficiency significantly in the analysis of printed geometries. With this background in mind, an extension of this efficient numerical technique is to incorporate an optimization algorithm and to assess its potential as a computer-aided design (CAD) tool. Therefore, we have employed the gradient search and genetic algorithms, in conjunction with the electromagnetic (EM) simulation technique, to a number of representative examples of interest     

Information permeability for stereo matching

A novel local stereo matching algorithm is introduced to address the fundamental challenge of stereo algorithms, accuracy and computational complexity dilemma. The time consuming intensity dependent aggregation procedure of local methods is improved in terms of both speed and precision. Providing connected 2D support regions, the proposed approach exploits a new paradigm, namely separable successive weighted summation (SWS) among horizontal and vertical directions enabling constant operational complexity. The weights are determined by four-neighborhood intensity similarity of pixels and utilized to model the information transfer rate, permeability, towards the corresponding direction. The same procedure is also utilized to diffuse information through overlapped pixels during occlusion handling after detecting unreliable disparity assignments. Successive weighted summation adaptively cumulates the support data based on local characteristics, enabling disparity maps to preserve object boundaries and depth discontinuities. According to the experimental results on Middlebury stereo benchmark, the proposed method is one of the most effective local stereo algorithm providing high quality disparity models by unifying constant time filtering and weighted aggregation. Hence, the proposed algorithm provides a competitive alternative for various local methods in terms of achieving precise and consistent disparity maps from stereo video within fast execution time.     

Use of Asymptotic Waveform Evaluation Technique in the Analysis of Multilayer Structures With Doubly Periodic Dielectric Gratings

The reflection and dispersion characteristics of multilayer structures that involve periodically implanted material blocks are obtained by using the MoM solution of the volume integral equation. The asymptotic waveform evaluation (AWE) technique is utilized to obtain a Pade approximation of the solution in terms of a parameter such as frequency or incident angle. The use of AWE technique enables a fast sweep with respect to the approximation parameter. Moreover, a robust method for extracting the dispersion characteristics of periodic structures via Pade approximation is proposed. The AWE procedure requires the calculation of high order derivatives of the complicated kernel function that consists of Green's functions for stratified medium. These derivatives are calculated by employing the automatic differentiation theory. The reflection coefficient, propagation constant and band diagram of the structure are obtained both via point-by-point simulations and through the use of AWE technique. It is observed that AWE technique increases the computational efficiency without losing accuracy.     

Multimodal concept detection in broadcast media: KavTan

Concept detection stands as an important problem for efficient indexing and retrieval in large video archives. In this work, the KavTan System, which performs high-level semantic classification in one of the largest TV archives of Turkey, is presented. In this system, concept detection is performed using generalized visual and audio concept detection modules that are supported by video text detection, audio keyword spotting and specialized audio-visual semantic detection components. The performance of the presented framework was assessed objectively over a wide range of semantic concepts (5 high-level, 14 visual, 9 audio, 2 supplementary) by using a significant amount of precisely labeled ground truth data. KavTan System achieves successful high-level concept detection performance in unconstrained TV broadcast by efficiently utilizing multimodal information that is systematically extracted from both spatial and temporal extent of multimedia data.