Toward semantic indexing and retrieval using hierarchical audio models

Semantic-level content analysis is a crucial issue in achieving efficient content retrieval and management. We propose a hierarchical approach that models the statistical characteristics of audio events over a time series to accomplish semantic context detection. Two stages, audio event and semantic context modeling, are devised to bridge the semantic gap between physical audio features and semantic concepts. In this work, hidden Markov models (HMMs) are used to model four representative audio events, i.e., gunshot, explosion, engine, and car-braking, in action movies. At the semantic-context level, Gaussian mixture models (GMMs) and ergodic HMMs are investigated to fuse the characteristics and correlations between various audio events. They provide cues for detecting gunplay and car-chasing scenes, two semantic contexts we focus on in this work. The promising experimental results demonstrate the effectiveness of the proposed approach and exhibit that the proposed framework provides a foundation in semantic indexing and retrieval. Moreover, the two fusion schemes are compared, and the relations between audio event and semantic context are studied.     

Machine annotation and retrieval for digital imagery of historical materials

Annotating digital imagery of historical materials for the purpose of computer-based retrieval is a labor-intensive task for many historians and digital collection managers. We have explored the possibilities of automated annotation and retrieval of images from collections of art and cultural images. In this paper, we introduce the application of the ALIP (Automatic Linguistic Indexing of Pictures) system, developed at Penn State, to the problem of machine-assisted annotation of images of historical materials. The ALIP system learns the expertise of a human annotator on the basis of a small collection of annotated representative images. The learned knowledge about the domain-specific concepts is stored as a dictionary of statistical models in a computer-based knowledge base. When an un-annotated image is presented to ALIP, the system computes the statistical likelihood of the image resembling each of the learned statistical models and the best concept is selected to annotate the image. Experimental results, obtained using the Emperor image collection of the Chinese Memory Net project, are reported and discussed. The system has been trained using subsets of images and metadata from the Emperor collection. Finally, we introduce an integration of wavelet-based annotation and wavelet-based progressive displaying of very high resolution copyright-protected images. A preliminary version of this work has been presented at the DELOS-NSF Workshop on Multimedia in Digital Libraries, Crete, Greece, June 2003. The work was completed when Kurt Grieb and Ya Zhang were students of The Pennsylvania State University. James Z. Wang and Jia Li are also affiliated with Department of Computer Science and Engineering, The Pennsylvania State University. Yixin Chen is also with the Research Institute for Children, Children's Hospital, New Orleans.     

Application of acoustic directional data for audio event recognition via HMM/CRF in perimeter surveillance systems

Audio event detection (AED) and recognition is a signal processing and analysis domain used in a wide range of applications including surveillance, home automation and behavioral assessment. The field presents numerous challenges to the current state-of-the-art due to its highly nonlinear nature. High false alarm rates (FARs) in such applications particularly limit the capabilities of vision-based perimeter monitoring systems by inducing high operator dependence. On the other hand, conventional fence-based vibration detectors and pressure-driven “taut wires” offer high sensitivity at the cost of a high FAR due to debris, animals and weather.This work reports an audio event identification methodology implemented as a test-bed system for a surveillance application to reduce FAR, maximize blind-spot coverage and improve audio event classification accuracy. The first phase utilizes a nonlinear autoregressive classifier to locate and classify discrete audio events via an exogenous sound direction variable to improve classifier confidence. The second phase implements a time-series-based system to recognize various audio activity groups from nominal everyday sound events such as traffic and muffled speech. The discretely labeled data is thus trained with HMM and Conditional Random Field classifiers and reports a substantial improvement in classification accuracies of indoor human activities.     

Links between probabilistic automata and hidden Markov models: probability distributions, learning models and induction algorithms

This article presents an overview of Probabilistic Automata (PA) and discrete Hidden Markov Models (HMMs), and aims at clarifying the links between them. The first part of this work concentrates on probability distributions generated by these models. Necessary and sufficient conditions for an automaton to define a probabilistic language are detailed. It is proved that probabilistic deterministic automata (PDFA) form a proper subclass of probabilistic non-deterministic automata (PNFA). Two families of equivalent models are described next. On one hand, HMMs and PNFA with no final probabilities generate distributions over complete finite prefix-free sets. On the other hand, HMMs with final probabilities and probabilistic automata generate distributions over strings of finite length. The second part of this article presents several learning models, which formalize the problem of PA induction or, equivalently, the problem of HMM topology induction and parameter estimation. These learning models include the PAC and identification with probability 1 frameworks. Links with Bayesian learning are also discussed. The last part of this article presents an overview of induction algorithms for PA or HMMs using state merging, state splitting, parameter pruning and error-correcting techniques.     

FESORIA: An integrated system for analysis,management and smart presentation of audio/video streaming services

Due to the elevated consumption of resources, the high cost of the production of contents and the quality of service required in audio/video streaming services, it is extremely important to optimize all the elements involved in the deployment of these services. With this goal in mind, provider companies have developed their management and presentation tools. At the same time, some specific tools for audio/video streaming analysis have appeared. Data are collected from servers and proxies by analyzing their log files in order to generate different types of reports. In spite of their utility, there is a disconnection between these types of tools. In this way, several important relationships between collected data are lost and the influence of other important aspects such as the behaviour of the users and their relationship with the subject or the length of the contents is not considered. This generates inaccurate analyses and the impossibility to improve the presentation, for example by generating recommendations using the information gathered from the analysis tool. Fesoria is a system which combines both characteristics. It is an analysis tool and, at the same time, a system to manage the whole audio/video service. Fesoria is able to process the logs gathered from the streaming servers and proxies, and combine the extracted information with other types of data, such as content metadata, content distribution networks architecture, user preferences, etc. All this information is analyzed in order to generate reports on service performance, access evolution and users’ preferences, and thus to improve the presentation of the services. The system has been used in real audio/video services since 2001 with satisfactory results.

Challenging games help students learn: An empirical study on engagement,flow and immersion in game-based learning

In this paper, we investigate the impact of flow (operationalized as heightened challenge and skill), engagement, and immersion on learning in game-based learning environments. The data was gathered through a survey from players (N = 173) of two learning games (Quantum Spectre: N = 134 and Spumone: N = 40). The results show that engagement in the game has a clear positive effect on learning, however, we did not find a significant effect between immersion in the game and learning. Challenge of the game had a positive effect on learning both directly and via the increased engagement. Being skilled in the game did not affect learning directly but by increasing engagement in the game. Both the challenge of the game and being skilled in the game had a positive effect on both being engaged and immersed in the game. The challenge in the game was an especially strong predictor of learning outcomes. For the design of educational games, the results suggest that the challenge of the game should be able to keep up with the learners growing abilities and learning in order to endorse continued learning in game-based learning environments.     

An implementation of design-based learning through creating educational computer games: A case study on mathematics learning during design and computing

This mixed-method case study examined the potential of computer-assisted, math game making activities in facilitating design-based math learning for school children. Sixty-four middle school children participated in Scratch-based, math game making activities. Data were collected via activity and conversation observation, artifact analysis, interviewing, and survey. The study findings indicated that participants developed significantly more positive dispositions toward mathematics after computer game making. The study also found that experience-driven game design processes helped to activate children's reflection on everyday mathematical experiences. Mathematical thinking and content experience were intertwined within the process of computer game authoring. On the other hand, children designers were involved in game-world and story crafting more than mathematical representation. And it was still challenging for them to perform computer game coding with abstract reasoning.     

Model learning: a survey of foundations,tools and applications

Software systems are present all around us and playing their vital roles in our daily life. The correct functioning of these systems is of prime concern. In addition to classical testing techniques, formal techniques like model checking are used to reinforce the quality and reliability of software systems. However, obtaining of behavior model, which is essential for model-based techniques, of unknown software systems is a challenging task. To mitigate this problem, an emerging black-box analysis technique, called Model Learning, can be applied. It complements existing model-based testing and verification approaches by providing behavior models of blackbox systems fully automatically. This paper surveys the model learning technique, which recently has attracted much attention from researchers, especially from the domains of testing and verification. First, we review the background and foundations of model learning, which form the basis of subsequent sections. Second, we present some well-known model learning tools and provide their merits and shortcomings in the form of a comparison table. Third, we describe the successful applications of model learning in multidisciplinary fields, current challenges along with possible future works, and concluding remarks.     

HMM-based handwritten word recognition: on the optimization of the number of states, training iterations and Gaussian components

In off-line handwriting recognition, classifiers based on hidden Markov models (HMMs) have become very popular. However, while there exist well-established training algorithms which optimize the transition and output probabilities of a given HMM architecture, the architecture itself, and in particular the number of states, must be chosen “by hand”. Also the number of training iterations and the output distributions need to be defined by the system designer. In this paper we examine several optimization strategies for an HMM classifier that works with continuous feature values. The proposed optimization strategies are evaluated in the context of a handwritten word recognition task.     

Exploring the impact of individualism and uncertainty avoidance in Web-based electronic learning: An empirical analysis in European higher education

Our research specifically focuses on the effects of the national cultural background of educators on the acceptance and usage of ICT, particularly the Web as an extensive and expanding information base that provides the ultimate in resource-rich learning. Most research has been used North Americans as subjects. For this reason, we interviewed European educators from diverse cultures; in particularly, we analysed the cultural differences and their moderating effects on acceptance-based relationships between European universities: European Nordic culture in contrast to European-Mediterranean culture.