Sub-event discovery and retrieval during natural hazards on social media data

Social media sites contain a considerable amount of data for natural calamities events, such as earthquakes, snowstorms, mud-rock flows. With the increasing amount of social media data, an important task is to discover and retrieve sub-events over time. Especially in emergency situations, rescue and relief activities can be enhanced by identifying and retrieving sub-events of a natural hazard event. However, the existing event detection techniques in news-related reports cannot effectively work for social media data due to the unstructured of social network data. In this paper, we propose a new natural hazard sub-events discovery model SED (Sub-Events Discovery), which adopts multifarious features to detect sub-events. Moreover, in order to retrieve the sub-events over a specific event, we introduce a novel SER (Sub-Event Retrieval) algorithm from time-stamped social media data. Our novel approach SER makes use of automatically obtained messages from external search engines in the entire process. For purpose of determining the periodical convergence time for natural hazard event, our method provides online sub-events retrieval and sub-events discovery to meet the further needs. Next the improved estimation standards with timestamp are utilized in our experiments to verify the effectiveness and efficiency of SED model and SER algorithm.     

基于四元数门控图神经网络的脚本事件预测

脚本事件预测需要考虑两类信息来源:事件间的关联与事件内的交互。针对于事件间的关联,采用门控图神经网络对其进行建模。而对于事件内的交互,采用四元数对事件进行表征,接着通过四元数的哈密顿乘积来捕捉事件4个组成部分之间的交互。提出结合四元数和门控图神经网络来学习事件表示,它既考虑了外部事件图的交互作用,又考虑了事件内部的依赖关系。得到事件表示后,利用注意机制学习上下文事件表示和每个候选上下文表示的相对权值。然后通过权重计算上下文事件表示的和,再计算其与候选事件表示的欧氏距离。最后选择距离最小的候选事件作为正确的候选事件。在纽约时报语库上进行了实验,结果表明,通过多项选择叙事完形填空评价,本文的模型优于现有的基线模型     

基于超图模型的复杂视频事件检测

近年来,语义事件分析越来越受到重视,典型语义事件的检测与识别是一个具有挑战性的研究领域。提出了基于超图模型的复杂视频事件检测方法,通过分析对象的运动轨迹,检测出视频中的所有子事件并构建时序关系图及依赖关系图,从而生成子事件超图,并通过谱超图聚类分析来检测相应的复杂事件。采用图变换工具AGG进行模拟实验,其实验结果表明,该方法具有较高的准确率与召回率。     

E-LAMP: integration of innovative ideas for multimedia event detection

Detecting multimedia events in web videos is an emerging hot research area in the fields of multimedia and computer vision. In this paper, we introduce the core methods and technologies of the framework we developed recently for our Event Labeling through Analytic Media Processing (E-LAMP) system to deal with different aspects of the overall problem of event detection. More specifically, we have developed efficient methods for feature extraction so that we are able to handle large collections of video data with thousands of hours of videos. Second, we represent the extracted raw features in a spatial bag-of-words model with more effective tilings such that the spatial layout information of different features and different events can be better captured, thus the overall detection performance can be improved. Third, different from widely used early and late fusion schemes, a novel algorithm is developed to learn a more robust and discriminative intermediate feature representation from multiple features so that better event models can be built upon it. Finally, to tackle the additional challenge of event detection with only very few positive exemplars, we have developed a novel algorithm which is able to effectively adapt the knowledge learnt from auxiliary sources to assist the event detection. Both our empirical results and the official evaluation results on TRECVID MED’11 and MED’12 demonstrate the excellent performance of the integration of these ideas.     

基于多任务学习的中文事件抽取联合模型

事件抽取旨在从非结构化的文本中提取人们感兴趣的信息,并以结构化的形式呈现给用户.当前,大多数中文事件抽取系统采用连续的管道模型,即：先识别事件触发词,后识别事件元素.其容易产生级联错误,且处于下游的任务无法将信息反馈至上游任务,辅助上游任务的识别.将事件抽取看作序列标注任务,构建了基于CRF多任务学习的中文事件抽取联合模型.针对仅基于CRF的事件抽取联合模型的缺陷进行了两个扩展：首先,采用分类训练策略解决联合模型中事件元素的多标签问题（即：当一个事件提及中包含多个事件时,同一个实体往往会在不同的事件中扮演不同的角色）.其次,由于处于同一事件大类下的事件子类,其事件元素存在高度的相互关联性.为此,提出采用多任务学习方法对各事件子类进行互增强的联合学习,进而有效缓解分类训练后的语料稀疏问题.在ACE 2005中文语料上的实验证明了该方法的有效性.     

HyperED: A hierarchy-aware network based on hyperbolic geometry for event detection

Event detection plays an essential role in the task of event extraction. It aims at identifying event trigger words in a sentence and classifying event types. Generally, multiple event types are usually well-organized with a hierarchical structure in real-world scenarios, and hierarchical correlations between event types can be used to enhance event detection performance. However, such kind of hierarchical information has received insufficient attention which can lead to misclassification between multiple event types. In addition, the most existing methods perform event detection in Euclidean space, which cannot adequately represent hierarchical relationships. To address these issues, we propose a novel event detection network HyperED which embeds the event context and types in Poincaré ball of hyperbolic geometry to help learn hierarchical features between events. Specifically, for the event detection context, we first leverage the pre-trained BERT or BiLSTM in Euclidean space to learn the semantic features of ED sentences. Meanwhile, to make full use of the dependency knowledge, a GNN-based model is applied when encoding event types to learn the correlations between events. Then we use a simple neural-based transformation to project the embeddings into the Poincaré ball to capture hierarchical features, and a distance score in hyperbolic space is computed for prediction. The experiments on MAVEN and ACE 2005 datasets indicate the effectiveness of the HyperED model and prove the natural advantages of hyperbolic spaces in expressing hierarchies in an intuitive way.     

A Novel Web Video Event Mining Framework with the Integration of Correlation and Co-Occurrence Information

The massive web videos prompt an imperative demand on efficiently grasping the major events.However, the distinct characteristics of web videos, such as the limited number of features, the noisy text information, and the unavoidable error in near-duplicate keyframes (NDKs) detection, make web video event mining a challenging task.In this paper, we propose a novel four-stage framework to improve the performance of web video event mining.Data preprocessing is the first stage.Multiple Correspondence Analysis (MCA) is then applied to explore the correlation between terms and classes, targeting for bridging the gap between NDKs and high-level semantic concepts.Next, co-occurrence information is used to detect the similarity between NDKs and classes using the NDK-within-video information.Finally, both of them are integrated for web video event mining through negative NDK pruning and positive NDK enhancement.Moreover, both NDKs and terms with relatively low frequencies are treated as useful information in our experiments.Experimental results on large-scale web videos from YouTube demonstrate that the proposed framework outperforms several existing mining methods and obtains good results for web video event mining.     

A framework of joint object tracking and event detection

This paper describes a probabilistic framework for simultaneously performing object tracking and event detection in monocular videos. Mathematically, we cast the problem of jointly tracking and detecting semantic events as a principled model-based search problem in a multi-dimensional state space, where the tracking trajectory and event type are discovered via maximum a posteriori (MAP) optimization. The benefit of this approach comes from its combined utilization of particle probabilistic representation, multiple hypothesis retention, efficient particle propagation, and temporal optimization. We present qualitative and quantitative results from realistic video sequences to demonstrate the effectiveness of this approach.     

Learning and parsing video events with goal and intent prediction

In this paper, we present a framework for parsing video events with stochastic Temporal And–Or Graph (T-AOG) and unsupervised learning of the T-AOG from video. This T-AOG represents a stochastic event grammar. The alphabet of the T-AOG consists of a set of grounded spatial relations including the poses of agents and their interactions with objects in the scene. The terminal nodes of the T-AOG are atomic actions which are specified by a number of grounded relations over image frames. An And-node represents a sequence of actions. An Or-node represents a number of alternative ways of such concatenations. The And–Or nodes in the T-AOG can generate a set of valid temporal configurations of atomic actions, which can be equivalently represented as the language of a stochastic context-free grammar (SCFG). For each And-node we model the temporal relations of its children nodes to distinguish events with similar structures but different temporal patterns and interpolate missing portions of events. This makes the T-AOG grammar context-sensitive. We propose an unsupervised learning algorithm to learn the atomic actions, the temporal relations and the And–Or nodes under the information projection principle in a coherent probabilistic framework. We also propose an event parsing algorithm based on the T-AOG which can understand events, infer the goal of agents, and predict their plausible intended actions. In comparison with existing methods, our paper makes the following contributions. (i) We represent events by a T-AOG with hierarchical compositions of events and the temporal relations between the sub-events. (ii) We learn the grammar, including atomic actions and temporal relations, automatically from the video data without manual supervision. (iii) Our algorithm infers the goal of agents and predicts their intents by a top-down process, handles events insertion and multi-agent events, keeps all possible interpretations of the video to preserve the ambiguities, and achieves the globally optimal parsing solution in a Bayesian framework. (iv) The algorithm uses event context to improve the detection of atomic actions, segment and recognize objects in the scene. Extensive experiments, including indoor and out door scenes, single and multiple agents events, are conducted to validate the effectiveness of the proposed approach.     

融入中文语义信息及越南语句法特征的越南语事件检测方法

当前基于深度学习的事件检测模型都依赖足够数量的标注数据,而标注数据的稀缺及事件类型歧义为越南语事件检测带来了极大的挑战。根据“表达相同观点但语言不同的句子通常有相同或相似的语义成分”这一多语言一致性特征,该文提出了一种融入中文语义信息及越南语句法特征的越南语事件检测框架。首先通过共享编码器策略和交叉注意力网络将中文信息融入越南语中,然后使用图卷积网络融入越南语依存句法信息,最后在中文事件类型指导下实现越南语事件检测。实验结果表明,在中文语义信息和越南语句法特征的指导下越南语事件检测取得了较好的效果。     

Leveraging enterprise knowledge graph to infer web events’ influences via self-supervised learning

Knowledge graph (KG) techniques have achieved successful results in many tasks, especially in semantic web and natural language processing domains. In recent years, representation learning on KG has been successfully applied to e-business applications, such as event-driven automatic investment strategies. However, there is still limited research about learning events’ influence on KG for modern quantitative investment. In this paper, we propose a novel event influence learning framework to predict stock market trends, called ST-Trend, leveraging enterprise knowledge graph to represent company correlation relationships, for mining the deep background knowledge of web events, with three self-supervised learning tasks. In particular, we devise two jointly self-supervised tasks to identify the relations between web events and companies. The first task is for generating ground-truth event-company correlation labels based on the enterprise knowledge graph. The second task is used to train how to identify the correlated companies of an event based on the generated correlation labels, with the encoding of web events, company features, and technical sequential data. We then design the prediction network to infer an event’s influence on stock price trends of the identified correlated companies based on the enterprise KG. Finally, we perform extensive experiments on a massive real-life dataset to validate the effectiveness of our proposed framework, and the experimental results demonstrate its superior performance in predicting stock market trends via considering events’ influences with the enterprise knowledge graph.     

Abnormality detection in crowd videos by tracking sparse components

Abnormality detection in crowded scenes plays a very important role in automatic monitoring of surveillance feeds. Here we present a novel framework for abnormality detection in crowd videos. The key idea of the approach is that rarely or sparsely occurring events correspond to abnormal activities, while the regularly or commonly occurring events correspond to the normal activities. Each input video is represented using feature matrices that capture the nature of activity taking place while maintaining the spatial and temporal structure of the video. The feature matrices are decomposed into their low-rank and sparse components where sparse component corresponds to the abnormal activities. The approach does not require any explicit modeling of crowd behavior or training, but the information from training data can be seamlessly incorporated if it is available. The estimation is further improved by ensuring temporal and spatial coherence of sparse component across the videos using a Kalman filter-like framework. This not only results in reduction of outliers and noise but also fills missing regions in the sparse component. Localization of the anomalies is obtained as a by-product of the proposed approach. Evaluation on the UMN and UCSD datasets and comparisons with several state-of-the-art crowd abnormality detection approaches shows the effectiveness of the proposed approach. We also show results on a challenging crowd dataset created as part of this effort, with videos downloaded from the web.     

Structuring low-quality videotaped lectures for cross-reference browsing by video text analysis

This paper presents an unified approach in analyzing and structuring the content of videotaped lectures for distance learning applications. By structuring lecture videos, we can support topic indexing and semantic querying of multimedia documents captured in the traditional classrooms. Our goal in this paper is to automatically construct the cross references of lecture videos and textual documents so as to facilitate the synchronized browsing and presentation of multimedia information. The major issues involved in our approach are topical event detection, video text analysis and the matching of slide shots and external documents. In topical event detection, a novel transition detector is proposed to rapidly locate the slide shot boundaries by computing the changes of text and background regions in videos. For each detected topical event, multiple keyframes are extracted for video text detection, super-resolution reconstruction, binarization and recognition. A new approach for the reconstruction of high-resolution textboxes based on linear interpolation and multi-frame integration is also proposed for the effective binarization and recognition. The recognized characters are utilized to match the video slide shots and external documents based on our proposed title and content similarity measures.     

Event graphs for information retrieval and multi-document summarization

With the number of documents describing real-world events and event-oriented information needs rapidly growing on a daily basis, the need for efficient retrieval and concise presentation of event-related information is becoming apparent. Nonetheless, the majority of information retrieval and text summarization methods rely on shallow document representations that do not account for the semantics of events. In this article, we present event graphs, a novel event-based document representation model that filters and structures the information about events described in text. To construct the event graphs, we combine machine learning and rule-based models to extract sentence-level event mentions and determine the temporal relations between them. Building on event graphs, we present novel models for information retrieval and multi-document summarization. The information retrieval model measures the similarity between queries and documents by computing graph kernels over event graphs. The extractive multi-document summarization model selects sentences based on the relevance of the individual event mentions and the temporal structure of events. Experimental evaluation shows that our retrieval model significantly outperforms well-established retrieval models on event-oriented test collections, while the summarization model outperforms competitive models from shared multi-document summarization tasks.