PandaDB: Intelligent Management System for Heterogeneous Data

With the development of big data application, the demand of large-scale structured/unstructured data fusion management and analysis is becoming increasingly prominent. However, the differences in management, process, retrieval of structured/unstructured data brings challenges for fusion management and analysis. This study proposes an extended property graph model for heterogeneous data fusion management and semantic computing, and defines related property operators and query syntax. Based on the intelligent property graph model, this study implements PandaDB, an intelligent fusion management system for heterogeneous data. This study depicts the architecture, storage mechanism, query mechanism, property co-storage, AI algorithm scheduling, and distributed architecture of PandaDB. Test experiments and cases show that the co-storage mechanism and distributed architecture of PandaDB have good performance acceleration effects, and can be applied in some scenarios of fusion data intelligent management such as entity disambiguation of academic knowledge graph.     

面向多模态视频时刻检索的查询感知跨模态双重对比学习网络

近期,跨模态视频语料库时刻检索（VCMR）这一新任务被提出,它的目标是从未分段的视频语料库中检索出与查询语句相对应的一小段视频片段.现有的跨模态视频文本检索工作的关键点在于不同模态特征的对齐和融合,然而,简单地执行跨模态对齐和融合不能确保来自相同模态且语义相似的数据在联合特征空间下保持接近,也未考虑查询语句的语义.为了解决上述问题,本文提出了一种面向多模态视频片段检索的查询感知跨模态双重对比学习网络（QACLN）,该网络通过结合模态间和模态内的双重对比学习来获取不同模态数据的统一语义表示.具体地,本文提出了一种查询感知的跨模态语义融合策略,根据感知到的查询语义自适应地融合视频的视觉模态特征和字幕模态特征等多模态特征,获得视频的查询感知多模态联合表示.此外,提出了一种面向视频和查询语句的模态间及模态内双重对比学习机制,以增强不同模态的语义对齐和融合,从而提高不同模态数据表示的可分辨性和语义一致性.最后,采用一维卷积边界回归和跨模态语义相似度计算来完成时刻定位和视频检索.大量实验验证表明,所提出的QACLN优于基准方法.     

PandaDB:一种异构数据智能融合管理系统

随着大数据应用的不断深入,对大规模结构化/非结构化数据进行融合管理和分析的需求日益凸显.然而,结构化/非结构化数据在存储管理方式、信息获取方式、检索方式方面的差异给融合管理和分析带来了技术挑战.本文提出了适用于异构数据融合管理和语义计算的属性图扩展模型,并定义了相关属性操作符和查询语法.接着,基于智能属性图模型提出异构数据智能融合管理系统PandaDB,并详细介绍了PandaDB的总体架构、存储机制、查询机制、属性协存和AI算法集成机制.性能测试和应用案例证明,PandaDB的协存机制、分布式架构和语义索引机制对大规模异构数据的即席查询和分析具有较好的性能表现,该系统可实际应用于学术图谱实体消歧与可视化等融合数据管理场景.     

A Novel Approach Towards Large Scale Cross-Media Retrieval

With the rapid development of Internet and multimedia technology,cross-media retrieval is concerned to retrieve all the related media objects with multi-modality by submitting a query media object.Unfortunately,the complexity and the heterogeneity of multi-modality have posed the following two major challenges for cross-media retrieval:1) how to construct a unified and compact model for media objects with multi-modality,2) how to improve the performance of retrieval for large scale cross-media database.In this paper,we propose a novel method which is dedicate to solving these issues to achieve effective and accurate cross-media retrieval.Firstly,a multi-modality semantic relationship graph(MSRG) is constructed using the semantic correlation amongst the media objects with multi-modality.Secondly,all the media objects in MSRG are mapped onto an isomorphic semantic space.Further,an efficient indexing MK-tree based on heterogeneous data distribution is proposed to manage the media objects within the semantic space and improve the performance of cross-media retrieval.Extensive experiments on real large scale cross-media datasets indicate that our proposal dramatically improves the accuracy and efficiency of cross-media retrieval,outperforming the existing methods significantly.     

Cross the data desert: generating textual-visual summary on the evolutionary microblog stream

Effectively and efficiently summarizing social media is crucial and non-trivial to analyze social media. On social streams, events which are the main concept of semantic similar social messages, often bring us a firsthand story of daily news. However, to identify the valuable news, it is almost impossible to plough through millions of multi-modal messages one by one with traditional methods. Thus, it is urgent to summarize events with a few representative data samples on the streams. In this paper, we provide a vivid textual-visual media summarization approach for microblog streams, which exploits the incremental latent semantic analysis (LSA) of detected events. Firstly, with a novel weighting scheme for keyword relationship, we can detect and track daily sub-events on a keyword relation graph (WordGraph) of microblog streams effectively. Then, to summarize the stream with representative texts and images, we use cross-modal fusion to analyze the semantics of microblog texts and images incrementally and separately, with a novel incremental cross-modal LSA algorithm. The experimental results on a real microblog dataset show that our method is at least 1.31% better and 23.67% faster than existing state-of-the-art methods, and cross-modal fusion can improve the summarization performance by 4.16% on average.

     

基于标记增强的离散跨模态哈希方法

跨模态哈希通过将不同模态的数据映射为同一空间中更紧凑的哈希码,可以大大提升跨模态检索的效率.然而现有跨模态哈希方法通常使用二元相似性矩阵,不能准确描述样本间的语义相似关系,并且存在平方复杂度问题.为了更好地挖掘数据间的语义相似关系,提出了一个基于标记增强的离散跨模态哈希方法.首先借助迁移学习的先验知识生成样本的标记分布,然后通过标记分布构建描述度更强的语义相似性矩阵,再通过一个高效的离散优化算法生成哈希码,避免了量化误差问题.最后,在两个基准数据集上的实验结果验证了所提方法在跨模态检索任务上的有效性.     

Cross-modal alignment with graph reasoning for image-text retrieval

Image-text retrieval task has received a lot of attention in the modern research field of artificial intelligence. It still remains challenging since image and text are heterogeneous cross-modal data. The key issue of image-text retrieval is how to learn a common feature space while semantic correspondence between image and text remains. Existing works cannot gain fine cross-modal feature representation because the semantic relation between local features is not effectively utilized and the noise information is not suppressed. In order to address these issues, we propose a Cross-modal Alignment with Graph Reasoning (CAGR) model, in which the refined cross-modal features in the common feature space are learned and then a fine-grained cross-modal alignment method is implemented. Specifically, we introduce a graph reasoning module to explore semantic connection for local elements in each modality and measure their importance by self-attention mechanism. In a multi-step reasoning manner, the visual semantic graph and textual semantic graph can be effectively learned and the refined visual and textual features can be obtained. Finally, to measure the similarity between image and text, a novel alignment approach named cross-modal attentional fine-grained alignment is used to compute similarity score between two sets of features. Our model achieves the competitive performance compared with the state-of-the-art methods on Flickr30K dataset and MS-COCO dataset. Extensive experiments demonstrate the effectiveness of our model.

     

基于跨模态自蒸馏的零样本草图检索

零样本草图检索将未见类的草图作为查询样本,用于检索未见类的图像。因此,这个任务同时面临两个挑战：草图和图像之间的模态差异以及可见类和未见类的不一致性。过去的方法通过将草图和图像投射到一个公共空间来消除模态差异,还通过利用语义嵌入（如词向量和词相似度）来弥合可见类和未见类的语义不一致。在本文中,我们提出了跨模态自蒸馏方法,从知识蒸馏的角度研究可泛化的特征,无需语义嵌入参与训练。具体而言,我们首先通过传统的知识蒸馏将预训练的图像识别网络的知识迁移到学生网络。然后,通过草图和图像的跨模态相关性,跨模态自蒸馏将上述知识间接地迁移到草图模态的识别上,提升草图特征的判别性和泛化性。为了进一步提升知识在草图模态内的集成和传播,我们进一步地提出草图自蒸馏。通过为数据学习辨别性的且泛化的特征,学生网络消除了模态差异和语义不一致性。我们在三个基准数据集,即Sketchy、TU-Berlin和QuickDraw,进行了广泛的实验,证明了我们提出的跨模态自蒸馏方法与当前方法相比较的优越性。     

Social big data: Recent achievements and new challenges

Big data has become an important issue for a large number of research areas such as data mining, machine learning, computational intelligence, information fusion, the semantic Web, and social networks. The rise of different big data frameworks such as Apache Hadoop and, more recently, Spark, for massive data processing based on the MapReduce paradigm has allowed for the efficient utilisation of data mining methods and machine learning algorithms in different domains. A number of libraries such as Mahout and SparkMLib have been designed to develop new efficient applications based on machine learning algorithms. The combination of big data technologies and traditional machine learning algorithms has generated new and interesting challenges in other areas as social media and social networks. These new challenges are focused mainly on problems such as data processing, data storage, data representation, and how data can be used for pattern mining, analysing user behaviours, and visualizing and tracking data, among others. In this paper, we present a revision of the new methodologies that is designed to allow for efficient data mining and information fusion from social media and of the new applications and frameworks that are currently appearing under the “umbrella” of the social networks, social media and big data paradigms.     

基于虚拟属性学习的文本-图像行人检索方法

文本-图像行人检索旨在从行人数据库中查找符合特定文本描述的行人图像.近年来受到学术界和工业界的广泛关注.该任务同时面临两个挑战:细粒度检索以及图像与文本之间的异构鸿沟.部分方法提出使用有监督属性学习提取属性相关特征,在细粒度上关联图像和文本.然而属性标签难以获取,导致这类方法在实践中表现不佳.如何在没有属性标注的情况下提取属性相关特征,建立细粒度的跨模态语义关联成为亟待解决的关键问题.为解决这个问题,融合预训练技术提出基于虚拟属性学习的文本-图像行人检索方法,通过无监督属性学习建立细粒度的跨模态语义关联.第一,基于行人属性的不变性和跨模态语义一致性提出语义引导的属性解耦方法,所提方法利用行人的身份标签作为监督信号引导模型解耦属性相关特征.第二,基于属性之间的关联构建语义图提出基于语义推理的特征学习模块,所提模块通过图模型在属性之间交换信息增强特征的跨模态识别能力.在公开的文本-图像行人检索数据集CUHK-PEDES和跨模态检索数据集Flickr30k上与现有方法进行实验对比,实验结果表明了所提方法的有效性.     

基于大数据的掌上医疗器械检索平台研究

为了提高掌上医疗器械的信息化检索和管理能力,提出基于大数据的掌上医疗器械检索方法,构建掌上医疗器械检索的大数据分布模型,采用有向图模型构建掌上医疗器械信息库的检索节点分布结构模型,在掌上医疗器械信息库库中进行语义关联规则分析,采用字符串的匹配技术,建立掌上医疗器械信息库检索的模糊决策模型,采用大数据融合方法实现掌上医疗器械检索的算法设计,结合自相关特征匹配方法实现掌上医疗器械信息库的语义特征提取,实现掌上医疗器械检索平台的优化设计。仿真结果表明,采用该方法进行掌上医疗器械检索的智能性较好,检索的查准性较高,时延较低。     

基于语义融合和多重相似性学习的跨模态检索

针对现有跨模态检索方法不能充分挖掘模态之间的相似性信息的问题,提出一种基于语义融合和多重相似性学习（CFMSL）方法。首先,在特征提取过程中融合不同模态的语义信息,加强不同模态特征间的交互,使得模型能够充分挖掘模态间的关联信息。然后,利用生成器将单模态特征和融合模态特征映射到公共子空间中,通过最大化锚点与正例样本之间的相似性和最小化锚点与负例样本间的相似性得到具有判别性的特征进行模态对齐。最后,基于决策融合方式对相似性列表进行重排序,使得最终排序结果同时考虑单模态特征和融合模态特征,提高检索性能。通过在Pascal Sentences、Wikipedia、NUS-WIDE-10K这3个广泛使用的图文数据集上进行实验,实验结果表明CFMSL模型能够有效提高跨模态检索任务的性能。     

Spatial and semantical label inference for social media

Exploring the spatial and semantical knowledge from messages in social media offers us an opportunity to get a deeper understanding about the mobility and activity of users, which can be leveraged to improve the service quality of online applications like recommender systems. In this paper, we investigate the problem of the spatial and semantical label inference, where the challenges come from three aspects: diverse heterogeneous information, uncertainty of individual mobility, and large-scale sparse data. We address the challenges by exploring two types of data fusion, the fusion of heterogeneous social networks and the fusion of heterogeneous features. We build a 4-dimensional tensor, called spatial–temporal semantical tensor (STST), to model the individual mobility and activity by fusing two heterogeneous social networks, a social media network and a location-based social network (LBSN). To address the challenge arising from diverse heterogeneous information and the uncertainty of individual mobility, we construct three types of heterogeneous features and fuse them with STST by exploring their interdependency relationships. Particularly, a spatial tendency feature is constructed to constrain the inference of individual mobility and reduce the uncertainty. To deal with large-scale sparse data, we propose a parallel contextual tensor factorization (PCTF) to concurrently factorize STST. Finally, we integrate these components into an inference framework, called spatial and semantical label inference SSLI. The results of extensive experiments conducted on real datasets and synthetic datasets verify the effectiveness and efficiency of SSLI.     

跨模态检索技术研究综述

跨模态检索可以通过一种模态检索出其他模态的信息,已经成为大数据时代的研究热点。研究者基于实值表示和二进制表示两种方法来减小不同模态信息的语义差距并进行有效的相似度对比,但仍会有检索效率低或信息丢失的问题。目前,如何进一步提高检索效率和信息利用率是跨模态检索研究面临的关键挑战。介绍了跨模态检索研究中基于实值表示和二进制表示两种方法的发展现状;分析对比了包含两种表示技术下以建模技术和相似性对比为主线的五种跨模态检索方法：子空间学习、主题统计模型学习、深度学习、传统哈希和深度哈希;对最新的多模态数据集进行总结,为相关的研究和工程人员提供有价值的参考资料;分析了跨模态检索面临的挑战并指出了该领域未来研究方向。     

深度学习跨模态图文检索研究综述

随着深度神经网络的兴起,多模态学习受到广泛关注.跨模态检索是多模态学习的重要分支,其目的在于挖掘不同模态样本之间的关系,即通过一种模态样本来检索具有近似语义的另一种模态样本.近年来,跨模态检索逐渐成为国内外学术界研究的前沿和热点,是信息检索领域未来发展的重要方向.首先,聚焦于深度学习跨模态图文检索研究的最新进展,对基于...     

异质媒体分析技术研究进展

在异质媒体应用迅速兴起,线上内容和线下服务对网络用户影响日益深刻的背景下,介绍了异质媒体分析的相关概念和方法,对异质媒体的多源自然属性和社会属性进行有效感知,揭示海量异质媒体的语义多样性、复杂关联和内在信息传播机制。文章主要内容涵盖以下几方面:首先,讨论异质媒体数据的跨平台、多模态和来源广泛等特性及其带来的挑战和机遇,介绍异质媒体分析技术的特点和传统单一媒体分析的不同之处,以及异质媒体研究可能带来的科学和社会影响力;其次,分别从异质媒体语义分析与理解、异质媒体关联建模和异质媒体社群分析等三个方面介绍异质媒体分析技术的国内外研究现状;最后,介绍作者及所在研究团队在异质语义分析理解,异质媒体中热点事件和话题分析以及异质媒体用户行为分析等方面的最新研究成果。     

Sparse semantic metric learning for image retrieval

Typical content-based image retrieval solutions usually cannot achieve satisfactory performance due to the semantic gap challenge. With the popularity of social media applications, large amounts of social images associated with user tagging information are available, which can be leveraged to boost image retrieval. In this paper, we propose a sparse semantic metric learning (SSML) algorithm by discovering knowledge from these social media resources, and apply the learned metric to search relevant images for users. Different from the traditional metric learning approaches that use similar or dissimilar constraints over a homogeneous visual space, the proposed method exploits heterogeneous information from two views of images and formulates the learning problem with the following principles. The semantic structure in the text space is expected to be preserved for the transformed space. To prevent overfitting the noisy, incomplete, or subjective tagging information of images, we expect that the mapping space by the learned metric does not deviate from the original visual space. In addition, the metric is straightforward constrained to be row-wise sparse with the ?_2,1-norm to suppress certain noisy or redundant visual feature dimensions. We present an iterative algorithm with proved convergence to solve the optimization problem. With the learned metric for image retrieval, we conduct extensive experiments on a real-world dataset and validate the effectiveness of our approach compared with other related work.     

语义相似性保持的判别式跨模态哈希

针对跨模态哈希检索方法中存在标签语义利用不充分,从而导致哈希码判别能力弱、检索精度低的问题,提出了一种语义相似性保持的判别式跨模态哈希方法.该方法将异构模态的特征数据投影到一个公共子空间,并结合多标签核判别分析方法将标签语义中的判别信息和潜在关联嵌入到公共子空间中;通过最小化公共子空间与哈希码之间的量化误差提高哈希码的判别能力;此外,利用标签构建语义相似性矩阵,并将语义相似性保留到所学的哈希码中,进一步提升哈希码的检索精度.在LabelMe、MIRFlickr-25k、NUS-WIDE三个基准数据集上进行了大量实验,其结果验证了该方法的有效性.     

面向海洋的多模态智能计算：挑战、进展和展望

海洋是高质量发展的要地,海洋科学大数据的发展为认知和经略海洋带来机遇的同时也引入了新的挑战。海洋科学大数据具有超多模态的显著特征,目前尚未形成面向海洋领域特色的多模态智能计算理论体系和技术框架。因此,本文首次从多模态数据技术的视角,系统性介绍面向海洋现象/过程的智能感知、认知和预知的交叉研究进展。首先,通过梳理海洋科学大数据全生命周期的阶段演进过程,明确海洋多模态智能计算的研究对象、科学问题和典型应用场景。其次,在海洋多模态大数据内容分析、推理预测和高性能计算3个典型应用场景中展开现有工作的系统性梳理和介绍。最后,针对海洋数据分布和计算模式的差异性,提出海洋多模态大数据表征建模、跨模态关联、推理预测以及高性能计算4个关键科学问题中的挑战,并提出未来展望。