跨模LDA融合的多模态数据主题分析方法

随着互联网的高速发展,社会大众可以通过网络对医疗事件以及医患关系自由地发表个人意见和观点言论,这对于引导公众正确的价值导向有着重大研究意义.然而,仅考虑单模态数据的主题分析算法不能精准地把握整个舆情事件的真相,存在主题提取不准确、个人情感先入为主等问题.提出一种基于LDA的多模态数据主题分析算法MD＿LDA(multimodal data topic analysis based on LDA).通过对各模态主题分析结果进行决策级融合来计算多模态的主题分析结果,进而解决传统方法对多模态数据考虑不全面的缺陷.实验结果表明,针对多模态舆情事件,在主题词的提取效果上,所提出的MD＿LDA算法优于单一模态数据进行主题分析的算法.而相对于传统的关键词提取算法TF＿IDF与TextRank和MD＿LDA算法的准确率以及主题词提取效率均有所提高,验证了结合多模态数据进行主题分析的MD＿LDA算法的有效性.     

Finding story chains in newswire articles using random walks

Massive amounts of information about news events are published on the Internet every day in online newspapers, blogs, and social network messages. While search engines like Google help retrieve information using keywords, the large volumes of unstructured search results returned by search engines make it hard to track the evolution of an event. A story chain is composed of a set of news articles that reveal hidden relationships among different events. Traditional keyword-based search engines provide limited support for finding story chains. In this paper, we propose a random walk based algorithm to find story chains. When breaking news happens, many media outlets report the same event. We have two pruning mechanisms in the algorithm to automatically exclude redundant articles from the story chain and to ensure efficiency of the algorithm. We further explore how named entities and word relevance can help find relevant news articles and improve algorithm efficiency by creating a co-clustering based correlation graph. Experimental results show that our proposed algorithm can generate coherent story chains without redundancy. The efficiency of the algorithm is significantly improved on the correlation graph.     

关联模态补偿的视频动作识别算法

随着深度摄像机的发展,不同模态的视频数据更易获得.基于多模态数据的视频动作识别也受到越来越广泛的关注.不同模态的数据能够从多个角度对视频动作进行描述,如何有效地利用多模态数据并形成优势互补是视频动作识别中的重要方向.提出了一种基于关联模态补偿的视频动作识别算法.该方法以RGB和光流场视频数据为源模态,以3D骨架数据为辅助模态,利用源模态和辅助模态高层特征空间的关联性,补偿源模态的特征提取.该算法基于卷积神经网络和长短期记忆网络,对源模态数据和辅助模态数据进行时空域特征建模.在此基础上,提出了基于残差子网络的模态适应模块,通过统一源模态特征和辅助模态特征的数据分布,实现辅助模态对源模态的特征补偿.考虑到源模态数据和辅助模态数据在动作类别或动作样本等方面存在不同程度的对齐情况,设计了多层次模态适应算法,以适应不同的训练数据.所提算法仅在训练过程中需要辅助模态的帮助,在测试过程中可以仅根据源模态数据进行动作的识别,极大地拓展了该算法的实用性.在通用公共数据集上的实验结果表明,相比于现有动作识别算法,该算法取得了更加优越的性能.     

基于要素提取关联词对的中文报道关系检测

针对现有中文报道关系检测的检测代价即误报率和丢失率较高的问题,在多向量空间模型基础上提取不同向量的要素(时间、地点、人物和内容)特征词组成关联词对,使用支持向量机(SVM)方法整合关联词对相似度和余弦相似度,从而提出了一种提取要素关联词对报道关系检测方法。所提方法补充表示了报道内容,为检测提供了更多的比较依据,识别代价降低了将近11%。实验结果验证了算法的有效性。     

多模态融合网络的睡眠分期研究

针对睡眠多导图中各模态信息在睡眠各阶段存在差异性,而导致特征利用不充分的问题,本文提出了一种基于通道注意力机制和多模态门控机制的睡眠分期模型。首先利用残差收缩网络设计各模态特征提取网络用于提取各模态特征,并在通道维度上进行拼接融合,利用通道注意力机制进一步对融合特征进行重标定得到睡眠多导图的时不变特征;之后提出了一种基于自适应门控机制的多模态门控模块,对各模态特征及时不变特征按照重要程度进行加权融合,实现特征融合;最后利用双向长短时记忆网络提取睡眠多导图的时序特征。实验结果表明,本文提出的睡眠分期模型在欧洲数据格式睡眠数据集(sleep-European data format, sleep-EDF)上准确率为87.6%,$ {M_{{F_1}}} $为82.0%,取得了目前最好的分期效果。     

基于社交媒体的事件感知与多模态事件脉络生成

随着信息技术的发展和社交媒体的流行,普通用户已经完成了从信息接受者到信息产生者的转变,每个人都可以实时分享自己身边的信息,也可以转发自己感兴趣的内容,这使得社交媒体的数据量迅速增长。在海量数据中蕴含着丰富的社会事件发生和发展的记录,如何有效地从这些数据中挖掘出有价值的信息成为了当前信息领域的重要问题。针对该问题,介绍了基于社交媒体的事件感知与多模态事件脉络生成。基于社交媒体的事件感知与多模态事件脉络生成旨在通过分析社交媒体中的文本、时间、图像、评论、观点、情感和用户交互等多模态数据,感知事件并刻画事件的关系,从而实现对事件的总结。讨论了基于社交媒体的事件感知与多模态事件脉络生成的描述模型、概念、发展历史、关键技术与挑战以及其广泛的应用领域,综述了社交媒体分析在事件感知和事件总结方面的研究进展,并对其未来发展进行了展望。     

基于贝叶斯信念网络的话题识别模型

针对贝叶斯信念网络应用于话题识别进行了研究, 提出了新的话题识别模型。模型的拓扑结构包括新报道、报道术语、事件术语、话题四层节点, 用弧标明索引关系。在贝叶斯概率和条件独立性假设的基础上, 模型运用条件概率计算新报道和已有话题簇的相似度, 从而实现话题识别。考虑到核心报道、核心事件的重要性, 对不同层次的权重计算进行了调整。实验采用DET曲线评测法对模型性能进行测试, 实验结果显示, 调整后的权重计算可在一定程度上提高新模型的性能, 与向量空间模型相比, 在相同阈值下新模型的漏报率与误报率有所降低。     

AI-enabled and multimodal data driven smart health monitoring of wind power systems: A case study

The development of AI has enabled the fault detection of industrial components to be achieved through the combination with deep learning. A detection method combined with deep learning has also emerged for the fault detection of fan blades, such as models based on neural networks using the appearance or sound of the blades. However, the detection model obtained from a single data type often has limitations, such as low accuracy and overfitting. This is also the problem with fan blade detection. In contrast, multimodal data fusion detection models are often more stable. The modality diversity of blade diagnosis is strong, and it can be achieved from multiple modalities such as image, sound, and vibration. To improve the accuracy of fault diagnosis of fan blades, this article proposes a multimodal double-layer detection system (MTDS) based on decision-level and feature-level fusion. The system includes a wind turbine simulation platform and a multimodal detection system. It mainly obtains different modal data of the simulated wind turbine from the image, sound, and vibration signals, including blade images through unmanned aerial vehicle photography, blade vibration signals through electronic vibrators, and blade sound signals through microphones. The highly correlated sound and vibration modal data are fused at the feature level, and a detection model based on the sound and vibration mixed mode is implemented using a sound-vibration-CNN (SV-CNN) proposed in this case. Then, a detection model of the image mode is trained based on the blade image using a Convolution Block Attention Module ResNet (CBAM-ResNet) network. Finally, the detection input of the two modal models is fed into a perceptron to obtain the final prediction result, and the decision-level fusion is implemented to achieve fan blade detection based on multimodal, namely the implementation of MTDS.     

基于跨模态门控机制和改进融合方法的多模态情感分析

如何有效挖掘单模态表征并实现多模态信息的充分融合是多模态情感分析研究的重点之一。针对多模态情感分析中的模态间噪声和多模态特征融合不充分等问题,提出一种基于跨模态门控机制和改进融合方法的多模态情感分析模型。首先,利用跨模态门控机制去除模态间噪声,提取互补信息以增强模态表示。然后,利用权重和相似约束分别关注不同模态情感贡献的差异性和情感表达的一致性。最后,结合模态的多层次表示获得情感分析的结果。在三个公开数据集上的实验结果表明,所提模型是有效的,相比已有一些模型取得了更好的性能。     

融合多尺度残差和注意力机制的特发性肺纤维化进展预测

目的 特发性肺纤维化（idiopathic pulmonary fibrosis,IPF）是一种致死率极高的肺间质疾病,呈渐进式发展且诊断手段有限,给病情的预后带来极大的困难。目前已有的肺功能衰退严重程度预测方法仍存在准确率较低的问题。方法 提出了一种融合多尺度残差和注意力机制的特发性肺纤维化进展预测模型,包括计算机断层扫描（computed tomography,CT）特征提取网络和多模态特征预测网络,以此预测IPF患者不同周数的用力肺活量（forced vital capacity,FVC）。CT特征提取网络以InceptionV1为骨干网络,添加残差模块和改进的CBAM通道注意力模块（improved channel attention of convolutional block attention module,CBAM-ICA）来扩大网络的感受野,关注肺部区域的有效特征,添加与卷积层并行的空洞卷积模块,补充丢失的细节信息,并对改进的多尺度CT特征融合模块进行两次串联堆叠,提升网络获取CT特征的能力,最终获得更加有效的多尺度CT特征信息。多模态预测网络利用CT特征和临床特征进行融合形成的多模态特征对IPF患者不同周数的FVC值进行预测,提高了网络的预测性能。结果 在OSIC （Open Source Imaging Consortium）提供的肺纤维化测试数据集上的实验结果表明,本文提出的特发性肺纤维化进展预测模型得到了-6.810 7的拉普拉斯对数似然得分,较其他方法具有更好的性能。结论 提出的融合多尺度残差和注意力机制的特发性肺纤维化进展预测模型对预测IPF患者不同周数的FVC值具有一定的有效性,可以帮助医生更好地了解IPF患者肺功能衰退的严重程度,对IPF患者的预后具有指导意义。     

融合扩展信息瓶颈理论的话题关联检测方法研究

话题关联检测的关键任务在于判断给定报道对是否属于同一话题. 现有判断方法往往忽略种子事件与其直接相关事件之间的层次关系．为此,通过分析报道内部语义分布规律及篇章结构,并依据语义分布规则,利用语义分布规律改进信息瓶颈（Information bottleneck,IB）算法,用于子话题逻辑语义单元的划分,并利用这些逻辑语义单元表示报道,进行话题关联检测. 实验证明该方法有较快的收敛速度,并在一定程度上提高了系统性能．     

Requiem for a theory: the ‘ story grammar’ story

Abstract

Story grammar was a fairly hot topic in artificial intelligence (AI) from the mid-1970s to the early 1980s. Now, however, it has virtually disappeared as a research issue in AI. The process of science is one where theories are proposed, experiments are conducted, hypotheses are tested, theories are revised and, ultimately, theories are either accepted or rejected. The story grammar story is surveyed with a view towards describing what it was, and explaining what happened to it, from the perspective of the scientific process.     

Multimodal Data Modeling Technology and Its App-19lication for Cloud-edge-device Collaboration

In the cloud-edge-device collaboration architecture, data types are diverse, and there are differences in storage resources and computing resources at all levels, which brings new challenges to data management. The existing data models or simple superposition of data models are difficult to meet the requirements of multimodal data management and collaborative management in the cloud-edge-device. Therefore, research on multimodal data modeling technology for cloud-edge-device collaboration has become an important issue. The core is how to efficiently obtain the query results that meet the needs of the application from the cloud-edge-device architecture. Starting from the data types of the three-layer data of cloud-edge-device, in this paper we propose a multimodal data modeling technology for cloud-edge-device collaboration, give the definition of multimodal data model based on tuples, and design six base classes to achieve a unified representation of multimodal data. The basic data operation architecture of cloud-edge-device collaborative query is also proposed to meet the query requirements of cloud-edge-device business scenarios. The integrity constraints of the multimodal data model are given, which lays a theoretical foundation for query optimization. Finally, a demonstration application of the multimodal data model for cloud-edge-device collaboration is given, and the proposed data model storage method is verified from the three aspects of data storage time, storage space, and query time. The experimental results show that the proposed scheme can effectively represent the multimodal data in the cloud-edge-device collaboration architecture.     

Multi-level feature optimization and multimodal contextual fusion for sentiment analysis and emotion classification

The availability of the humongous amount of multimodal content on the internet, the multimodal sentiment classification, and emotion detection has become the most researched topic. The feature selection, context extraction, and multi-modal fusion are the most important challenges in multimodal sentiment classification and affective computing. To address these challenges this paper presents multilevel feature optimization and multimodal contextual fusion technique. The evolutionary computing based feature selection models extract a subset of features from multiple modalities. The contextual information between the neighboring utterances is extracted using bidirectional long-short-term-memory at multiple levels. Initially, bimodal fusion is performed by fusing a combination of two unimodal modalities at a time and finally, trimodal fusion is performed by fusing all three modalities. The result of the proposed method is demonstrated using two publically available datasets such as CMU-MOSI for sentiment classification and IEMOCAP for affective computing. Incorporating a subset of features and contextual information, the proposed model obtains better classification accuracy than the two standard baselines by over 3% and 6% in sentiment and emotion classification, respectively.     

基于深度情感唤醒网络的多模态情感分析与情绪识别

随着网络平台上各类图像、视频数据的快速增长,多模态情感分析与情绪识别已成为一个日益热门的研究领域.相比于单模态情感分析,多模态情感分析中的模态融合是一个亟待解决的关键问题.受到认知科学中情感唤起模型的启发,提出一种能够模拟人类处理多通道输入信息机制的深度情感唤醒网络(DEAN),该网络可实现多模态信息的有机融合,既能处理情绪的连贯性,又能避免融合机制的选择不当而带来的问题.DEAN网络主要由以下3部分组成:跨模态Transformer模块,用以模拟人类知觉分析系统的功能;多模态BiLSTM系统,用以模拟认知比较器;多模态门控模块,用以模拟情感唤起模型中的激活结构.在多模态情感分析与情绪识别的3个经典数据集上进行的比较实验结果表明,DEAN模型在各数据集上的性能均超越了目前最先进的情感分析模型.     

基于巡逻小车的人脸遮挡异常事件实时检测

近年来,随着城市化进程的加快,银行、政府、学校等场所的安全越来越成为人们关心的问题,智能监控已成为目前的一个研究热点问题. 本文主要研究室内巡逻小车监控下,面部异常遮挡问题. 本文首先对监控视频进行前景提取;接着,基于提取的前景,进行肩部定位、头部区域椭圆拟合;然后通过肤色检测判断人脸区域;最后,通过Haar检测器检测人脸区域的眼睛和嘴巴,以此来判断是否存在异常遮挡行为. 实验结果表明,本文提出的算法能够实时、有效的检测出面部异常遮挡问题.     

基于动态注意力机制和多模态循环融合的帕金森氏症识别

PD （Parkinson’s disease） 的运动障碍会累及口、咽、腭肌以及面部肌肉,引起声带震颤和面部运动迟缓,为利用声纹和面部特征识别PD患者提供了可能。为了有效利用以上两种特征以提高PD 识别率,提出了基于多尺度特征与动态注意力机制的多模态循环融合模型对患者进行识别检测。首先,设计了多尺度特征提取网络,将高、低层级特征的语义信息融合以得到完整的特征信息;其次,在多尺度特征融合过程中为了充分考虑模态间的相关性和互补性,提出了以不同模态信息互为辅助条件生成注意力特征图的动态注意力机制算法,降低特征融合时信息的冗余;最后设计了多模态循环融合模型,通过计算循环矩阵的每个行向量与特征向量间的哈达玛积得到更有效的融合特征,提高了模型性能。在自建数据集上进行的多组实验结果表明,提出的方法识别准确率高达96.24%,优于当前流行的单模态和多模态识别算法,可以有效区分PD患者和HP （healthy people）,为高效识别PD患者奠定了基础。     

基于词共现和情感元素的突发话题检测算法

随着自媒体的迅速发展,微博中的舆情监控和舆情疏导成为一项重大的研究课题. 为了解决传统话题检测方法对于微博中大数据的分析往往具有复杂度高、实时性低、影响力小等问题,提出一种基于词共现和情感分析的突发话题检测方法. 通过研究微博中情感的突发和共现关系,从而建立情感子空间模型;通过该模型对微博中的信息流进行分类,最后对每个类别中的微博进行主题词提取,实现话题检测的目的. 在NLPIR微博内容语料库上的实验结果表明,该方法能够有效地从大规模微博信息中检测突发新闻,提高突发新闻的识别率.     

基于SGC-LDA模型的财经文本主题研究

传统财经领域研究通常关注结构化数据,较少关注非结构化的财经类文本数据,并且财经文本数据蕴含的信息量巨大。针对上述问题,提出SGC-LDA(sliding-window,genetic factor and common financial topic LDA)财经文本主题研究方法。基于通用财经主题的文本噪声过滤建模,以降低噪声数据的影响;基于滑动窗口技术,同时引入财经遗传因子,保证主题的连续性;完成能够实现财经文本主题模型的SGC-LDA算法。基于真实财经文本的实证研究表明,财经文本主题主要由投资理财、民生时事、商业动态、金融市场、宏观经济、产业经济六个主要部分组成;结合财经主题特征词和财经文本对财经主题的扩充,能够更完整准确地描述其财经主题。同时模型本身表现出一定的去噪能力,且与基准模型的对比分析,也证实了所提出模型在财经主题建模方面优越的分类性能和主题连续性。