跨模态检索技术研究综述

跨模态检索可以通过一种模态检索出其他模态的信息,已经成为大数据时代的研究热点。研究者基于实值表示和二进制表示两种方法来减小不同模态信息的语义差距并进行有效的相似度对比,但仍会有检索效率低或信息丢失的问题。目前,如何进一步提高检索效率和信息利用率是跨模态检索研究面临的关键挑战。介绍了跨模态检索研究中基于实值表示和二进制表示两种方法的发展现状;分析对比了包含两种表示技术下以建模技术和相似性对比为主线的五种跨模态检索方法：子空间学习、主题统计模型学习、深度学习、传统哈希和深度哈希;对最新的多模态数据集进行总结,为相关的研究和工程人员提供有价值的参考资料;分析了跨模态检索面临的挑战并指出了该领域未来研究方向。     

Cross-modal face identity aftereffects and their relation to priming.

We tested the magnitude of the face identity aftereffect following adaptation to different modes of adaptors in four experiments. The perceptual midpoint between two morphed famous faces was measured pre- and post-adaptation. Significant aftereffects were observed for visual (faces) and nonvisual adaptors (voices and names) but not nonspecific semantic information (e.g., occupations). Aftereffects were also observed following imagination and adaptation to an associated person. The strongest aftereffects were found adapting to facial caricatures. These results are discussed in terms of cross-modal adaptation occurring at various loci within the face-recognition system analogous to priming. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

基于毫米波雷达点云和视觉信息差异性特征注意力融合的3D目标检测

自动驾驶中传感器融合是感知系统的重要组成部分,雷达点云信息和视觉信息融合可以提高车辆的感知能力。然而现有的研究将雷达点投影到图像上时只是对雷达点简单的增加高度,无法提供更加准确的横向信息,缺乏空间信息。同时对两个模态只是进行简单的融合,虽然产生了一个联合表征,但不足以充分捕捉两种模态之间的复杂联系。文中同时增加了雷达点云的宽度来进行空间信息增强,另外设计了一种利用差异性特征注意力融合的方法,使两个模态进行跨模态交互融合。本文在具有挑战性的nuScenes数据集上对模型进行了评估,提出的模型的NDS评分和mAP分别达到了46.3%和33.9%,体现了优秀的性能。     

红外—可见光跨模态的行人检测综述

可见光图像在光照充足的条件下可以提供一系列辅助检测行人的信息,如颜色和纹理等信息,但在低照度场景下表现并不理想。红外图像虽然不能提供颜色和纹理信息,但红外图像根据热辐射差异成像而不依赖于光照条件这一特性,使其可以在低照度场景下有效区分行人区域与背景区域并提供清晰的行人轮廓信息。由于红外和可见光两种模态之间直观的互补性,同时使用红外和可见光图像的行人检测任务被认为是一个很有前景的研究方向,受到了广泛关注,大幅促进了在安防（如安全监控和自动驾驶）和疫情防控等领域应用的发展。本文对红外—可见光跨模态的行人检测工作进行全面梳理,并对未来方向进行深入思考。首先,该课题具有独特性质。可见光图像对应三通道的颜色信息而红外图像对应单通道的温差信息,如何在两种模态存在本质差异的前提下,充分利用二者的互补性是红外—可见光跨模态行人检测领域的核心挑战和主要任务。其次,近几年红外—可见光跨模态行人检测研究针对的问题可分为两类,即模态差异大和实际应用难。针对模态差异大的问题,可分为图像未对准和融合不充分两类问题。针对实际应用难的问题,又分为标注成本、实时检测和硬件成本3类问题。本文依次对跨模态行人检测的主要研究...     

Olfactory-induced synesthesias: A review and model.

Recent reviews of synesthesia concentrate upon rare neurodevelopmental examples and exclude common olfactory-induced experiences with which they may profitably be compared. Like the neurodevelopmental synesthesias, odor-induced experiences involve different sensory modalities; are reliable, asymmetric (concurrents cannot induce), and automatic; and the inducer-concurrent relationship is learnt. Unlike neurodevelopmental synesthesias, these experiences are universal and their synesthetic nature goes unrecognized. Olfaction's ability to universally induce concurrents may result from its unique neuroanatomy, affording dual access to neocortex. We propose that concurrents arise here via a twofold process: by direct neocortical activation, which recovers a configural memory, and by attribution of this memory to the olfactory modality by thalamic attentional processes. The implications of this for other forms of synesthesia are then examined. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

“三维视觉—语言”推理技术的前沿研究与最新趋势

三维视觉推理的核心思想是对点云场景中的视觉主体间的关系进行理解。非专业用户难以向计算机传达自己的意图,从而限制了该技术的普及与推广。为此,研究人员以自然语言作为语义背景和查询条件反映用户意图,进而与点云的信息进行交互以完成相应的任务。此种范式称做“三维视觉—语言”推理,在自动驾驶、机器人导航以及人机交互等众多领域广泛应用,已经成为计算机视觉领域中备受瞩目的研究方向。过去几年间,“三维视觉—语言”推理技术迅猛发展,呈现出百花齐放的趋势,但是目前依然缺乏对最新研究进展的全面总结。本文聚焦于两类最具代表性的研究工作,锚框预测和内容生成类的“三维视觉—语言”推理技术,系统性概括领域内研究的最新进展。首先,本文总结了“三维视觉—语言”推理的问题定义和现存挑战,同时概述了一些常见的骨干网络。其次,本文按照方法所关注的下游场景,对两类“三维视觉—语言”推理技术做了进一步细分,并深入探讨了各方法的优缺点。接下来,本文对比分析了各类方法在不同基准数据集上的性能。最后,本文展望了“三维视觉—语言”推理技术的未来发展前景,以期促进该领域的深入研究与广泛应用。     

基于锚点的无监督跨模态哈希算法

基于图的无监督跨模态哈希学习具有存储空间小、检索效率高等优点, 受到学术界和工业界的广泛关注, 已成为跨模态检索不可或缺的工具之一. 然而, 图构造的高计算复杂度阻碍其应用于大规模多模态应用. 主要尝试解决基于图的无监督跨模态哈希学习面临的两个重要挑战: 1)在无监督跨模态哈希学习中如何高效地构建图? 2)如何解决跨模态哈希学习中的离散值优化问题? 针对这两个问题, 分别提出基于锚点图的跨模态学习和可微分哈希层. 具体地, 首先从训练集中随机地选择若干图文对作为锚点集, 利用该锚点集作为中介计算每批数据的图矩阵, 以该图矩阵指导跨模态哈希学习, 从而能极大地降低空间与时间开销; 其次, 提出的可微分哈希层可在网络前向传播时直接由二值编码计算, 在反向传播时亦可产生梯度进行网络更新, 而无需连续值松弛, 从而具有更好的哈希编码效果; 最后, 引入跨模态排序损失, 使得在训练过程中考虑排序结果, 从而提升跨模态检索正确率. 通过在3个通用数据集上与10种跨模态哈希算法进行对比, 验证了提出算法的有效性.     

基于关联特征传播的跨模态检索

深度学习的快速发展和关联学习的深度研究,使得跨模态检索的性能有了很大提升.跨模态检索研究面临的挑战是：不同模态的数据在高层语义上具有关联关系,但在底层特征上存在异构鸿沟.现有方法主要通过单个相关性约束将不同模态的特征映射到具有一定相关性的特征空间中来解决底层特征上的异构鸿沟问题.然而,表征学习表明,不同层次的特征在帮助模型最终性能的提升上都会起作用.所以,现有方法学习到的单一特征空间的关联性是弱的,即该特征空间可能不是最优的检索空间.为解决该问题,提出了基于关联特征传播的跨模态检索模型,其基本思想是强化深度网络各层之间的关联性,即前一层具有一定关联的特征经过非线性变化传到后一层,有利于找到使2种模态关联性更强的特征空间.通过在Wikipedia,Pascal数据集上的大量实验验证得到,该方法提升了平均精度均值.     

The effect of voice onset time differences on lexical access in dutch.

Effects on spoken-word recognition of prevoicing differences in Dutch initial voiced plosives were examined. In 2 cross-modal identity-priming experiments, participants heard prime words and nonwords beginning with voiced plosives with 12, 6, or 0 periods of prevoicing or matched items beginning with voiceless plosives and made lexical decisions to visual tokens of those items. Six-period primes had the same effect as 12-period primes. Zero-period primes had a different effect, but only when their voiceless counterparts were real words. Listeners could nevertheless discriminate the 6-period primes from the 12- and 0-period primes. Phonetic detail appears to influence lexical access only to the extent that it is useful: In Dutch, presence versus absence of prevoicing is more informative than amount of prevoicing. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Cross-modal illusory conjunctions between vision and touch.

Cross-modal illusory conjunctions (ICs) happen when, under conditions of divided attention, felt textures are reported as being seen or vice versa. Experiments provided evidence for these errors, demonstrated that ICs are more frequent if tactile and visual stimuli are in the same hemispace, and showed that ICs still occur under forced-choice conditions but do not occur when attention to the felt texture is increased. Cross-modal ICs were also found in a patient with parietal damage even with relatively long presentations of visual stimuli. The data are consistent with there being cross-modal integration of sensory information, with the modality of origin sometimes being misattributed when attention is constrained. The empirical conclusions from the experiments are supported by formal models. (PsycINFO Database Record (c) 2010 APA, all rights reserved)