基于特征融合的边缘引导乳腺超声图像分割方法

针对乳腺超声图像边缘模糊、斑点噪声多、对比度低等问题,提出了一种融合多特征的边缘引导多尺度选择性核U-Net(Edge-guided Multi-scale Selective Kernel U-Net, EMSK U-Net)方法。EMSK U-Net采用基于U-Net的对称编解码结构可以适应小数据集医学图像分割的特点,将扩张卷积与传统卷积构成选择性核模块作用于编码路径,并提取下采样过程中的选择性核特征进行边缘检测任务,在丰富图像空间信息的同时细化边缘信息,有效缓解斑点噪声和边缘模糊的问题,在一定程度上可以提升小目标的检测精度。然后在解码路径通过多尺度特征加权聚合获取丰富的深层语义信息,多种信息之间相互补充,从而提升网络的分割性能。在3个公开的乳腺超声图像数据集上的实验结果表明,与其他分割方法相比,EMSK U-Net算法各项指标表现良好,分割性能有显著提升。     

改进U-Net的高分辨率遥感图像轻量化分割

针对传统图像分割方法分割效率低下, 遥感图像特征复杂多样, 复杂场景下分割性能受到限制等问题, 在基于U-Net网络架构的基础上, 提出一种能够较好提取遥感图像特征并兼顾效率的改进U-Net模型. 首先, 以EfficientNetV2作为U-Net的编码网络, 增强特征提取能力, 提高训练和推理效率, 然后在解码部分使用卷积结构重参数化方法并结合通道注意力机制, 几乎不增加推理时间的前提下提升网络性能, 最后结合多尺度卷积融合模块, 提高网络对不同尺度目标的特征提取能力和更好地结合上下文信息. 实验表明, 改进的网络在遥感图像分割性能提升的同时分割效率也提高.     

面向医学图像分割的超像素U-Net网络设计

近年来,超像素在医学图像处理领域的应用愈加广泛,现有的方法取得了较好的效果,如LAW, SLIC等.然而,这些方法在处理医学图像得到超像素时,位于组织边缘像素点的划分仍存在类别模糊问题.为此,提出一种基于U-Net网络的超像素分割方法.首先,通过双边滤波模型过滤外部噪声,增强超像素信息;然后,结合U-Net卷积网络学习图像特征.该方法为U-Net网络中每个特征尺度的卷积层后嵌入一个规范层,用于增强网络对参数的敏感性.实验结果表明,该方法有效提高了医学图像超像素的分割精度,与groundtruth相比,其改善了超像素边缘分类的准确性,优化了超像素分割结果,在精确度、召回率、F-measure和分割速度等性能指标上均取得了更好的效果.     

模糊C-均值聚类法在医学图像分析中的应用

主要针对医学图像提出了基于模糊均值聚类的改进算法和应用.该方法分为3步,第1步是像素的模糊化,通过模糊期望值构造冗余图像;第2步是通过冗余图像和原始图像进行聚类分割;第3步是三维显示.由于利用冗余图像增加了每个像素的特征量,该算法增强了聚类分割的精确度.同时,还给出了应用自行开发的三维医学图像处理与分析系统对多种医学图像(包括CT、螺旋CT和MRI)的处理结果.由于对薄骨和关节接合处骨骼的较好识别,使其重建后的三维模型可以清晰地再现解剖结构,取得了较好的效果.     

基于MITK的血管三维重建

利用二维血管内超声图像序列重建三维血管模型,并对三维模型进行虚拟剖切,可以方便地看到内部组织,便于观察和诊断.针对血管内超声图像亮度变化小、形状特征不明显和图像分割效果不好等问题,基于MITK平台,采用光线投射算法对二维超声图像序列进行体绘制三维重建.对重建模型进行旋转、缩放和任意平面裁剪等交互操作,裁剪掉一部分无关体素,有助于医生观察血管的内部结构和细节信息.此外,通过调节体素的阻光度值,可以得到层次清晰的三维血管模型.     

结合形态学重建和超像素的多特征FCM分割算法

针对现有模糊聚类分割算法对噪声的鲁棒性差且提取的图像特征不充分等问题,本文提出了一种结合形态学重建和超像素的多特征模糊C-均值(FCM)分割算法.首先,利用形态学闭合重建处理原图像,提高了算法的鲁棒性和细节保护能力.其次,采用Mean-Shift方法预分割重建图像,获得一组超像素区域.再次,提取重建图像各像素的颜色特征、纹理特征和梯度特征,利用平均策略定义各超像素的颜色特征、纹理特征和梯度特征,组成多维特征向量.最后,运用最大熵正则化的加权模糊C-均值算法(EWFCM)的框架,以超像素为单位,以核诱导距离作为距离度量来聚类多维特征向量.选取BSDS300数据集中的6幅图像完成实验对比.结果表明,本文算法具有更高的分割精度.     

基于改进U-Net的宫颈细胞核图像分割

原始的U-Net采用跳跃结构结合高低层的图像信息,使得U-Net模型有良好的分割效果,但是分割结果在宫颈细胞核边缘依然存在分割欠佳、过分割和欠分割等不足.由此提出了改进型U-Net网络图像分割方法.首先将稠密连接的DenseNet引入U-Net的编码器部分,以解决编码器部分相对简单,不能提取相对抽象的高层语义特征.然后对二元交叉熵损失函数中的宫颈细胞核和背景给予不同的权重,使网络更加注重细胞核特征的学习.最后在池化操作过程中,对池化域内的像素值分配合理的权值,解决池化层丢失信息的问题.实验证明,改进型U-Net网络使宫颈细胞核分割效果更好,模型也越鲁棒,过分割和欠分割比率也越少.显然,改进型U-Net是更有效的图像分割方法.     

融合注意力和多尺度特征的街景图像语义分割

为了解决在街道场景图像语义分割任务中传统U-Net网络在多尺度类别下目标分割的准确率较低和图像上下文特征的关联性较差等问题,提出一种改进U-Net的语义分割网络AS-UNet,实现对街道场景图像的精确分割.首先,在U-Net网络中融入空间通道挤压激励(spatial and channel squeeze&excitation block, scSE)注意力机制模块,在通道和空间两个维度来引导卷积神经网络关注与分割任务相关的语义类别,以提取更多有效的语义信息;其次,为了获取图像的全局上下文信息,聚合多尺度特征图来进行特征增强,将空洞空间金字塔池化(atrous spatial pyramid pooling, ASPP)多尺度特征融合模块嵌入到U-Net网络中;最后,通过组合使用交叉熵损失函数和Dice损失函数来解决街道场景目标类别不平衡的问题,进一步提升分割的准确性.实验结果表明,在街道场景Cityscapes数据集和Cam Vid数据集上AS-UNet网络模型的平均交并比(mean intersection over union, MIo U)相较于传统U-Net网络分别提...     

U-Net网络医学图像分割应用综述

病灶精确分割对患者病情评估和治疗方案制定有重要意义,由于医学图像中病灶与周围组织的对比度低,同一疾病病灶边缘和形状存在很大差异,从而增加了分割难度。U-Net是近些年深度学习研究中的热点,为医生提供了一致性的量化病灶方法,一定程度上提高了分割性能,广泛应用于医学图像语义分割领域。本文对U-Net网络进行全面综述。阐述U-Net网络的基本结构和工作原理;从编码器个数、多个U-Net级联、与U-Net结合的其他模型以及3D U-Net等方面对U-Net网络模型的改进进行总结;从卷积操作、下采样操作、上采样操作、跳跃连接、模型优化策略和数据增强等方面对U-Net网络结构改进进行总结;从残差思想、密集思想、注意力机制和多机制组合等方面对U-Net的改进机制进行总结;对U-Net网络未来的发展方向进行展望。本文对U-Net网络的原理、结构和模型进行详细总结,对U-Net网络的发展具有一定积极意义。     

基于U-Net改进的多尺度融合超声神经分割算法研究

大量传统的颈部超声神经检测算法,检测敏感性低,假阳性数量大,低层特征利用率不足。而颈部超声图像数量较少,边缘模糊且对噪声敏感。对此,提出一种改进型U-Net分支融合算法：改进损失函数,获得高质量的候选样本;使用多尺度卷积结构替换原结构中普通卷积层,增强特征提取能力;结合扩张卷积替换中、深层池化操作,提高低层特征利用率。通过对比实验验证了所提算法的算法性能。实验表明,与传统的U-Net和SegNet卷积网络对于小尺寸超声神经分割的结果相比,所提算法的分割效果较两者分别提升了近9%和17%,且对于正常尺寸和小尺寸的神经分割均有较高的分割精度。     

A computational model of bounded developable surfaces with application to image‐based three‐dimensional reconstruction

Developable surfaces have been extensively studied in computer graphics because they are involved in a large body of applications. This type of surfaces has also been used in computer vision and document processing in the context of three‐dimensional (3D) reconstruction for book digitization and augmented reality. Indeed, the shape of a smoothly deformed piece of paper can be very well modeled by a developable surface. Most of the existing developable surface parameterizations do not handle boundaries or are driven by overly large parameter sets. These two characteristics become issues in the context of developable surface reconstruction from real observations. Our main contribution is a generative model of bounded developable surfaces that solves these two issues. Our model is governed by intuitive parameters whose number depends on the actual deformation and including the “flat shape boundary”. A vast majority of the existing image‐based paper 3D reconstruction methods either require a tightly controlled environment or restricts the set of possible deformations. We propose an algorithm for reconstructing our model's parameters from a general smooth 3D surface interpolating a sparse cloud of 3D points. The latter is assumed to be reconstructed from images of a static piece of paper or any other developable surface. Our 3D reconstruction method is well adapted to the use of keypoint matches over multiple images. In this context, the initial 3D point cloud is reconstructed by structure‐from‐motion for which mature and reliable algorithms now exist and the thin‐plate spline is used as a general smooth surface model. After initialization, our model's parameters are refined with model‐based bundle adjustment. We experimentally validated our model and 3D reconstruction algorithm for shape capture and augmented reality on seven real datasets. The first six datasets consist of multiple images or videos and a sparse set of 3D points obtained by structure‐from‐motion. The last dataset is a dense 3D point cloud acquired by structured light. Our implementation has been made publicly available on the authors' web home pages. Copyright © 2012 John Wiley & Sons, Ltd.     

The road to surgical simulation and surgical navigation

The recent advantage of the power of graphic workstations has made it possible to handle 3D human structures in an interactive way. Real-time imaging of medical 3D or 4D images can be used not only for diagnosis, but also for various novel medical treatments. By elaborating on the history of the establishment of our laboratory, which focuses on medical virtual reality, we describe our experience of developing surgery simulation and surgery navigation systems according to our research results. In the case of surgical simulation, we mention two kinds of virtual surgery simulators that produce the haptic sensation of surgical maneuvers in the user’s fingers. Regarding surgical navigation systems, we explain the necessity of the augmented reality function for the encouragement of the ability of robotic surgery and its trial for clinical case.     

多尺度融合增强的纵膈淋巴结超声弹性图像分割

目的 支气管超声弹性成像具有丰富的通道语义信息,精准的分割纵膈淋巴结对诊断肺癌是否转移具有重要意义,也对癌症的分期和治疗有着重要作用。目前,超声弹性图像分割研究较少,没有充分挖掘图像通道特征之间的关系。因此,提出一种结合注意力机制的多尺度融合增强的纵膈淋巴结超声弹性图像分割U-Net(attention-based multi-scale fusion enhanced ultrasound elastic images segmentation network for mediastinal lymph node, AMFE-UNet)。方法首先,考虑到图像可以提供纵膈淋巴结的位置和通道信息,设计密集卷积网络（dense convolutional network,DenseNet）作为模型编码器;其次,结合注意力机制和空洞卷积设计多尺度融合增强解码器,从多尺度和范围对结节的边界和纹理进行建模;最后,用选择性内核网络设计跳跃连接,将编码器的中间特征与解码器的输出特征充分融合。根据解码器特征进行数值或通道融合的方式不同,将AMFE-UNet分为A和B两个子型。结果 在超声弹性图像数据集...     

Real-time advanced spinal surgery via visible patient model and augmented reality system

This paper presents an advanced augmented reality system for spinal surgery assistance, and develops entry-point guidance prior to vertebroplasty spinal surgery. Based on image-based marker detection and tracking, the proposed camera-projector system superimposes pre-operative 3-D images onto patients. The patients’ preoperative 3-D image model is registered by projecting it onto the patient such that the synthetic 3-D model merges with the real patient image, enabling the surgeon to see through the patients’ anatomy. The proposed method is much simpler than heavy and computationally challenging navigation systems, and also reduces radiation exposure. The system is experimentally tested on a preoperative 3D model, dummy patient model and animal cadaver model. The feasibility and accuracy of the proposed system is verified on three patients undergoing spinal surgery in the operating theater. The results of these clinical trials are extremely promising, with surgeons reporting favorably on the reduced time of finding a suitable entry point and reduced radiation dose to patients.     

InstanceFusion: Real-time Instance-level 3D Reconstruction Using a Single RGBD Camera

We present InstanceFusion, a robust real-time system to detect, segment, and reconstruct instance-level 3D objects of indoor scenes with a hand-held RGBD camera. It combines the strengths of deep learning and traditional SLAM techniques to produce visually compelling 3D semantic models. The key success comes from our novel segmentation scheme and the efficient instance-level data fusion, which are both implemented on GPU. Specifically, for each incoming RGBD frame, we take the advantages of the RGBD features, the 3D point cloud, and the reconstructed model to perform instance-level segmentation. The corresponding RGBD data along with the instance ID are then fused to the surfel-based models. In order to sufficiently store and update these data, we design and implement a new data structure using the OpenGL Shading Language. Experimental results show that our method advances the state-of-the-art (SOTA) methods in instance segmentation and data fusion by a big margin. In addition, our instance segmentation improves the precision of 3D reconstruction, especially in the loop closure. InstanceFusion system runs 20.5Hz on a consumer-level GPU, which supports a number of augmented reality (AR) applications (e.g., 3D model registration, virtual interaction, AR map) and robot applications (e.g., navigation, manipulation, grasping). To facilitate future research and reproduce our system more easily, the source code, data, and the trained model are released on Github: https://github.com/Fancomi2017/InstanceFusion .     

基于多尺度特征融合的改进臂丛神经分割方法

臂丛神经超声影像信噪比（SNR）低、边缘模糊且人工分割难度较大。现有的分割模型虽然取得了一些成果,但碍于臂丛神经结构目标区域小、形状不规则,分割效果欠佳。针对上述问题,设计基于多尺度特征融合的臂丛神经分割模型,即针对神经部位分割的特征金字塔网络（Ner-FPN）。在特征提取阶段,设计一种仿Xception的结构进行多尺度特征提取;在预测分割阶段,采用双向FPN结构进行特征融合预测。在Kaggle臂丛神经超声影像分割竞赛的BP数据集上的实验结果表明,Ner-FPN模型对臂丛神经分割的Dice相似系数（DSC）可达0.703,与主流的深度学习分割模型U-Net、SegNet相比,分别提高了10.7个百分点和14.5个百分点,对比相同数据集中的其他改进模型QU-Net和Efficient+U-Net,DSC分别提高了5.5个百分点和3.4个百分点,可见所提模型能够起到辅助诊断的效果。     

语义驱动下水面场景中AR效果提升方法

增强现实应用场景不断拓展,但水面检测领域由于物理光学的反射等特性的影响制约了水面增强现实的应用。对此提出一种语义驱动下的基于ORB-SLAM2系统在水面场景下进行实时增强现实效果提升的方法。将视频帧传入深度学习语义分割网络ICNet中,把分割后得到的标签图和原视频帧一同输入SLAM系统的前端进行追踪及地图构建。在语义的指导下,水域内拟合平面并根据水面反射原理将3D模型放置在平面的同时,在关于平面镜面对称的位置生成其倒影。倒影的颜色依据提出的混色模型进行渲染。实验结果表明,在户外水面场景下增强现实的效果得到了提升,虚拟物体与真实物体视觉一致性也更为连贯。     

Deep learning-based smart task assistance in wearable augmented reality

Wearable augmented reality (AR) smart glasses have been utilized in various applications such as training, maintenance, and collaboration. However, most previous research on wearable AR technology did not effectively supported situation-aware task assistance because of AR marker-based static visualization and registration. In this study, a smart and user-centric task assistance method is proposed, which combines deep learning-based object detection and instance segmentation with wearable AR technology to provide more effective visual guidance with less cognitive load. In particular, instance segmentation using the Mask R-CNN and markerless AR are combined to overlay the 3D spatial mapping of an actual object onto its surrounding real environment. In addition, 3D spatial information with instance segmentation is used to provide 3D task guidance and navigation, which helps the user to more easily identify and understand physical objects while moving around in the physical environment. Furthermore, 2.5D or 3D replicas support the 3D annotation and collaboration between different workers without predefined 3D models. Therefore, the user can perform more realistic manufacturing tasks in dynamic environments. To verify the usability and usefulness of the proposed method, we performed quantitative and qualitative analyses by conducting two user studies: 1) matching a virtual object to a real object in a real environment, and 2) performing a realistic task, that is, the maintenance and inspection of a 3D printer. We also implemented several viable applications supporting task assistance using the proposed deep learning-based task assistance in wearable AR.     

Acquisition of three-dimensional information of brain structures using endoneurosonography

Recent trends in minimally invasive brain surgery aim at using the joint acquisition of endoscopic and ultrasound images, a technique called endoneurosonography (ENS). Endoscopic images are useful, but they cannot provide information beyond opaque structures; therefore, it is necessary to complement them with other medical images. In this sense, ultrasound images provide a good option because they are cheap and easy to acquire in intraoperative scenarios.This paper presents a methodology for acquiring 3D information on brain structures using endoneurosonography. The endoscopic cameras and the ultrasound sensor are inserted into the brain during surgery. The information from the endoscopic cameras is used for stereo reconstruction, while the ultrasound information is used to segment regions of interest. The aim of this work is solving four specific problems: calibration of the endoscopic system; tracking of the ultrasound probe through endoscopic images; segmentation of brain structures from ultrasound images; and 3D reconstruction of brain tumors. At the end of the paper, the complete methodology is explained and experiments are done on an ENS database of brain phantoms.