A software system for evaluation and training of spatial reasoning and neuroanatomical knowledge in a virtual environment

This paper describes the design and development of a software tool for the evaluation and training of surgical residents using an interactive, immersive, virtual environment. Our objective was to develop a tool to evaluate user spatial reasoning skills and knowledge in a neuroanatomical context, as well as to augment their performance through interactivity. In the visualization, manually segmented anatomical surface images of MRI scans of the brain were rendered using a stereo display to improve depth cues. A magnetically tracked wand was used as a 3D input device for localization tasks within the brain. The movement of the wand was made to correspond to movement of a spherical cursor within the rendered scene, providing a reference for localization. Users can be tested on their ability to localize structures within the 3D scene, and their ability to place anatomical features at the appropriate locations within the rendering.     

Volume cutout

We present a novel method for cutting out 3D volumetric structures based on simple strokes that are drawn directly on volume rendered images. The freehand strokes roughly mark out objects of interest and background. Our system then automatically segments the regions of interest and refines their boundaries in the rendered image. These 2D segmentation results provide constraints for 3D segmentation in the volume dataset. The objects of interest are then efficiently cut out from the volume via a combination of our novel two-pass graph cuts algorithm and the pre-computed over-segmentation. Our method improves traditional, fully automatic segmentation by involving human users in the process, yet minimizing user input and providing timely feedback. Our experiments show that our method extracts volumetric structures precisely and efficiently while requiring little skill or effort from the user.     

A review of atlas-based segmentation for magnetic resonance brain images

Normal and abnormal brains can be segmented by registering the target image with an atlas. Here, an atlas is defined as the combination of an intensity image (template) and its segmented image (the atlas labels). After registering the atlas template and the target image, the atlas labels are propagated to the target image. We define this process as atlas-based segmentation. In recent years, researchers have investigated registration algorithms to match atlases to query subjects and also strategies for atlas construction. In this paper we present a review of the automated approaches for atlas-based segmentation of magnetic resonance brain images. We aim to point out the strengths and weaknesses of atlas-based methods and suggest new research directions. We use two different criteria to present the methods. First, we refer to the algorithms according to their atlas-based strategy: label propagation, multi-atlas methods, and probabilistic techniques. Subsequently, we classify the methods according to their medical target: the brain and its internal structures, tissue segmentation in healthy subjects, tissue segmentation in fetus, neonates and elderly subjects, and segmentation of damaged brains. A quantitative comparison of the results reported in the literature is also presented.     

Automated atlas-based segmentation of NISSL-stained mouse brain sections using supervised learning

The problem of segmentation of mouse brain images into anatomical structures is an important stage of practically every analytical procedure for these images. The present study suggests a new approach to automated segmentation of anatomical structures in the images of NISSL-stained histological sections of mouse brain. The segmentation algorithm is based on the method of supervised learning using the existing anatomical labeling of the corresponding sections from a specialized mouse brain atlas. A mouse brain section to be segmented into anatomical structures is preliminarily associated with a section from the mouse brain atlas displaying the maximum similarity. The image of this section is then preprocessed in order to enhance its quality and to make it as close to the corresponding atlas image as possible. An efficient algorithm of luminance equalization, an extension of the well-known Retinex algorithm is proposed. A random forest is trained on pixel feature vectors constructed based on the atlas section images and the corresponding class labels associated with anatomical structures extracted from the atlas anatomical labeling. The trained classifier is then applied to classify pixels of an experimental section into anatomical structures. A new combination of features based on superpixels and location priors is suggested. Accuracy of the obtained result is increased by using Markov random field. Procedures of luminance equalization and subsequent segmentation into anatomical structures have been tested on real experimental sections.     

Adaptive pixon represented segmentation (APRS) for 3D MR brain images based on mean shift and Markov random fields

In this paper, we proposed an adaptive pixon represented segmentation (APRS) algorithm for 3D magnetic resonance (MR) brain images. Different from traditional method, an adaptive mean shift algorithm was adopted to adaptively smooth the query image and create a pixon-based image representation. Then K-means algorithm was employed to provide an initial segmentation by classifying the pixons in image into a predefined number of tissue classes. By using this segmentation as initialization, expectation-maximization (EM) iterations composed of bias correction, a priori digital brain atlas information, and Markov random field (MRF) segmentation were processed. Pixons were assigned with final labels when the algorithm converges. The adoption of bias correction and brain atlas made the current method more suitable for brain image segmentation than the previous pixon based segmentation algorithm. The proposed method was validated on both simulated normal brain images from BrainWeb and real brain images from the IBSR public dataset. Compared with some other popular MRI segmentation methods, the proposed method exhibited a higher degree of accuracy in segmenting both simulated and real 3D MRI brain data. The experimental results were numerically assessed using Dice and Tanimoto coefficients.     

基于脑图谱和模糊聚类的磁共振图像分割标注

利用TT Atlas中丰富的结构信息,文章提出了一种自动分割脑MRI(magnetic resonance image)图像的方法.这种方法可分为两步.首先,将MRI图像和TT Atlas配准,通过图像和医学图谱的匹配,利用图谱中结构信息的先验知识,就可以对图像作初步的分割标注.然后,利用这个预分割的模板对MRI图像进行模糊聚类分割,从而提高分割的精度.为了自动地将预模板中的结构信息用于分割,文章还提出了一种引入形状因子的FCM聚类算法.除了在匹配时需要手工定出一些点之外,该方法基本上是自动的.     

CT系列图像的三维层次化子块生长分割

准确地从CT系列图像提取感兴趣的组织是手术规划的基础,针对肝脏轮廓分割存在分割不全的问题,提出了基于三维区域生长算法的腹部CT图像分割方法。算法首先由用户选择若干个生长点,然后充分利用CT系列图像层间的相似性,提出基于子块的改进区域生长算法,实现三维的层次化子块区域生长,以更准确提取肝脏区域,其中生长准则由系统分析用户选择的生长点的邻域子块属性获得,以减少用户的干预。实验结果表明,算法能在较少的干预下快速分割出来CT系列图像中的肝脏轮廓。     

3D anatomical shape atlas construction using mesh quality preserved deformable models

3D anatomical shape atlas construction has been extensively studied in medical image analysis research, owing to its importance in model-based image segmentation, longitudinal studies and populational statistical analysis, etc. Among multiple steps of 3D shape atlas construction, establishing anatomical correspondences across subjects, i.e., surface registration, is probably the most critical but challenging one. Adaptive focus deformable model (AFDM) [1] was proposed to tackle this problem by exploiting cross-scale geometry characteristics of 3D anatomy surfaces. Although the effectiveness of AFDM has been proved in various studies, its performance is highly dependent on the quality of 3D surface meshes, which often degrades along with the iterations of deformable surface registration (the process of correspondence matching). In this paper, we propose a new framework for 3D anatomical shape atlas construction. Our method aims to robustly establish correspondences across different subjects and simultaneously generate high-quality surface meshes without removing shape details. Mathematically, a new energy term is embedded into the original energy function of AFDM to preserve surface mesh qualities during deformable surface matching. More specifically, we employ the Laplacian representation to encode shape details and smoothness constraints. An expectation–maximization style algorithm is designed to optimize multiple energy terms alternatively until convergence. We demonstrate the performance of our method via a set of diverse applications, including a population of sparse cardiac MRI slices with 2D labels, 3D high resolution CT cardiac images and rodent brain MRIs with multiple structures. The constructed shape atlases exhibit good mesh qualities and preserve fine shape details. The constructed shape atlases can further benefit other research topics such as segmentation and statistical analysis.     

Liver vasculature refinement with multiple 3D structuring element shapes

Delineating anatomical structures and other regions of interest is an important component of assisting and automating specific diagnostic, radiological, and surgical tasks. In this paper, a segmentation approach for liver region delineation is proposed, which is based on hysteresis thresholding followed by texture analysis with statistical moments. After that, the region growing method is applied to extract a hepatic vessel tree followed by hepatic vasculature refinement with multiple 3D structuring element shapes. The structure and morphology of the vascular network and its relationship with tumors and liver segments are of major interest to surgeons planning liver surgeries. Knowing the refined major vasculatures is important for surgeons to plan resection into liver segments for tumor treatment, and dissection into right and left lobes to assess accurate liver volume in determining donor suitability for liver transplantation. Therefore, an automated hepatic vessel segmentation scheme followed by vasculature refinement is recommended for planning tumor resections and living donor liver transplants. In addition, these vessel extraction and refinement methods combined with liver region segmentation techniques can also be applicable to extract tree-like organ structures such as carotid artery, renal artery, coronary artery, and airway paths from various medical imaging modalities.     

Urban 3D segmentation and modelling from street view images and LiDAR point clouds

3D urban maps with semantic labels and metric information are not only essential for the next generation robots such autonomous vehicles and city drones, but also help to visualize and augment local environment in mobile user applications. The machine vision challenge is to generate accurate urban maps from existing data with minimal manual annotation. In this work, we propose a novel methodology that takes GPS registered LiDAR (Light Detection And Ranging) point clouds and street view images as inputs and creates semantic labels for the 3D points clouds using a hybrid of rule-based parsing and learning-based labelling that combine point cloud and photometric features. The rule-based parsing boosts segmentation of simple and large structures such as street surfaces and building facades that span almost 75% of the point cloud data. For more complex structures, such as cars, trees and pedestrians, we adopt boosted decision trees that exploit both structure (LiDAR) and photometric (street view) features. We provide qualitative examples of our methodology in 3D visualization where we construct parametric graphical models from labelled data and in 2D image segmentation where 3D labels are back projected to the street view images. In quantitative evaluation we report classification accuracy and computing times and compare results to competing methods with three popular databases: NAVTEQ True, Paris-Rue-Madame and TLS (terrestrial laser scanned) Velodyne.     

等强度婴儿脑MR图像分割的深度学习方法综述

磁共振（magnetic resonance,MR）成像作为一种安全非侵入式的成像技术,可以提供高分辨率且具有不同对比度的大脑图像,被越来越多地应用于婴儿大脑研究中。将婴儿脑MR图像准确地分割为灰质、白质和脑脊液,是研究早期大脑发育模式不可或缺的基础处理环节。由于在等强度阶段（6~9月龄）婴儿脑MR图像中,灰质和白质信号强度基本一致,组织对比度极低,导致此阶段的脑组织分割非常具有挑战性。基于深度学习的等强度婴儿脑MR图像分割方法,由于其卓越的性能受到研究人员的广泛关注,但目前尚未有文献对该领域的方法进行系统总结和分析。因此本文对目前基于深度学习的等强度婴儿脑MR图像分割方法进行了系统总结,从基本思想、网络架构、性能及优缺点4个方面进行了介绍。并针对其中的典型算法在iSeg-2017数据集上的分割结果进行了对比分析,最后对等强度婴儿脑MR图像分割中存在的问题及未来研究方向进行展望。本文通过对目前基于深度学习的等强度婴儿脑MR图像分割方法进行总结,可以看出深度学习方法已经在等强度期婴儿脑分割中展现出巨大优势,相比传统方法在分割精度和效率上均有较大提升,将进一步促进人类人脑早期发育研究。     

三维人体运动特征可视化与交互式运动分割

为了从运动序列中提取不同类型的运动片段进行人体动画创作,提出一种三维人体运动数据可视化与交互式分割技术.首先,采用人体各主要骨骼夹角作为对原始运动数据的几何特征表示,并提出一种启发式方法自动检测其潜在分割点,最后将提取的运动特征可视化并使用可交互用户界面对其进行精确分割.实验结果表明。该特征表示及交互式分割方法能够更加方便、准确、高效地对包含多种类型的长序列三维人体运动数据进行分割.     

基于Virtools虚拟现实技术的三维解剖图谱开发

解剖图谱在医学教育中一直占据非常重要的地位,然而传统图谱空间信息不足,现有电子图谱图像失真且交互性差。运用虚拟现实技术开发交互性强的三维人体解剖学图谱,首先利用CT设备扫描的真实人体断层数据进行三维重建,获得结构完整的人体局部解剖三维模型;然后基于解剖学知识区分不同结构并利用3DSMAX建立骨骼、血管等独立模型并设置贴图,再导入Virtools中,配以文字、图片等多媒体材料,设置丰富的人机交互手段;最后打包生成可在Windows下直接使用的三维图谱软件。将支持游戏开发的三维引擎用于人体解剖图谱的制作中,强调其交互的丰富性,同时具有真实感强、开发周期短等优点,图谱软件有利于课堂施教和学生自学。     

High-quality multimodal volume rendering for preoperative planning of neurosurgical interventions

Surgical approaches tailored to an individual patient's anatomy and pathology have become standard in neurosurgery. Precise preoperative planning of these procedures, however, is necessary to achieve an optimal therapeutic effect. Therefore, multiple radiological imaging modalities are used prior to surgery to delineate the patient's anatomy, neurological function, and metabolic processes. Developing a three-dimensional perception of the surgical approach, however, is traditionally still done by mentally fusing multiple modalities. Concurrent 3D visualization of these datasets can, therefore, improve the planning process significantly. In this paper we introduce an application for planning of individual neurosurgical approaches with high-quality interactive multimodal volume rendering. The application consists of three main modules which allow to (1) plan the optimal skin incision and opening of the skull tailored to the underlying pathology; (2) visualize superficial brain anatomy, function and metabolism; and (3) plan the patient-specific approach for surgery of deep-seated lesions. The visualization is based on direct multi-volume raycasting on graphics hardware, where multiple volumes from different modalities can be displayed concurrently at interactive frame rates. Graphics memory limitations are avoided by performing raycasting on bricked volumes. For preprocessing tasks such as registration or segmentation, the visualization modules are integrated into a larger framework, thus supporting the entire workflow of preoperative planning.     

Distance visualization for interactive 3D implant planning

An instant and quantitative assessment of spatial distances between two objects plays an important role in interactive applications such as virtual model assembly, medical operation planning, or computational steering. While some research has been done on the development of distance-based measures between two objects, only very few attempts have been reported to visualize such measures in interactive scenarios. In this paper we present two different approaches for this purpose, and we investigate the effectiveness of these approaches for intuitive 3D implant positioning in a medical operation planning system. The first approach uses cylindrical glyphs to depict distances, which smoothly adapt their shape and color to changing distances when the objects are moved. This approach computes distances directly on the polygonal object representations by means of ray/triangle mesh intersection. The second approach introduces a set of slices as additional geometric structures, and uses color coding on surfaces to indicate distances. This approach obtains distances from a precomputed distance field of each object. The major findings of the performed user study indicate that a visualization that can facilitate an instant and quantitative analysis of distances between two objects in interactive 3D scenarios is demanding, yet can be achieved by including additional monocular cues into the visualization.     

An interactive computer‐based simulation system for endovascular aneurysm repair surgeries

This paper presents an interactive simulation system for surgical procedures of endovascular aneurysm repair. It extracts anatomical structure of clinic interest from patient‐specific X‐ray computed tomography or magnetic resonance imaging data by image segmentation techniques, and then reconstructs surface triangular meshes of these anatomical structures from the volumetric data. The core of the system is an interactive computer‐based simulation module. It consists of a physical modeling unit, a collision detection unit, a visualization unit, and a control unit. The integration of these units together makes it possible for users to interact with the system in real time, performing virtual catheterization, angiography, and stent graft deployment under a user‐specified rendering mode. The prototype system can be used as a cost‐efficient tool for surgical planning with patient‐specific anatomical geometry and for practice of surgical procedures before actual operation. Copyright © 2016 John Wiley & Sons, Ltd.     

Deep learning-based smart task assistance in wearable augmented reality

Wearable augmented reality (AR) smart glasses have been utilized in various applications such as training, maintenance, and collaboration. However, most previous research on wearable AR technology did not effectively supported situation-aware task assistance because of AR marker-based static visualization and registration. In this study, a smart and user-centric task assistance method is proposed, which combines deep learning-based object detection and instance segmentation with wearable AR technology to provide more effective visual guidance with less cognitive load. In particular, instance segmentation using the Mask R-CNN and markerless AR are combined to overlay the 3D spatial mapping of an actual object onto its surrounding real environment. In addition, 3D spatial information with instance segmentation is used to provide 3D task guidance and navigation, which helps the user to more easily identify and understand physical objects while moving around in the physical environment. Furthermore, 2.5D or 3D replicas support the 3D annotation and collaboration between different workers without predefined 3D models. Therefore, the user can perform more realistic manufacturing tasks in dynamic environments. To verify the usability and usefulness of the proposed method, we performed quantitative and qualitative analyses by conducting two user studies: 1) matching a virtual object to a real object in a real environment, and 2) performing a realistic task, that is, the maintenance and inspection of a 3D printer. We also implemented several viable applications supporting task assistance using the proposed deep learning-based task assistance in wearable AR.     

多图谱与联合标签融合策略相结合的主动脉CT图像分割

主动脉图像自动分割技术在主动脉疾病的早期诊断、风险评估及手术治疗中发挥重要作用。本文采用了基于多图谱的医学图像分割技术,并将之与联合标签融合（Joint label fusion,JLF）策略相结合应用于3D主动脉CT图像的自动分割问题中。联合标签融合策略考虑了各个图谱之间的相互关系,能够有效抑制图谱间冗余信息的干扰,进而提高标签融合精度。本文提出了一种图谱更新算法以应对图谱数量不足的问题,在提高分割精度的同时,保持了较低的计算复杂度。在15例主动脉CT图像数据上的分割结果表明,本文方法能有效地对3D主动脉图像进行分割,与3种基于传统融合方式的图谱分割法相比,本文方法具有更高的分割精度。     

Investigating three-dimensional sketching for early conceptual design—Results from expert discussions and user studies

As immersive 3D user interfaces reach broader acceptance, their use as sketching media is attracting both commercial and academic interests. So far, little is known about user requirements and cognitive aspects of immersive 3D sketching. Also the latter's integration into the workflow of virtual product development is far from being solved.We present results from two focus group expert discussions, a comparative user study on immersive 3D sketching conducted among professional furniture designers and a qualitative content analysis of user statements. The results of the focus group discussions show a strong interest in using the three-dimensional (3D) space as a medium for conceptual design. Users expect it to provide new means for the sketching process, namely spatiality, one-to-one proportions, associations, and formability. Eight groups of functions required for 3D sketching were outlined during the discussions.The comparative study was intended to find and investigate advantages of immersive three-dimensional space and its additional degrees-of-freedom for creative/reflective externalization processes. We compared a 3D and a 2D baseline condition in the same technical environment, a VR-Cave system. In both conditions, no haptic feedback was provided and the 2D condition was not intended to simulate traditional 2D sketching (on paper). The results from our user study show that both the sketching process and the resulting sketches differ in the 2D and 3D condition, namely in terms of the perceived fluency of sketch creation, in terms of the perceived appropriateness for the task, and in terms of the perceived stimulation by the medium, the movement speed, the sketch sizes, the degree of detail, the functional aspects, and the usage time. In order to validate the results of the focus group discussions, we produced a questionnaire to check for the subjectively perceived advantages and disadvantages in both the 2D and 3D conditions. A qualitative content analysis of the user statements revealed that the biggest advantage of 3D sketching lies in the sketching process itself. In particular, the participants emphasized the system's ability to foster inspiration and to improve the recognition of spatiality and spatial thinking.We argue that both 2D and 3D sketching are relevant for early conceptual design. As we progress towards 3D sketching, new tangible interactive tools are needed, which account for the user's perceptual and cognitive abilities.     

An Evaluation of Visualization Techniques to Illustrate Statistical Deformation Models

As collections of 2D/3D images continue to grow, interest in effective ways to visualize and explore the statistical morphological properties of a group of images has surged. Recently, deformation models have emerged as simple methods to capture the variability and statistical properties of a collection of images. Such models have proven to be effective in tasks such as image classification, generation, registration, segmentation, and analysis of modes of variation. A crucial element missing from most statistical models has been an effective way to summarize and visualize the statistical morphological properties of a group of images. This paper evaluates different visualization techniques that can be extended and used to illustrate the information captured by such statistical models. First, four illustration techniques are described as methods to summarize the statistical morphological properties as captured by deformation models. Second, results of a user study conducted to compare the effectiveness of each visualization technique are presented. After comparing the performance of 40 subjects, we found that statistical annotation techniques present significant benefits when analyzing the structural properties of a group of images.