Multimodal Pre-training Method for Vision-language Understanding and Generation

Most existing vision-language pre-training methods focus on understanding tasks and use BERT-like loss functions (masked language modeling and image-text matching) during pre-training. Despite their good performance in the understanding of downstream tasks, such as visual question answering, image-text retrieval, and visual entailment, these methods cannot generate information. To tackle this problem, this study proposes Unified multimodal pre-training for Vision-Language understanding and generation (UniVL). The proposed UniVL is capable of handling both understanding tasks and generation tasks. It expands existing pre-training paradigms and uses random masks and causal masks simultaneously, where causal masks are triangular masks that mask future tokens, and such pre-trained models can have autoregressive generation abilities. Moreover, several vision-language understanding tasks are turned into text generation tasks according to specifications, and the prompt-based method is employed for fine-tuning of different downstream tasks. The experiments show that there is a trade-off between understanding tasks and generation tasks when the same model is used, and a feasible way to improve both tasks is to use more data. The proposed UniVL framework attains comparable performance to recent vision-language pre-training methods in both understanding tasks and generation tasks. Moreover, the prompt-based generation method is more effective and even outperforms discriminative methods in few-shot scenarios.     

基于多尺度特征融合的皮肤烧伤创面分级算法

为实现遭受重大火灾等灾害后,对伤员皮肤烧伤自动化分级,加快诊断效率,提出提出一种用于皮肤烧伤分类的轻量化模型BI-YOLOv5算法。替换Swish激活函数,提高模型收敛能力及检测效率;使用K-means++算法对anchors聚类分析,增强对不同尺度目标的适应能力;修改特征提取网络,提取多个尺度的特征信息,建立多尺度特征融合网络,提高模型对深层特征信息的利用率,提高小面积烧伤的识别精度。实验结果表明,BI-YOLOv5算法在检测并区分不同烧伤类别及环境干扰下烧伤检测有较高的精度和效率,mAP达到97.6,对比YOLOv5提升8.4个百分点。     

Visual learning and classification of human epithelial type 2 cell images through spontaneous activity patterns

Identifying the presence of anti-nuclear antibody (ANA) in human epithelial type 2 (HEp-2) cells via the indirect immunofluorescence (IIF) protocol is commonly used to diagnose various connective tissue diseases in clinical pathology tests. As it is a labour and time intensive diagnostic process, several computer aided diagnostic (CAD) systems have been proposed. However, the existing CAD systems suffer from numerous shortcomings due to the selection of features, which is commonly based on expert experience. Such a choice of features may not work well when the CAD systems are retasked to another dataset. To address this, in our previous work, we proposed a novel approach that learns a set of filters from HEp-2 cell images. It is inspired by the receptive fields in the mammalian's vision system, since the receptive fields can be thought as a set of filters for similar shapes. We obtain robust filters for HEp-2 cell classification by employing the independent component analysis (ICA) framework. Although, this approach may be held back due to one particular problem; ICA learning requires a sufficiently large volume of training data which is not always available. In this paper, we demonstrate a biologically inspired solution to address this issue via the use of spontaneous activity patterns (SAP). The spontaneous activity patterns, which are related to the spontaneous neural activities initialised by the chemical release in the brain, are found as the typical stimuli for the visual cell development of newborn animals. In the classification system for HEp-2 cells, we propose to model SAP as a set of small image patches containing randomly positioned Gaussian spots. The SAP image patches are generated and mixed with the training images in order to learn filters via the ICA framework. The obtained filters are adopted to extract the set of responses from a HEp-2 cell image. We then employ regions from this set of responses and stack them into “cubic regions”, and apply a classification based on the correlation information of the features. We show that applying the additional SAP leads to a better classification performance on HEp-2 cell images compared to using only the existing patterns for training ICA filters. The improvement on classification is particularly significant when there are not enough specimen images available in the training set, as SAP adds more variations to the existing data that makes the learned ICA model more robust. We show that the proposed approach consistently outperforms three recently proposed CAD systems on two publicly available datasets: ICPR HEp-2 contest and SNPHEp-2.     

3D soft-tissue tracking using spatial-color joint probability distribution and thin-plate spline model

Visual tracking techniques based on stereo endoscope are developed to measure tissue motion in robot-assisted minimally invasive surgery. However, accurate 3D tracking of tissue surfaces remains challenging due to complicated deformation, poor imaging conditions, specular reflections and other dynamic effects during surgery. This study employs a robust and efficient 3D tracking scheme with two independent recursive processes, namely kernel-based inter-frame motion estimation and model-based intra-frame 3D matching. In the first process, target region is represented in joint spatial-color space for robust estimation. By defining a probabilistic similarity measure, a mean-shift-based iterative algorithm is derived for location of the target region in a new image. In the second process, the thin-plate spline model is used to fit the 3D shape of tissue surfaces around the target region. An iterative algorithm based on an efficient second-order minimization technique is derived to compute optimal model parameters. The two processes can be computed in parallel. Their outputs are combined to recover 3D information about the target region. The performance of the proposed method is validated using phantom heart videos and in vivo videos acquired by the daVinci^®${\mathrm{daVinci}}^{\circledR }$ surgical robotic platform and a synthesized data set with known ground truth.     

Laplacian colormaps: a framework for structure‐preserving color transformations

Mappings between color spaces are ubiquitous in image processing problems such as gamut mapping, decolorization, and image optimization for color‐blind people. Simple color transformations often result in information loss and ambiguities, and one wishes to find an image‐specific transformation that would preserve as much as possible the structure of the original image in the target color space. In this paper, we propose Laplacian colormaps, a generic framework for structure‐preserving color transformations between images. We use the image Laplacian to capture the structural information, and show that if the color transformation between two images preserves the structure, the respective Laplacians have similar eigenvectors, or in other words, are approximately jointly diagonalizable. Employing the relation between joint diagonalizability and commutativity of matrices, we use Laplacians commutativity as a criterion of color mapping quality and minimize it w.r.t. the parameters of a color transformation to achieve optimal structure preservation. We show numerous applications of our approach, including color‐to‐gray conversion, gamut mapping, multispectral image fusion, and image optimization for color deficient viewers.     

Examining the Effects of Conformal Terrain Features in Advanced Head‐Up Displays on Flight Performance and Pilot Situation Awareness

Synthetic vision systems (SVS) render terrain features for pilots through cockpit displays using a GPS database and three‐dimensional graphical models. Enhanced vision systems (EVS) present infrared imagery of terrain using a forward‐looking sensor in the nose of an aircraft. The ultimate goal of SVS and EVS technologies is to support pilots in achieving safety under low‐visibility and night conditions comparable to clear, day conditions. This study assessed pilot performance and situation awareness (SA) effects of SVS and EVS imagery in an advanced head‐up display (HUD) during a simulated landing approach under instrument meteorological conditions. Videos of the landing with various HUD configurations were presented to eight pilots with a superimposed tracking task. The independent variables included four HUD feature configurations (baseline [no terrain imagery], SVS, EVS, and a combination of SVS and EVS), two visibility conditions, and four legs of the flight. Results indicated that SVS increased overall SA but degraded flight path control performance because of visual confusion with other display features. EVS increased flight path control accuracy but decreased system (aircraft) awareness because of visual distractions. The combination of SVS and EVS generated offsetting effects. Display configurations did not affect pilot spatial awareness. Flight performance was not different among phases of the approach, but levels and types of pilot SA did vary from leg to leg. These results are applicable to development of adaptive HUD features to support pilot performance. They support the use of multidimensional measures of SA for insight on pilot information processing with advanced aviation displays. © 2012 Wiley Periodicals, Inc.     

Evaluation of stereo correspondence algorithms and their implementation on FPGA

The accuracy of stereo vision has been considerably improved in the last decade, but real-time stereo matching is still a challenge for embedded systems where the limited resources do not permit fast operation of sophisticated approaches. This work presents an evaluation of area-based algorithms used for calculating distance in stereoscopic vision systems, their hardware architectures for implementation on FPGA and the cost of their accuracies in terms of FPGA hardware resources. The results show the trade-off between the quality of such maps and the hardware resources which each solution demands, so they serve as a guide for implementing stereo correspondence algorithms in real-time processing systems.     

浅谈电子产品对儿童的影响及对策

在现代社会中,平板电脑、智能手机在人们日常生活的方方面面发挥着巨大的作用。通过分析电子产品在儿童和青少年的视力、身体发育、社交能力和学习能力等方面的负面影响,提出了加强亲子陪伴、合理引导控制等应对措施。