基于机器学习的刀具表面缺陷检测及分类方法

刀具在生产的过程中,由于人员、机器、环境等多方面原因,刀具的表面会出现各种缺陷,如划痕、碰撞凹坑、涂层剥落和边缘豁口;这些缺陷会严重影响刀具的质量和外观,对于刀具的缺陷检测,目前主要采用人工目检的方式,人工检测方法效率和准确率都比较低;为解决上述问题,提出一种刀具缺陷的自动化检测及分类算法;针对刀具图像的预处理,提出了一种基于双边滤波的降噪方法和基于差分的对比度增强算法;对于刀具的缺陷检测任务,提出了基于图像差分的缺陷检测算法;对于缺陷的分类任务,提出了一种基于SVM的分类算法,即通过提取缺陷区域的形状、纹理等特征来训练SVM分类器;最后对提出的缺陷检测及分类算法进行实验,结果表明算法的缺陷检出率达97.2%,分类准确率可达94.3%;算法能够很好地满足工业需求,可以替代人工实现刀具缺陷的自动化和高效率检测。     

Automatic defect detection of metro tunnel surfaces using a vision-based inspection system

Due to the impact of the surrounding environment changes, train-induced vibration, and human interference, damage to metro tunnel surfaces frequently occurs. Therefore, accidents caused by the tunnel surface damage may happen at any time, since the lack of adequate and efficient maintenance. To our knowledge, effective maintenance heavily depends on the all-round and accurate defect inspection, which is a challenging task, due to the harsh environment (e.g., insufficient illumination, the limited time window for inspection, etc.). To address these problems, we design an automatic Metro Tunnel Surface Inspection System (MTSIS) for the efficient and accurate defect detection, which covers the design of hardware and software parts. For the hardware component, we devise a data collection system to capture tunnel surface images with high resolution at high speed. For the software part, we present a tunnel surface image pre-processing approach and a defect detection method to recognize defects with high accuracy. The image pre-processing approach includes image contrast enhancement and image stitching in a coarse-to-fine manner, which are employed to improve the quality of raw images and to avoid repeating detection for overlapped regions of the captured tunnel images respectively. To achieve automatic tunnel surface defect detection with high precision, we propose a multi-layer feature fusion network, based on the Faster Region-based Convolutional Neural Network (Faster RCNN). Our image pre-processing and the defect detection methods also promising performance in terms of recall and precision, which is demonstrated through a series of practical experimental results. Moreover, our MTSIS has been successfully applied on several metro lines.     

Autoencoder-based anomaly detection for surface defect inspection

In this paper, the unsupervised autoencoder learning for automated defect detection in manufacturing is evaluated, where only the defect-free samples are required for the model training. The loss function of a Convolutional Autoencoder (CAE) model only aims at minimizing the reconstruction errors, and makes the representative features widely spread. The proposed CAE in this study incorporates a regularization that improves the feature distribution of defect-free samples within a tight range. It makes the representative feature vectors of all training samples as close as possible to the mean feature vector so that a defect sample in the evaluation stage can generate a distinct distance from the trained center of defect-free samples. The proposed CAE model with regularizations has been tested on a variety of material surfaces, including textural and patterned surfaces in images. The experimental results reveal that the proposed CAE with regularizations significantly outperforms the conventional CAE for defect detection applications in the industry.     

Continuous frame motion sensitive self-supervised collaborative network for video representation learning

Motion, as a feature of video that changes in temporal sequences, is crucial to visual understanding. The powerful video representation and extraction models are typically able to focus attention on motion features in challenging dynamic environments to complete more complex video understanding tasks. However, previous approaches discriminate mainly based on similar features in the spatial or temporal domain, ignoring the interdependence of consecutive video frames. In this paper, we propose the motion sensitive self-supervised collaborative network, a video representation learning framework that exploits a pretext task to assist feature comparison and strengthen the spatiotemporal discrimination power of the model. Specifically, we first propose the motion-aware module, which extracts consecutive motion features from the spatial regions by frame difference. The global–local contrastive module is then introduced, with context and enhanced video snippets being defined as appropriate positive samples for a broader feature similarity comparison. Finally, we introduce the snippet operation prediction module, which further assists contrastive learning to obtain more reliable global semantics by sensing changes in continuous frame features. Experimental results demonstrate that our work can effectively extract robust motion features and achieve competitive performance compared with other state-of-the-art self-supervised methods on downstream action recognition and video retrieval tasks.     

Selecting informative data for defect segmentation from imbalanced datasets via active learning

The storage and labeling of industrial data incur significant costs during the development of defect detection algorithms. Active learning solves these problems by selecting the most informative data among the given unlabeled data. The existing active learning methods for image segmentation focus on studying natural images and medical images, with less attention given to industrial images, and little research has been performed on imbalanced data. To solve these problems, we propose an active learning framework to selecting informative data for defect segmentation under imbalanced data. In the initialization stage, the framework uses self-supervised learning to initialize the data so that the initialization data contain more defect data, thereby solving the cold-start problem. During the iterative stage, we design the main body of the active learning framework, which is composed of a segmentation learner and a reconstruction learner. These learners use supervised learning to further improve the framework’s ability to select informative data. The experimental results obtained on public and self-owned datasets show that the framework can save 70% of the required storage space and greatly reduce the cost of labeling. The intersection over union value proves that the designed framework can achieve the equivalent effect of labeling the whole dataset by labeling partial data.     

Defect-aware transformer network for intelligent visual surface defect detection

Surface defect detection plays an increasing role in intelligent manufacturing and product life-cycle management, such as quality inspection, process monitoring, and preventive maintenance. The existing intelligent methods almost adopt convolution architecture, and the limited receptive field hinders performance improvement of defect detection. In general, a larger receptive field can bring richer contextual information, resulting in better performance. Although operations such as dilated convolution can expand the receptive field, this improvement is still limited. Recently, benefitting from the ability to model long-range dependencies, Transformer-based models achieve great success in computer vision and image processing. However, applying Transformer-based models without modification is not desirable because there is no awareness and pertinence to defects. In this paper, an intelligent method is proposed by using defect-aware Transformer network (DAT-Net). In DAT-Net, Transformer replaces convolution in encoder to overcome the difficulty of modeling long-range dependencies. Defect-aware module assembled by basic weight matrixes is incorporated into Transformer to perceive and capture geometry and characteristic of defect. Graph position encoding by constructing a dynamic graph on tokens is designed to provide auxiliary positional information, which brings desired improved performance and fine adaptability. Specially, we carry out field experiments and painstakingly construct blade defect and tool wear datasets to compare DAT-Net with other methods. The comprehensive experiments demonstrate that DAT-Net has superior performance with 90.19 mIoU on blade defect dataset and 87.24 mIoU on tool wear dataset.     

Clustered defect detection of high quality chips using self-supervised multilayer perceptron

During electrical testing, each die on a wafer must be tested to determine whether it functions as originally designed. When defects, including scratches, stains or localized failed patterns, are clustered on the wafer, the tester may not detect all of the defective dies in the flawed area. A testing factory must assign a few workers to check the wafers and hand-mark the defective dies in the flawed region or close to the flawed region, to ensure that no defective die is present in the final assembly. This work presents an automatic wafer-scale defect cluster identifier that uses a multilayer perceptron to detect the defect cluster and mark all of the defective dies. The proposed identifier is compared with an existing tool used in industry. The experimental results confirm that the proposed algorithm is more effective at identifying defects and outperforms the present approach.     

Deep learning based online metallic surface defect detection method for wire and arc additive manufacturing

Wire and arc additive manufacturing (WAAM) is an emerging manufacturing technology that is widely used in different manufacturing industries. To achieve fully automated production, WAAM requires a dependable, efficient, and automatic defect detection system. Although machine learning is dominant in the object detection domain, classic algorithms have defect detection difficulty in WAAM due to complex defect types and noisy detection environments. This paper presents a deep learning-based novel automatic defect detection solution, you only look once (YOLO)-attention, based on YOLOv4, which achieves both fast and accurate defect detection for WAAM. YOLO-attention makes improvements on three existing object detection models: the channel-wise attention mechanism, multiple spatial pyramid pooling, and exponential moving average. The evaluation on the WAAM defect dataset shows that our model obtains a 94.5 mean average precision (mAP) with at least 42 frames per second. This method has been applied to additive manufacturing of single-pass, multi-pass deposition and parts. It demonstrates its feasibility in practical industrial applications and has potential as a vision-based methodology that can be implemented in real-time defect detection systems.     

基于异常检测的产品表面缺陷检测与分割

工业制造中缺陷样本难以获得且缺陷表现形式多样,只用训练正样本的异常检测技术越来越多地被应用于产品表面缺陷检测。异常检测一般通过评估产品图像的异常分数对产品进行有无缺陷的判断,缺乏对缺陷位置的描述,最新提出的异常分割方法对此进行了改进,但对缺陷区域的分割不够精确。基于异常检测方法,使用标准化流来判断产品表面是否有缺陷,采用多尺度特征融合并对齐来初步定位缺陷位置,结合梯度和最大信息熵,使用分水岭算法对初定位结果进行优化得到缺陷分割掩码。在丽盛制板,KolektorSDD和AITEX3个表面缺陷数据集的检测与分割结果均优于其他同类方法。此外,在小样本数据集上也能达到良好的检测与分割精度。     

基于自监督学习的病理图像层次分割

在苏木精-伊红(HE)染色病理图像中，细胞染色分布的不均匀和各类组织形态的多样性给病理图像的自动分割带来极大挑战。为解决该问题，提出了一种基于自监督学习的病理图像三步层次分割方法，对病理图像中各类组织进行由粗略到精细的全自动逐层分割。首先，根据互信息的计算结果在RGB色彩空间中进行特征选择；其次，采用K-means聚类将图像初步分割为各类组织结构的色彩稳定区域与模糊区域；然后，以色彩稳定区域为训练集采用朴素贝叶斯分类对模糊区域进行进一步分割，得到完整的细胞核、细胞质和胞外间隙这三类组织结构；最后，对细胞核部分进行结合形状和色彩强度的混合分水岭分割得到细胞核间的精确边界，进而量化计算细胞核个数、核占比、核质比等指标。对脑膜瘤HE染色病理图像的分割实验结果表明，所提方法对于染色和细胞形态差异保持较高的鲁棒性，各类组织区域分割误差在5%以内，在细胞核分割精度的对比实验中平均正确率在96%以上，满足临床自动图像分析的要求，其量化结果可以为定量病理分析提供依据。     

Automated defect inspection system for metal surfaces based on deep learning and data augmentation

Recent efforts to create a smart factory have inspired research that analyzes process data collected from Internet of Things (IOT) sensors, to predict product quality in real time. This requires an automatic defect inspection system that quantifies product quality data by detecting and classifying defects in real time. In this study, we propose a vision-based defect inspection system to inspect metal surface defects. In recent years, deep convolutional neural networks (DCNNs) have been used in many manufacturing industries and have demonstrated the excellent performance as a defect classification method. A sufficient amount of training data must be acquired, to ensure high performance using a DCNN. However, owing to the nature of the metal manufacturing industry, it is difficult to obtain enough data because some defects occur rarely. Owing to this imbalanced data problem, the generalization performance of the DCNN-based classification algorithm is lowered. In this study, we propose a new convolutional variational autoencoder (CVAE) and deep CNN-based defect classification algorithm to solve this problem. The CVAE-based data generation technology generates sufficient defect data to train the classification model. A conditional CVAE (CCVAE) is proposed to generate images for each defect type in a single CVAE model. We also propose a classifier based on a DCNN with high generalization performance using data generated from the CCVAE. In order to verify the performance of the proposed method, we performed experiments using defect images obtained from an actual metal production line. The results showed that the proposed method exhibited an excellent performance.     

Automatic image segmentation using constraint learning and propagation

In this paper, we propose automatic image segmentation using constraint learning and propagation. Recently, kernel learning is receiving much attention because a learned kernel can fit the given data better than a predefined kernel. To effectively learn the constraints generated by initial seeds for image segmentation, we employ kernel propagation (KP) based on kernel learning. The key idea of KP is first to learn a small-sized seed-kernel matrix and then propagate it into a large-sized full-kernel matrix. By applying KP to automatic image segmentation, we design a novel segmentation method to achieve high performance. First, we generate pairwise constraints, i.e., must-link and cannot-link, from initially selected seeds to make the seed-kernel matrix. To select the optimal initial seeds, we utilize global k-means clustering (GKM) and self-tuning spectral clustering (SSC). Next, we propagate the seed-kernel matrix into the full-kernel matrix of the entire image, and thus image segmentation results are obtained. We test our method on the Berkeley segmentation database, and the experimental results demonstrate that the proposed method is very effective in automatic image segmentation.     

基于自我监督学习策略的层智能图卷积网络

为了解决当前图卷积网络需要依赖大型数据集,从而导致时间和空间复杂度上升问题,提出了基于自我监督学习策略的层智能图卷积网络(RRLFS-L-GCN)。首先,通过在层智能图卷积网络(layer-wise graph convolutional network, L-GCN)中添加多任务机制以提高算法的泛化能力;然后,设计一种随机删除固定步长边(aandomly remove links with a fixed step, RRLFS)的自我监督学习策略,从而提出基于自我监督学习策略的层智能图卷积网络算法;最后,通过边预测验证RRLFS-L-GCN的性能。实验结果表明,该算法的识别率最高可达97.13%。对于Cora测试集,该算法所得识别准确率比未改进的层智能图卷积网络算法提高了6.73%。对于PubMed测试集,该算法所得识别准确率比未改进的层智能图卷积网络算法提高了8.13%。与图卷积网络相比,在Citeseer数据集上,识别准确率提高了18.43%。     

全卷积语义分割与物体检测网络

目的 目前主流物体检测算法需要预先划定默认框,通过对默认框的筛选剔除得到物体框。为了保证足够的召回率,就必须要预设足够密集和多尺度的默认框,这就导致了图像中各个区域被重复检测,造成了极大的计算浪费。提出一种不需要划定默认框,实现完全端到端深度学习语义分割及物体检测的多任务深度学习模型（FCDN）,使得检测模型能够在保证精度的同时提高检测速度。方法 首先分析了被检测物体数量不可预知是目前主流物体检测算法需要预先划定默认框的原因,由于目前深度学习物体检测算法都是由图像分类模型拓展而来,被检测数量的无法预知导致无法设置检测模型的输出,为了保证召回率,必须要对足够密集和多尺度的默认框进行分类识别;物体检测任务需要物体的类别信息以实现对不同类物体的识别,也需要物体的边界信息以实现对各个物体的区分、定位;语义分割提取了丰富的物体类别信息,可以根据语义分割图识别物体的种类,同时采用语义分割的思想,设计模块提取图像中物体的边界关键点,结合语义分割图和边界关键点分布图,从而完成物体的识别和定位。结果 为了验证基于语义分割思想的物体检测方法的可行性,训练模型并在VOC（visual object classes）2007 test数据集上进行测试,与目前主流物体检测算法进行性能对比,结果表明,利用新模型可以同时实现语义分割和物体检测任务,在训练样本相同的条件下训练后,其物体检测精度优于经典的物体检测模型;在算法的运行速度上,相比于FCN,减少了8 ms,比较接近于YOLO（you only look once）等快速检测算法。结论 本文提出了一种新的物体检测思路,不再以图像分类为检测基础,不需要对预设的密集且多尺度的默认框进行分类识别;实验结果表明充分利用语义分割提取的丰富信息,根据语义分割图和边界关键点完成物体检测的方法是可行的,该方法避免了对图像的重复检测和计算浪费;同时通过减少语义分割预测的像素点数量来提高检测效率,并通过实验验证简化后的语义分割结果仍足够进行物体检测任务。     

Neural network based detection of local textile defects

A new approach for the segmentation of local textile defects using feed-forward neural network is presented. Every fabric defect alters the gray-level arrangement of neighboring pixels, and this change is used to segment the defects. The feature vector for every pixel is extracted from the gray-level arrangement of its neighboring pixels. Principal component analysis using singular value decomposition is used to reduce the dimension of feature vectors. Experimental results using this approach illustrate a high degree of robustness for the detection of a variety of fabric defects. The acceptance of a visual inspection system depends on economical aspects as well. Therefore, a new low-cost solution for the fast web inspection using linear neural network is also presented. The experimental results obtained from the real fabric defects, for the two approaches proposed in this paper, have confirmed their usefulness.     

苹果采摘机器人监测系统和表面缺陷检测方法研究

针对目前苹果采摘作业多以人工采摘为主,且机械化采摘质量和效率不高等问题,提出一种苹果采摘机器人监测系统。监测系统通过种植园环境、机器人运行状态、机器人作业质量以及后台监控系统四个模块,完成从采收前环境监测到采收后质量监测的全程监控。同时,为监测采摘后苹果的质量问题,提出了一种基于Mo-M2Det的苹果表面缺陷检测方法。以减少参数量和计算量为目的,改进M2Det目标检测网络,实现轻量化和高精度检测。实验结果表明,改进的M2Det目标检测网络检测准确率达到了98.45%,且模型参数量减少了56.3%。在实际应用中,改进的轻量化网络模型部署在检测平台上具有较好的检测效果。     

深度学习多部位病灶检测与分割

目的 多部位病灶具有大小各异和类型多样的特点,对其准确检测和分割具有一定的难度。为此,本文设计了一种2.5D深度卷积神经网络模型,实现对多种病灶类型的计算机断层扫描（computed tomography,CT）图像的病灶检测与分割。方法 利用密集卷积网络和双向特征金字塔网络组成的骨干网络提取图像中的多尺度和多维度信息,输入为带有标注的中央切片和提供空间信息的相邻切片共同组合而成的CT切片组。将融合空间信息的特征图送入区域建议网络并生成候选区域样本,再由多阈值级联网络组成的Cascade R-CNN（region convolutional neural networks）筛选高质量样本送入检测与分割分支进行训练。结果 本文模型在DeepLesion数据集上进行验证。结果表明,在测试集上的平均检测精度为83.15%,分割预测结果与真实标签的端点平均距离误差为1.27 mm,直径平均误差为1.69 mm,分割性能优于MULAN（multitask universal lesion analysis network for joint lesion detection,tagging and segmentation）和Auto RECIST（response evaluation criteria in solid tumors）,且推断每幅图像平均时间花费仅91.7 ms。结论 对于多种部位的CT图像,本文模型取得良好的检测与分割性能,并且预测时间花费较少,适用病变类别与DeepLesion数据集类似的CT图像实现病灶检测与分割。本文模型在一定程度上能满足医疗人员利用计算机分析多部位CT图像的需求。     

Interactive image segmentation using probabilistic hypergraphs

This paper introduces a novel interactive framework for segmenting images using probabilistic hypergraphs which model the spatial and appearance relations among image pixels. The probabilistic hypergraph provides us a means to pose image segmentation as a machine learning problem. In particular, we assume that a small set of pixels, which are referred to as seed pixels, are labeled as the object and background. The seed pixels are used to estimate the labels of the unlabeled pixels by learning on a hypergraph via minimizing a quadratic smoothness term formed by a hypergraph Laplacian matrix subject to the known label constraints. We derive a natural probabilistic interpretation of this smoothness term, and provide a detailed discussion on the relation of our method to other hypergraph and graph based learning methods. We also present a front-to-end image segmentation system based on the proposed method, which is shown to achieve promising quantitative and qualitative results on the commonly used GrabCut dataset.     

Rumor detection with self-supervised learning on texts and social graph

Rumor detection has become an emerging and active research field in recent years. At the core is to model the rumor characteristics inherent in rich information, such as propagation patterns in social network and semantic patterns in post content, and differentiate them from the truth. However, existing works on rumor detection fall short in modeling heterogeneous information, either using one single information source only (e.g., social network, or post content) or ignoring the relations among multiple sources (e.g., fusing social and content features via simple concatenation).Therefore, they possibly have drawbacks in comprehensively understanding the rumors, and detecting them accurately. In this work, we explore contrastive self-supervised learning on heterogeneous information sources, so as to reveal their relations and characterize rumors better. Technically, we supplement the main supervised task of detection with an auxiliary self-supervised task, which enriches post representations via post self-discrimination.Specifically, given two heterogeneous views of a post (i.e., representations encoding social patterns and semantic patterns), the discrimination is done by maximizing the mutual information between different views of the same post compared to that of other posts. We devise cluster-wise and instance-wise approaches to generate the views and conduct the discrimination, considering different relations of information sources. We term this framework as self-supervised rumor detection (SRD). Extensive experiments on three real-world datasets validate the effectiveness of SRD for automatic rumor detection on social media.     

Automating image segmentation verification and validation by learning test oracles

An image segmentation algorithm delineates (an) object(s) of interest in an image. Its output is referred to as a segmentation. Developing these algorithms is a manual, iterative process involving repetitive verification and validation tasks. This process is time-consuming and depends on the availability of experts, who may be a scarce resource (e.g., medical experts). We propose a framework referred to as Image Segmentation Automated Oracle (ISAO) that uses machine learning to construct an oracle, which can then be used to automatically verify the correctness of image segmentations, thus saving substantial resources and making the image segmentation verification and validation task significantly more efficient. The framework also gives informative feedback to the developer as the segmentation algorithm evolves and provides a systematic means of testing different parametric configurations of the algorithm. During the initial learning phase, segmentations from the first few (optimally two) versions of the segmentation algorithm are manually verified by experts. The similarity of successive segmentations of the same images is also measured in various ways. This information is then fed to a machine learning algorithm to construct a classifier that distinguishes between consistent and inconsistent segmentation pairs (as determined by an expert) based on the values of the similarity measures associated with each segmentation pair. Once the accuracy of the classifier is deemed satisfactory to support a consistency determination, the classifier is then used to determine whether the segmentations that are produced by subsequent versions of the algorithm under test, are (in)consistent with already verified segmentations from previous versions. This information is then used to automatically draw conclusions about the correctness of the segmentations. We have successfully applied this approach to 3D segmentations of the cardiac left ventricle obtained from CT scans and have obtained promising results (accuracies of 95%). Even though more experiments are needed to quantify the effectiveness of the approach in real-world applications, ISAO shows promise in increasing the quality and testing efficiency of image segmentation algorithms.