Automated visual inspection: a survey

This paper surveys publications, reports, and articles dealing with automated visual inspection for industry. The references are organized according to their contents: overview and discussions, rationales, components and design considerations, commercially available systems, applications. A number of applications and their inspection methodologies are discussed in detail: the inspection of printed circuit boards, photomasks, integrated circuit chips. Other inspection applications are listed as a bibliography. A list of selectively annotated references in commercially available visual inspection tools is also included.     

Overall Wafer Effectiveness (OWE): A novel industry standard for semiconductor ecosystem as a whole

As semiconductor industry reached nanotechnology generation and consumer electronics era, the competition is no longer among individual semiconductor companies. Indeed, the collaborations among horizontally specialized value providers are critical for the success of the companies as well as the whole ecosystem. This paper aims to propose a novel index, i.e., Overall Wafer Effectiveness (OWE), to measure wafer productivity and drive various improvement directions for semiconductor ecosystem as a whole. Furthermore, the proposed OWE can be easily extended to incorporate additional attributes such as mask-field-utilization, throughput, and yield for effective management. We conducted a number of case studies in real settings. The results have shown that OWE can be employed as a semiconductor industry standard to drive collaborative efforts among IC designers, equipment vendors, and manufacturers in the ecosystem to enhance total wafer effectiveness. This paper concludes with discussions on value propositions of proposed OWE indices and future research directions.     

Automated inspection planning of free-form shape parts by laser scanning

The inspection operation accounts for a large portion of manufacturing lead time, and its importance in quality control cannot be overemphasized. In recent years, due to the development of laser technology, the accuracy of laser scanners has been improved significantly so that they can be used in a production environment. They are noncontact-type-measuring devices and usually have the scanning speed that is 50–100 times faster than that of coordinate measuring machines. This laser-scanning technology provides us a platform that enables us to perform a 100% inspection of complicated shape parts. This research proposes algorithms that lead to the automation of laser scanner-based inspection operations. The proposed algorithms consist of three steps: firstly, all possible accessible directions at each sampled point on a part surface are generated considering constraints existing in a laser scanning operation. The constraints include satisfying the view angle, the depth of view, checking interference with a part, and avoiding collision with the probe. Secondly, the number of scans and the most desired direction for each scan are calculated. Finally, the scan path that gives the minimum scan time is generated. The proposed algorithms are applied to sample parts and the results are discussed.     

Automated visual inspection of rolled metal surfaces

A prototype for an automated visual on-line metal strip inspection system is described. The system is capable of both detecting and classifying surface defects in copper alloy strips, and it has been installed for evaluation in a production line in a rolling mill. The image acquisition part of the system is based on a CCD line scan camera and condensing bright field illuminators. The inspection algorithms are based on morphological preprocessing and combined statistical and structural defect recognition. The image processing hardware consists of commercial modules. An analysis of this implementation is presented. A similar inspection principle has also been successfully applied to steel strip inspection.     

Industrial visual inspection of lime granules by neural networks

Lime granule quality inspection is an important task in the pulp and paper industry. In this paper a new method, build-up on a neural network and a path search method, is introduced for lime granule automatic visual inspection. Several correction steps to Landau's method are also introduced.     

Automated visual inspection: A tutorial

This article introduces the topic of automated visual inspection (AVI) as a subset of machine vision. The components of a generic AVI system will be presented and current applications of AVI systems will be discussed. Specifically, the applications of automated dimension verification, bare printed circuit board inspection, stuffed printed circuit board inspection and component identification using color technology will be covered in some detail.     

A hybrid approach based on Hotelling statistics for automated visual inspection of display blemishes in LCD panels

This paper presents an alternative hybrid approach based on Hotelling statistics, combining ant colony method and neural network model to automatically detect the display blemishes in liquid crystal display (LCD) panels. Owing to their space saving, energy efficiency, and low radiation, LCD’s have been widely applied in many high-tech industries. However, the display blemishes such as abnormal spots (white and black spots) and slight color variations (bright and dark regions) often exist in LCD’s. To detect these color unevenness blemish detection, this research proposes a multivariate statistic based hybrid defect detection approach. We first use multivariate Hotelling statistics to integrate different coordinates of color models and construct a Hotelling distance diagram to represent the degree of color variations for selecting suspected blemish regions. Then, an ant colony algorithm that integrates computer vision techniques precisely identifies the abnormal spot defects in the Hotelling distance diagram. And, the back propagation neural network model determines the regions of slight color variation blemishes based on the Hotelling distance values. Experimental results demonstrate the validness and efficiency of the proposed approach.     

A new strategy for defect inspection by the virtual inspection in semiconductor wafer fabrication

The defect of process equipments is a major factor that impairs the yields in the mass production of semiconductor wafer fabrication and it is a main supervision means to use high-resolution defect inspection tools to detect and monitor the defect damage. Due to the high investment costs of these inspection tools and the resulting decrease in the throughput, how to improve the sampling rate is an important issue for the associated inspection strategy. This paper proposes a new concept and implementation of virtual inspection (VI) to enhance the detection and monitoring of defect in semiconductor production process. The underlying theory of the VI concept is that the state variables identifications (SVIDs) of process equipments can reflect the process quality effectively and loyally. The approach of VI is to combine the application of the fault detection and classification (FDC), and the defect library and the re-engineering of inspection procedure to reach the full-scope of strategic objective. VI enables the defect monitoring to enter a new era by promoting the monitoring level of defect inspection from the previous lot-sampling basis to the wafer-sampling level, and hence upgrades the sampling strategy from random-sampling to full and right-sampling. In this study, various typical defect cases are utilized to illustrate how to create VI models and verify the reliability of the proposed approach. Furthermore, a feasible architecture of the VI implementation for mass production in semiconductor factory is presented in the paper.     

基于视觉注意机制的棉花污染物机器视觉检测算法

针对棉花中污染物这类大背景中的目标检测,模仿人类视觉注意机制,提出了一种检测算法:在预注意阶段,主相机获取全局图像,利用离散余弦变换和支持向量机提取和识别特征,确定目标所在的感兴趣区域;在注意阶段,对应感兴趣区域的从相机工作,获取该区域的局部图像,利用均值和方差方法识别污染物。实验表明,该算法能去除冗余数据,提高检测精确度。     

Automatic visual inspection: An approach with multi-instance learning

One of the industrial applications of computer vision is automatic visual inspection. In the last decade, standard supervised learning methods have been used to detect defects in different kind of products. These methods are trained with a set of images where every image has to be manually segmented and labeled by experts in the application domain. These manual segmentations require a large amount of high quality delineations (on pixels), which can be time consuming and often a difficult task. Multi-instance learning (MIL), in contrast to standard supervised classifiers, avoids this task and can, therefore, be trained with weakly labeled images. In this paper, we propose an approach for the automatic visual inspection that uses MIL for defect detection. The approach has been tested with data from three artificial benchmark datasets and three real-world industrial scenarios: inspection of artificial teeth, weld defect detection and fishbone detection. Results show that the proposed approach can be used with weakly labeled images for defect detection on automatic visual inspection systems. This approach is able to increase the area under the receiver-operating characteristic curve (AUC) up to 6.3% compared with the naïve MIL approach of propagating the bag labels.     

工业零件视觉检测系统中标定设备的误差分析

提出了一种简单、易于操作的自动标定设备,CCD摄像机可以在转盘旋转过程中,根据需要灵活的改变Z、Y、ω这三个自由度。并通过理论推导和实验验证了此自动设备的稳定性与可靠性。     

Knowledge augmented broad learning system for computer vision based mixed-type defect detection in semiconductor manufacturing

Defect detection is a critical measurement process for intelligent manufacturing systems to provide insights for product quality improvement. For complex products such as integrated circuit wafers, several types of defects are usually coupled in a piece of wafer to form a mixed-type defect, which poses a challenge to current defect detection methods. This paper proposed a knowledge augmented broad learning system with a knowledge module and broad selective sampling module, which provides a multichannel selective sampling network to decouple the mixed-type defects. In this model, each channel is equipped with a pre-trained deformable convolution model to extract the feature of a fixed single-type defect. The knowledge module is designed to activate the candidate network channel by pre-detection of wafer maps. The experiment results indicated that the proposed model outperforms conventional models and other deep learning models, which demonstrated that the knowledge augmented broad selective sampling mechanism is effective for mixed-type defect detection.     

视觉检测系统及其应用

视觉检测技术是建立在计算机视觉研究基础上的一门新兴检测技术，基于视觉传感器的检测系统具有抗干扰能力强。效率高，组成简单等优点，非常适合生产现场的在线，非接触检测及监控。本文介绍了该技术在烟草行业包装生产线上的应用方案。     

Instance invariant visual servoing framework for part‐aware autonomous vehicle inspection using MAVs

Visual servoing approaches navigate a robot to the desired pose with respect to a given object using image measurements. As a result, these approaches have several applications in manipulation, navigation and inspection. However, existing visual servoing approaches are instance specific, that is, they control camera motion between two views of the same object. In this paper, we present a framework for visual servoing to a novel object instance. We further employ our framework for the autonomous inspection of vehicles using Micro Aerial Vehicles (MAVs), which is vital for day‐to‐day maintenance, damage assessment, and merchandising a vehicle. This visual inspection task comprises the MAV visiting the essential parts of the vehicle, for example, wheels, lights, and so forth, to get a closer look at the damages incurred. Existing methods for autonomous inspection could not be extended for vehicles due to the following reasons: First, several existing methods require a 3D model of the structure, which is not available for every vehicle. Second, existing methods require expensive depth sensor for localization and path planning. Third, current approaches do not account for the semantic understanding of the vehicle, which is essential for identifying parts. Our instance invariant visual servoing framework is capable of autonomously navigating to every essential part of a vehicle for inspection and can be initialized from any random pose. To the best our knowledge, this is the first approach demonstrating fully autonomous visual inspection of vehicles using MAVs. We have validated the efficacy of our approach through a series of experiments in simulation and outdoor scenarios.     

Evaluating the 300 mm wafer-handling task in semiconductor industry

The semiconductor industry is moving from the production of 200 mm wafers to 300 mm wafers. With the increase in wafer size, the workload of wafer handling tasks is also increasing. This study evaluated the operator's handling capability, and the risk of having musculoskeletal disorders (MSDs) for handling 300 mm wafers. Twenty-four female operators from a semiconductor manufacturing company participated in the experiment. Subject’s psychophysical, and physiological responses were measured. Posture analysis and biomechanics analysis were also conducted to evaluate the local strain at different joints. The results suggest that loading and unloading front opening universal pod (FOUP) at 90/90 cm is best for enhancing handling capability, wrist posture, and minimizing L5/S1 compression force. About 30% of the operator's FOUP handling capabilities were lower than the safe load limit (25%ile Maximal alleptable weight of lift) recommended by SEMI (Semiconductor Equipment and Materials International). Handling wafers at 90/125 tends to increase the risk of causing shoulder and wrist injuries. Handling wafers at 90/35 tends to induce greater strength loading at hip joint, and cause greater wrist radial deviation. To minimize the risk of having MSDs, countermeasures such as training appropriate method of FOUP handling, minimizing manual carrying, specifying ergonomic machine equipment interface, and assigning male operators for high frequency FOUP handling tasks were proposed. Positive feedbacks are obtained.

Relevance to industry

The results of this study provided useful information to improve the design of 300 mm wafer handling task, which enhanced the safety and health of operators in the semiconductor manufacturing workplace.     

An industrial visual inspection system that uses inductive learning

This paper presents an industrial visual inspection system that uses inductive learning. The system employs RULES-3 inductive learning algorithm to extract the necessary set of rules and template matching technique to process an image. Twenty 3×3 masks are used to represent an image. Each example consists of 20 frequencies of each mask. The system was tested on five different types of tea or water cups in order to classify the good and bad items. The system was trained using five good cups and then tested for 113 unseen examples. The results obtained showed the high performance of the system: the efficiency of the system for correctly classifying unseen examples was 100%. The system can also decide what type of the cup is being processed.     

西林瓶缺陷自动视觉检测方法与系统研究

与传统的西林瓶缺陷人工检测方法相比,在线视觉自动检测具有巨大优越性。设计了一套西林瓶缺陷自动视觉在线检测系统,研究了系统的机械与电气结构,并设计了针对西林瓶的视觉成像与图像获取装置;根据西林瓶各部位的形状特殊性,针对性地提出了具体的检测方法。选取各类缺陷样品100瓶进行测试,结果表明该系统检测准确度较高,能满足实际生产的需求。     

管接头上密封圈数量在线视觉检测方法研究

针对在镀铬管接头密封圈数量的视觉检测应用中,图像中的工件方位不固定,易出现阴影和热区等问题,采用阈值分割、细化、尺寸滤波的基本图像处理步骤,并通过采取P tile方法确定图像分割的阈值,在细化后的图像中打断有分支的线条和曲率较大的线条,及在尺寸滤波后的图像中根据几何特征进行断开线条的连接等技术措施保证了图像处理结果的稳定性。实际图像的处理结果表明,该图像处理算法能够快速准确地检测出工件上的密封圈数量,满足应用的要求。     

Recent advances in the automatic inspection of integrated circuits for pattern defects

Automatic inspection has become an essential part of manufacturing technology for integrated circuit (IC) chips, but three trends in the geometries of ICs and the chips that they comprise have serious implications for inspection, making further advances in technology challenging. The individual devices (e.g., transistors) are becoming smaller, with the smallest features on some advanced products already crossing the optical resolution threshold; the chip areas are becoming larger; and the chips consist of more layers and undergo more processing steps. Not only are the smallest defects more difficult to detect due to the optical resolution limit, they are also much rarer because the tolerable defect density decreases as the chip area increases. This paper addresses automated IC inspection, surveying recent advances and future challenges. An overview of all inspection on IC chips during the manufacturing process is followed by a detailed discussion of pattern defect inspection (PDI) and its unique requirements, such as detection probability, false alarm rate, throughput, and minimum defect size. The core material of the paper consists of a discussion of approaches and systems for PDI, emphasizing recent developments, but reviewing older work to set the proper context. Both work reported in the literature and commercial systems are considered.