Precise 3D image alignment in micro-axial tomography

Micro (µ‐) axial tomography is a challenging technique in microscopy which improves quantitative imaging especially in cytogenetic applications by means of defined sample rotation under the microscope objective. The advantage of µ‐axial tomography is an effective improvement of the precision of distance measurements between point‐like objects. Under certain circumstances, the effective (3D) resolution can be improved by optimized acquisition depending on subsequent, multi‐perspective image recording of the same objects followed by reconstruction methods. This requires, however, a very precise alignment of the tilted views. We present a novel feature‐based image alignment method with a precision better than the full width at half maximum of the point spread function. The features are the positions (centres of gravity) of all fluorescent objects observed in the images (e.g. cell nuclei, fluorescent signals inside cell nuclei, fluorescent beads, etc.). Thus, real alignment precision depends on the localization precision of these objects. The method automatically determines the corresponding objects in subsequently tilted perspectives using a weighted bipartite graph. The optimum transformation function is computed in a least squares manner based on the coordinates of the centres of gravity of the matched objects. The theoretically feasible precision of the method was calculated using computer‐generated data and confirmed by tests on real image series obtained from data sets of 200 nm fluorescent nano‐particles. The advantages of the proposed algorithm are its speed and accuracy, which means that if enough objects are included, the real alignment precision is better than the axial localization precision of a single object. The alignment precision can be assessed directly from the algorithm's output. Thus, the method can be applied not only for image alignment and object matching in tilted view series in order to reconstruct (3D) images, but also to validate the experimental performance (e.g. mechanical precision of the tilting). In practice, the key application of the method is an improvement of the effective spatial (3D) resolution, because the well‐known spatial anisotropy in light microscopy can be overcome. This allows more precise distance measurements between point‐like objects.     

Correction of autofocusing errors due to specimen tilt for automated electron tomography

Transmission electron microscopy images acquired under tilted‐beam conditions experience an image shift as a function of defocus settings – a fact that is exploited as a method for defocus determination in most of the automated tomography data collection systems. Although the method was shown to be highly accurate for a large variety of specimens, we point out that in its original design it can strictly only be applied to images of untilted samples. The application to tilted samples and thus in automated electron tomography is impaired mainly due to a defocus change across the images, resulting in reduced accuracy. In this communication we present a method that can be used to improve the accuracy of the basic autofocusing procedures currently used in systems for automated electron tomography.     

Refinement procedure for the image alignment in high-resolution electron tomography

High-resolution electron tomography from a tilt series of transmission electron microscopy images requires an accurate image alignment procedure in order to maximise the resolution of the tomogram. This is the case in particular for ultra-high resolution where even very small misalignments between individual images can dramatically reduce the fidelity of the resultant reconstruction. A tomographic-reconstruction based and marker-free method is proposed, which uses an iterative optimisation of the tomogram resolution. The method utilises a search algorithm that maximises the contrast in tomogram sub-volumes. Unlike conventional cross-correlation analysis it provides the required correlation over a large tilt angle separation and guarantees a consistent alignment of images for the full range of object tilt angles. An assessment based on experimental reconstructions shows that the marker-free procedure is competitive to the reference of marker-based procedures at lower resolution and yields sub-pixel accuracy even for simulated high-resolution data.     

Automated high-throughput electron tomography by pre-calibration of image shifts

Electron tomography is a versatile method for obtaining three‐dimensional (3D) images with transmission electron microscopy. The technique is suitable to investigate cell organelles and tissue sections (100–500 nm thick) with 4–20 nm resolution. 3D reconstructions are obtained by processing a series of images acquired with the samples tilted over different angles. While tilting the sample, image shifts and defocus changes of several µm can occur. The current generation of automated acquisition software detects and corrects for these changes with a procedure that incorporates switching the electron optical magnification. We developed a novel method for data collection based on the measurement of shifts prior to data acquisition, which results in a five‐fold increase in speed, enabling the acquisition of 151 images in less than 20 min. The method will enhance the quality of a tilt series by minimizing the amount of required focus‐change compensation by aligning the optical axis to the tilt axis of the specimen stage. The alignment is achieved by invoking an amount of image shift as deduced from the mathematical model describing the effect of specimen tilt. As examples for application in biological and materials sciences 3D reconstructions of a mitochondrion and a zeolite crystal are presented.     

Registration of phase‐contrast images in propagation‐based X‐ray phase tomography

X‐ray phase tomography aims at reconstructing the 3D electron density distribution of an object. It offers enhanced sensitivity compared to attenuation‐based X‐ray absorption tomography. In propagation‐based methods, phase contrast is achieved by letting the beam propagate after interaction with the object. The phase shift is then retrieved at each projection angle, and subsequently used in tomographic reconstruction to obtain the refractive index decrement distribution, which is proportional to the electron density. Accurate phase retrieval is achieved by combining images at different propagation distances. For reconstructions of good quality, the phase‐contrast images recorded at different distances need to be accurately aligned. In this work, we characterise the artefacts related to misalignment of the phase‐contrast images, and investigate the use of different registration algorithms for aligning in‐line phase‐contrast images. The characterisation of artefacts is done by a simulation study and comparison with experimental data. Loss in resolution due to vibrations is found to be comparable to attenuation‐based computed tomography. Further, it is shown that registration of phase‐contrast images is nontrivial due to the difference in contrast between the different images, and the often periodical artefacts present in the phase‐contrast images if multilayer X‐ray optics are used. To address this, we compared two registration algorithms for aligning phase‐contrast images acquired by magnified X‐ray nanotomography: one based on cross‐correlation and one based on mutual information. We found that the mutual information‐based registration algorithm was more robust than a correlation‐based method.     

Alignment of tomographic projections using an incomplete set of fiducial markers

Reconstruction of three-dimensional images using tomography requires that the projections be aligned along a common rotational axis. We present here a solution to the problem of alignment for single-axis tomography using fiducial markers. The algorithm is based on iterative linearization of the projection equations and is least-squares-optimized by a linear least-squares solution instead of a gradient search. The algorithm does not require markers to be available in every projection, and initial estimates are unnecessary. Program execution is robust, fast, and can quickly align large data sets containing 256 or more projections.     

Precision metrology of integrated circuit critical dimensions using in situ differential scanning electron microscopy

A technique for performing precision measurements of critical dimensions (CD) on integrated circuit (IC) features using the scanning electron microscope (SEM) is presented. The method is based on obtaining true in situ differential SEM video signals of IC features. The technique is characterized by its high sensitivity to feature edges, and relative slopes. Results are presented on the application of this technique to SEM measurements of IC patterns in silicon, photoresist, and aluminum. The method affords the capability of accurate pattern alignment during SEM imaging preceding linewidth measurements.     

Calibration method of tilt and azimuth angles for alignment of TEM tomographic tilt series

This paper describes the calibration method of the tilt and azimuth angles of specimen using a digital protractor and a laser autocollimator for alignment of electron tomography. It also suggests an easy method to check whether the specimen is tilted by 180.0°, and whether the azimuth angle is 0.0°; the method involves the use of two images of a rod-shaped specimen collected before and after a 180.0° tilt. The method is based on the assumption that these images are symmetric about the tilt axis when the azimuth angle is 0.0°. In addition, we used an experiment to demonstrate the effect of the incorrect angles on reconstructed images and simulated the image quality against distance away from tilt axis.     

Automatic seamless mosaicing of microscopic images: enhancing appearance with colour degradation compensation and wavelet-based blending

In order to observe the fine details of biomedical specimens, various kinds of high-magnification microscopes are used. However, they suffer from a limited field of view when visualizing highly magnified specimens. Image mosaicing techniques are necessary to integrate two or more partially overlapping images into one and make the whole specimen visible. In this study, we propose a new system that automatically creates panoramic images by mosaicing all the microscopic images acquired from a specimen. Not only does it effectively compensate for the congenital narrowness in microscopic views, but it also results in the mosaiced image containing as little distortion with respect to the originals as possible. The system consists of four main steps: (1) feature point extraction using multiscale wavelet analysis, (2) image matching based on feature points or by projection profile alignment, (3) colour difference adjustment and optical degradation compensation with a Gaussian-like model and (4) wavelet-based image blending. In addition to providing a precise alignment, the proposed system also takes into account the colour deviations and degradation in image mosaicing. The visible seam lines are eliminated after image blending. The experimental results show that the system performs well on differently stained image sequences and is effective on acquired images with large colour variations and degradation. It is expected to be a practical tool for microscopic image mosaicing.     

Three‐dimensional imaging of whole mouse models: comparing nondestructive X‐ray phase‐contrast micro‐CT with cryotome‐based planar epi‐illumination imaging

In this study, we compare two evolving techniques for obtaining high‐resolution 3D anatomical data of a mouse specimen. On the one hand, we investigate cryotome‐based planar epi‐illumination imaging (cryo‐imaging). On the other hand, we examine X‐ray phase‐contrast micro‐computed tomography (micro‐CT) using synchrotron radiation. Cryo‐imaging is a technique in which an electron multiplying charge coupled camera takes images of a cryo‐frozen specimen during the sectioning process. Subsequent image alignment and virtual stacking result in volumetric data. X‐ray phase‐contrast imaging is based on the minute refraction of X‐rays inside the specimen and features higher soft‐tissue contrast than conventional, attenuation‐based micro‐CT. To explore the potential of both techniques for studying whole mouse disease models, one mouse specimen was imaged using both techniques. Obtained data are compared visually and quantitatively, specifically with regard to the visibility of fine anatomical details. Internal structure of the mouse specimen is visible in great detail with both techniques and the study shows in particular that soft‐tissue contrast is strongly enhanced in the X‐ray phase images compared to the attenuation‐based images. This identifies phase‐contrast micro‐CT as a powerful tool for the study of small animal disease models.     

Correlation-based methods of automatic particle detection in electron microscopy images with smoothing by anisotropic diffusion

Two methods of correlation‐based automatic particle detection in electron microscopy images are compared – computing a cross‐correlation function or a local correlation coefficient vs. azimuthally averaged reference projections (either from a model or from experimental particle images). The ability of smoothing images by anisotropic diffusion to improve the performance of particle detection is also considered. Anisotropic diffusion is an effective method of preprocessing that enhances the edges and overall shape of particles while reducing noise. It is found that anisotropic diffusion improves particle detection by a local correlation coefficient when projections from a high‐resolution reconstruction are used as references. When references from experimental particle images are used, a cross‐correlation function shows a more marked improvement in particle detection in images smoothed by anisotropic diffusion.     

Entropy‐assisted image segmentation for nano‐ and micro‐sized networks

Image analysis is an important tool for characterizing nano/micro network structures. To understand the connection, organization and proper alignment of network structures, the knowledge of the segments that represent the materials inside the image is very necessary. Image segmentation is generally carried out using statistical methods. In this study, we developed a simple and reliable masking method that improves the performance of the indicator kriging method by using entropy. This method selectively chooses important pixels in an image (optical or electron microscopy image) depending on the degree of information required to assist the thresholding step. Reasonable threshold values can be obtained by selectively choosing important pixels in a complex network image composed of extremely large numbers of thin and narrow objects. Thus, the overall image segmentation can be improved as the number of disconnected objects in the network is minimized. Moreover, we also proposed a new method for analyzing high‐pixel resolution images on a large scale and optimized the time‐consuming steps such as covariance estimation of low‐pixel resolution image, which is rescaled by performing the affine transformation on high‐pixel resolution images. Herein, image segmentation is executed in the original high‐pixel resolution image. This entropy‐based masking method of low‐pixel resolution significantly decreases the analysis time without sacrificing accuracy.     

基于旋转投影二进制描述符的空间目标位姿估计

为了实现基于点云的空间目标相对位姿快速估计,提出一种旋转投影二进制描述符(BRoPH)。该描述符首先建立特征点处的局部参考坐标系,然后通过旋转投影局部点云生成不同视角下的密度图像块和深度图像块,最后根据图像块生成特征点的多尺度二进制字符串。针对位姿估计对实时性的要求,在分析BRoPH Hamming距离分布的基础上,提出了基于Hamming距离阈值的特征匹配策略,用于剔除潜在的错误配对,加快位姿估计收敛速度。最后,在基于局部特征描述符位姿估计框架下分别与SHOT描述符和FPFH描述符进行了比较。结果表明:BRoPH描述符在仅需要SHOT和FPFH平均内存1/80的基础上,得到了远高于SHOT和FPFH的平均位姿估计精度,其平均姿态误差小于0.1°,平均位置误差小于1/180 R。此外,基于Hamming距离阈值的特征匹配策略使得BRoPH的位姿粗估计速度加快了7倍,总体位姿估计频率超过7Hz,比SHOT和FPFH分别快3~6.8倍。该方法具有占用内存小、计算速度快、位姿估计精度高和抗干扰能力强等优点,满足基于点云的空间目标位姿估计实时性要求。     

Non‐rigid alignment in electron tomography in materials science

Electron tomography is a key technique that enables the visualization of an object in three dimensions with a resolution of about a nanometre. High‐quality 3D reconstruction is possible thanks to the latest compressed sensing algorithms and/or better alignment and preprocessing of the 2D projections. Rigid alignment of 2D projections is routine in electron tomography. However, it cannot correct misalignments induced by (i) deformations of the sample due to radiation damage or (ii) drifting of the sample during the acquisition of an image in scanning transmission electron microscope mode. In both cases, those misalignments can give rise to artefacts in the reconstruction. We propose a simple‐to‐implement non‐rigid alignment technique to correct those artefacts. This technique is particularly suited for needle‐shaped samples in materials science. It is initiated by a rigid alignment of the projections and it is then followed by several rigid alignments of different parts of the projections. Piecewise linear deformations are applied to each projection to force them to simultaneously satisfy the rigid alignments of the different parts. The efficiency of this technique is demonstrated on three samples, an intermetallic sample with deformation misalignments due to a high electron dose typical to spectroscopic electron tomography, a porous silicon sample with an extremely thin end particularly sensitive to electron beam and another porous silicon sample that was drifting during image acquisitions.     

Correction of image drift and distortion in a scanning electron microscopy

Continuous research on small‐scale mechanical structures and systems has attracted strong demand for ultrafine deformation and strain measurements. Conventional optical microscope cannot meet such requirements owing to its lower spatial resolution. Therefore, high‐resolution scanning electron microscope has become the preferred system for high spatial resolution imaging and measurements. However, scanning electron microscope usually is contaminated by distortion and drift aberrations which cause serious errors to precise imaging and measurements of tiny structures. This paper develops a new method to correct drift and distortion aberrations of scanning electron microscope images, and evaluates the effect of correction by comparing corrected images with scanning electron microscope image of a standard sample. The drift correction is based on the interpolation scheme, where a series of images are captured at one location of the sample and perform image correlation between the first image and the consequent images to interpolate the drift–time relationship of scanning electron microscope images. The distortion correction employs the axial symmetry model of charged particle imaging theory to two images sharing with the same location of one object under different imaging fields of view. The difference apart from rigid displacement between the mentioned two images will give distortion parameters. Three‐order precision is considered in the model and experiment shows that one pixel maximum correction is obtained for the employed high‐resolution electron microscopic system.     

Projected potential profiles across interfaces obtained by reconstructing the exit face wave function from through focal series

An iterative method for reconstructing the exit face wave function from a through focal series of transmission electron microscopy image line profiles across an interface is presented. Apart from high-resolution images recorded with small changes in defocus, this method works also well for a large defocus range as used for Fresnel imaging. Using the phase-object approximation the projected electrostatic as well as the absorptive potential profiles across an interface are determined from this exit face wave function. A new experimental image alignment procedure was developed in order to align images with large relative defocus shift. The performance of this procedure is shown to be superior to other image alignment procedures existing in the literature. The reconstruction method is applied to both simulated and experimental images.     

Accurate marker-free alignment with simultaneous geometry determination and reconstruction of tilt series in electron tomography

An image alignment method for electron tomography is presented which is based on cross-correlation techniques and which includes a simultaneous refinement of the tilt geometry. A coarsely aligned tilt series is iteratively refined with a procedure consisting of two steps for each cycle: area matching and subsequent geometry correction. The first step, area matching, brings into register equivalent specimen regions in all images of the tilt series. It determines four parameters of a linear two-dimensional transformation, not just translation and rotation as is done during the preceding coarse alignment with conventional methods. The refinement procedure also differs from earlier methods in that the alignment references are now computed from already aligned images by reprojection of a backprojected volume. The second step, geometry correction, refines the initially inaccurate estimates of the geometrical parameters, including the direction of the tilt axis, a tilt angle offset, and the inclination of the specimen with respect to the support film or specimen holder. The correction values serve as an indicator for the progress of the refinement. For each new iteration, the correction values are used to compute an updated set of geometry parameters by a least squares fit. Model calculations show that it is essential to refine the geometrical parameters as well as the accurate alignment of the images to obtain a faithful map of the original structure.     

Reconstructing the microstructure of polyimide–silicalite mixed‐matrix membranes and their particle connectivity using FIB‐SEM tomography

Properties of a composite material made of a continuous matrix and particles often depend on microscopic details, such as contacts between particles. Focusing on processing raw focused‐ion beam scanning electron microscope (FIB‐SEM) tomography data, we reconstructed three mixed‐matrix membrane samples made of 6FDA‐ODA polyimide and silicalite‐1 particles. In the first step of image processing, backscattered electron (BSE) and secondary electron (SE) signals were mixed in a ratio that was expected to obtain a segmented 3D image with a realistic volume fraction of silicalite‐1. Second, after spatial alignment of the stacked FIB‐SEM data, the 3D image was smoothed using adaptive median and anisotropic nonlinear diffusion filters. Third, the image was segmented using the power watershed method coupled with a seeding algorithm based on geodesic reconstruction from the markers. If the resulting volume fraction did not match the target value quantified by chemical analysis of the sample, the BSE and SE signals were mixed in another ratio and the procedure was repeated until the target volume fraction was achieved. Otherwise, the segmented 3D image (replica) was accepted and its microstructure was thoroughly characterized with special attention paid to connectivity of the silicalite phase. In terms of the phase connectivity, Monte Carlo simulations based on the pure‐phase permeability values enabled us to calculate the effective permeability tensor, the main diagonal elements of which were compared with the experimental permeability. In line with the hypothesis proposed in our recent paper (?apek, P. et al. (2014) Comput. Mater. Sci. 89 , 142–156), the results confirmed that the existence of particle clusters was a key microstructural feature determining effective permeability.     

3D correlative morphological and elemental characterization of materials at the deep submicrometre scale

This paper shows how X‐ray computed nanotomography (CNT) can be correlated with focused ion beam time‐of‐flight secondary ion mass spectrometry (FIB‐TOF‐SIMS) tomography on the same sample to investigate both the morphological and elemental structure. This methodology is applicable to relatively large specimens with dimensions of several tens of microns whilst maintaining a high spatial resolution of the order of 100 nm. However, combining X‐ray CNT and FIB‐TOF‐SIMS tomography requires innovative sample preparation protocols to allow both experiments to be conducted on exactly the same sample without chemically or structurally modifying the sample between measurements. Moreover, dedicated algorithms have been developed for effective data fusion that is biased with nine degrees of freedom. This methodology has been tested using a porous and heterogeneous solid oxide fuel cell (SOFC) that has features varying in size by three orders of magnitude – from hundreds of nanometre large pores and grains to tens of micron wide functional layers.