一种Spark环境下的高效率大规模图数据处理机制

本文针对现有的图处理和图管理框架存在的效率低下以及数据存储结构等问题,提出了一种适合于大规模图数据处理机制。首先分析了目前的一些图处理模型以及图存储框架的优势与存在的不足。其次,通过对分布式计算的特性分析采取适合大规模图的分割算法、数据抽取的优化以及缓存、计算层与持久层结合机制三方面来设计本文的图数据处理框架。最后通过PageRank和SSSP算法来设计实验与MapReduce框架和采用HDFS作持久层的Spark框架做性能对比。实验证明本文提出的框架要比MapReduce框架快90倍,比采用HDFS作持久层的Spark框架快2倍,能够满足高效率图数据处理的应用前景。     

基于MapReduce和改进密度峰值的划分聚类算法

对于基于划分的聚类算法随机选取初始聚类中心导致初始中心敏感,聚类结果不稳定、集群效率低等问题,提出一种基于MapReduce框架和改进的密度峰值的划分聚类算法(based on MapReduce framework and im-proved density peak partition clustering algorithm,MR-IDPACA).首先,通过自然最近邻定义新的局部密度计算方式,将搜索样本密度峰值点作为划分聚类算法的初始聚类中心;其次针对算法在大规模数据下运行时间复杂,提出基于E2LSH(exact Euclidean locality sensitive hashing)的一种分区方法,即KLSH(K of locality sensitive hashing).通过该方法对数据分区后结合MapReduce框架并行搜寻初始聚类中心,有效减少了算法在搜索初始聚类中心时的运行时间;对于MapReduce框架中的数据倾斜问题,提出ME(multistage equilibrium)策略对中间数据进行多段均衡分区,以提升算法运行效率;在MapReduce框架下并行聚类,得到最终聚类结果.实验得出MR-IDPACA算法在单机环境下有着较高的准确率和较强的稳定性,集群性能上也有着较好的加速比和运行时间,聚类效果有所提升.     

融合区域和边缘特征的水平集水下图像分割

目的 水平集模型是图像分割中的一种先进方法,在陆地环境图像分割中展现出较好效果。特征融合策略被广泛引入到该模型框架,以拉伸目标-背景对比度,进而提高对高噪声、杂乱纹理等多类复杂图像的处理性能。然而,在水下环境中,由于水体高散射、强衰减等多因素的共同作用,使得现有图像特征及水平集模型难以适用于对水下图像的分割任务,分割结果与目标形态间存在较大差异。鉴于此,提出一种适用于水下图像分割的区域-边缘水平集模型,以提高水下图像目标分割的准确性。方法 综合应用图像的区域特征及边缘特征对水下目标进行辨识。对于区域特征,引入水下图像显著性特征;对于边缘特征,创新性地提出了一种基于深度信息的边缘特征提取方法。所提方法在融合区域级和边缘级特征的基础上,引入距离正则项对水平集函数进行规范,以增强水平集函数演化的稳定性。结果 基于YouTube和Bubblevision的水下数据集的实验结果表明,所提方法不仅对高散射强衰减的低对比度水下图像实现较好的分割效果,同时对处理强背景噪声图像也有较好的鲁棒性,与水平集分割方法（local pre-fitting,LPF）相比,分割精确度至少提高11.5%,与显著性检测方法（hierarchical co-salient detection via color names,HCN）相比,精确度提高6.7%左右。结论 实验表明区域-边缘特征融合以及其基础上的水平集模型能够较好地克服水下图像分割中的部分难点,所提方法能够较好分割水下目标区域并拟合目标轮廓,与现有方法对比获得了较好的分割结果。     

Effective multiresolution arc segmentation: algorithms and performance evaluation

Arc segmentation plays an important role in the process of graphics recognition from scanned images. The GREC arc segmentation contest shows that there is a lot of room for improvement in this area. This paper proposes a multiresolution arc segmentation method based on our previous seeded circular tracking algorithm which largely depends on the OOPSV model. The newly-introduced multiresolution paradigm can handle arcs/circles with large radii well. We describe new approaches for arc seed detection, arc localization, and arc verification, making the proposed method self-contained and more efficient. Moreover, this paper also brings major improvement to the dynamic adjustment algorithm of circular tracking to make it more robust. A systematic performance evaluation of the proposed method has been conducted using the third-party evaluation tool and test images obtained from the GREC arc segmentation contests. The overall performance over various arc angles, arc lengths, line thickness, noises, arc-arc intersections, and arc-line intersections has been measured. The experimental results and time complexity analyses on real scanned images are also reported and compared with other approaches. The evaluation result demonstrates the stable performance and the significant improvement on processing large arcs/circles of the MAS method.     

An improved K-means algorithm for underwater image background segmentation

Conventional algorithms fail to obtain satisfactory background segmentation results for underwater images. In this study, an improved K-means algorithm was developed for underwater image background segmentation to address the issue of improper K value determination and minimize the impact of initial centroid position of grayscale image during the gray level quantization of the conventional K-means algorithm. A total of 100 underwater images taken by an underwater robot were sampled to test the aforementioned algorithm in respect of background segmentation validity and time cost. The K value and initial centroid position of grayscale image were optimized. The results were compared to the other three existing algorithms, including the conventional K-means algorithm, the improved Otsu algorithm, and the Canny operator edge extraction method. The experimental results showed that the improved K-means underwater background segmentation algorithm could effectively segment the background of underwater images with a low color cast, low contrast, and blurred edges. Although its cost in time was higher than that of the other three algorithms, it none the less proved more efficient than the time-consuming manual segmentation method. The algorithm proposed in this paper could potentially be used in underwater environments for underwater background segmentation.

     

基于Co-PSPNet的轻量级水下鱼体图像分割算法

为解决水下环境中图像模糊和资源有限等问题,提出了一种基于Co-PSPNet网络的轻量级水下鱼体图像分割算法;主干网络使用了MobileNetV2,以降低网络的计算复杂度和参数量;引入了CoordConv模块,以增强网络对鱼体边界等细节信息的感知能力;将全局池化后的特征作为注意力机制网络的输入,以增强具有较高语义信息的特征;经过大量的实验评估,该算法在公开的水下鱼体图像数据集上取得了优越的性能;实际应用中,该算法满足了水下生态研究和水下机器人领域对水下鱼体图像分割的工程需求;通过对水下环境下图像模糊和资源限制等问题的解决,该算法为水下生态研究和水下机器人领域的应用提供了有效的图像分割解决方案。     

基于MapReduce框架下K-means的改进算法

针对海量数据背景下K-means聚类结果不稳定和收敛速度较慢的问题,提出了基于MapReduce框架下的K-means改进算法。首先,为了能获得K-means聚类的初始簇数,利用凝聚层次聚类法对数据集进行聚类,并用轮廓系数对聚类结果进行初步评价,将获得数据集的簇数作为K-means算法的初始簇中心进行聚类;其次,为了能适应于海量数据的聚类挖掘,将改进的K-means算法部署在MapReduce框架上进行运算。实验结果表明,在单机性能上,该方法具有较高的准确率和召回率,同时也具有较强的聚类稳定性;在集群性能上,也具有较好的加速比和运行速度。     

MapReduce-based clustering for near-duplicate image identification

In this paper, an effective algorithm is developed for tackling the problem of near-duplicate image identification from large-scale image sets, where the LLC (locality-constrained linear coding) method is seamlessly integrated with the maxIDF cut model to achieve more discriminative representations of images. By incorporating MapReduce framework for image clustering and pairwise merging, the near duplicates of images can be identified effectively from large-scale image sets. An intuitive strategy is also introduced to guide the process for parameter selection. Our experimental results on large-scale image sets have revealed that our algorithm can achieve significant improvement on both the accuracy rates and the computation efficiency as compared with other baseline methods.     

最小化多MapReduce任务总完工时间的分析模型及其应用

随着大规模的MapReduce集群广泛地用于大数据处理,特别是当有多个任务需要使用同一个Hadoop集群时,一个关键问题是如何最大限度地减少集群的工作时间,提高MapReduce作业的服务效率。可将多个MapReduce作业当做一个调度任务建模,观察发现多个任务的总完工时间和任务的执行顺序有密切关系。 研究目标是设计作业调度系统分析模型,最小化一批MapReduce作业的总完工时间。提出一个更好的调度策略和实现方法, 使整个调度系统符合经典Johnson算法的条件, 从而可使用经典Johnson算法在线性时间内获取总完工时间的最优解。同时,针对需要使用两个或多个资源池进行平衡的问题, 提出了一种线性时间解决方案, 优于已知的近似模拟方案。该理论模型可应用于提高系统响应速度、节能和负载均衡等方面, 对应的应用实例提供了证实。     

结合分水岭的纹理梯度各向异性图像分割

目的 目前,许多图像分割算法对含有丰富纹理信息的图像的分割效果并不理想,尤其是在不同纹理的边缘信息的保持方面。为了解决这一问题,提出一种基于连续纹理梯度信息的各向异性图像分割算法。方法 在分水岭算法的基础上,引入纹理梯度各向异性算法,能够在避免纹理信息影响分割效果的前提下,最大限度地保证纹理边缘信息的完整。针对纹理特征数据敏感的特性,本文将离散的图像高度信息映射到连续的纹理梯度空间,能够有效减少由细小差异造成的过分割现象。结果 本文方法在BSD500 Dataset和Stanford Background Dataset中选择了大量的纹理信息丰富的图片与最新的分割算法进行了实验与对比。本文方法在分割效果（降低过分割现象）、保持边缘信息和分割准确率等方面均获得明显改进,并在图像分割的平均准确率方面与最新算法进行比较发现,本文算法的平均分割准确率达到90.9%,明显超过了其他最新算法,验证了本文方法的有效性。结论 本文提出的基于分水岭的纹理梯度各向异性算法对纹理图像的分割具有保边和准确的特点,采用连续梯度空间的方法能够有效地减少传统分水岭算法的过分割现象。本文方法主要适用于纹理信息丰富（自然纹理和人工纹理）的图片。     

Optimized big data K-means clustering using MapReduce

Clustering analysis is one of the most commonly used data processing algorithms. Over half a century, K-means remains the most popular clustering algorithm because of its simplicity. Recently, as data volume continues to rise, some researchers turn to MapReduce to get high performance. However, MapReduce is unsuitable for iterated algorithms owing to repeated times of restarting jobs, big data reading and shuffling. In this paper, we address the problems of processing large-scale data using K-means clustering algorithm and propose a novel processing model in MapReduce to eliminate the iteration dependence and obtain high performance. We analyze and implement our idea. Extensive experiments on our cluster demonstrate that our proposed methods are efficient, robust and scalable.     

基于MapReduce的并行贝叶斯分类算法的设计与实现

针对现代大规模文本文档分类在单机计算机上训练和测试过程计算时间长,本文设计和实现了一种基于MapReduce架构的并行贝叶斯文本分类算法。在用普通PC搭建的Hadoop集群上研究实验,结果表明,基于MapReduce架构的贝叶斯文本自动分类算法处理大规模的文档自动分类时,在保证分类效果的情况下,并能获得接近线性的加速比。     

Automatic detection of Moroccan coastal upwelling zones using sea surface temperature images

An efficient unsupervised method is developed for automatic segmentation of the area covered by upwelling waters in the coastal ocean of Morocco using the Sea Surface Temperature (SST) satellite images. The proposed approach first uses the two popular unsupervised clustering techniques, k-means and fuzzy c-means (FCM), to provide different possible classifications to each SST image. Then several cluster validity indices are combined in order to determine the optimal number of clusters, followed by a cluster fusion scheme, which merges consecutive clusters to produce a first segmentation of upwelling area. The region-growing algorithm is then used to filter noisy residuals and to extract the final upwelling region. The performance of our algorithm is compared to a popular algorithm used to detect upwelling regions and is validated by an oceanographer over a database of 92 SST images covering each week of the years 2006 and 2007. The results show that our proposed method outperforms the latter algorithm, in terms of segmentation accuracy and computational efficiency.     

改进的并行关联规则增量挖掘算法

针对大数据环境下基于Can树(canonical order tree)的增量关联规则算法存在树结构空间占用过大、频繁模式挖掘效率不佳以及MapReduce集群并行化性能不足等问题,提出了一种基于粗糙集和归并剪枝方法改进的并行关联规则增量挖掘算法MR-PARIRM(MapReduce-based parallel association rules incremental mining algo-rithm using rough set and merge pruning).首先,设计了一种基于粗糙集的相似项合并策略RS-SIM(rough set based similar item merge)对数据集的相似项进行合并处理,并根据合并后的数据进行Can树构造,从而降低树结构的空间占用;其次,提出了一种归并剪枝策略MPS(merge pruning strategy)对树结构中的传播路径进行修剪合并,通过压缩频繁模式搜索空间来加快频繁项挖掘;最后,通过动态调度策略DSS(dynamic scheduling strategy)对异构式MapReduce集群中的计算任务进行动态调度,实现了负载均衡,有效提升了集群的并行化运算能力.最终的实验仿真结果表明,MR-PARIRM在大数据环境下具有相对较好的性能表现,适用于对大规模数据进行并行化处理.     

云环境中海量数据的并行分组密码体制研究

云计算环境中,飞速增长的海量数据的安全性越来越受到关注,分组密码算法是保证海量数据安全性的一个有效手段,但面对超大规模的数据量其效率是一个备受关注的问题。提出了一种基于MapReduce架构的并行分组密码机制,能够使标准的分组密码算法应用于大规模的集群环境中,通过并行化来提高海量数据加密与解密的执行效率,并设计了常用的几种并行工作模式。实验证明,提出的算法具有良好的可扩展性和高效的执行性能,能够适用于云计算环境中海量数据的安全保密,为进一步的研究工作奠定了基础。     

Efficient foreground extraction using RGB-D imaging

Image segmentation is one of the most important topics in the field of computer vision. As a result, many image segmentation approaches have been proposed, and interactive methods based on energy minimization such as GrabCut, have shown successful results. Automating the entire segmentation process is, however, very difficult because virtually all interactive methods require a considerable amount of user interaction. We believe that if additional information is provided to users in order to guide them effectively, the amount of interaction required can be reduced. Consequently, in this paper we propose an efficient foreground extraction algorithm, which utilizes depth information from RGB-D sensors such as Microsoft Kinect and offers users guidance in the foreground extraction process. Our approach can be applied as a pre-processing step for interactive and energy-minimization-based segmentation approaches. Our proposed method is able to segment the foreground from images and give hints that reduce interaction with users. In our method, we make use of the characteristics of depth information captured by RGB-D sensors and describe them using information from the structure tensor. Further, we show experimentally that our proposed method separates foreground from background sufficiently well for real world images.     

Image Segmentation Using a Local GMM in a Variational Framework

In this paper, we propose a new variational framework to solve the Gaussian mixture model (GMM) based methods for image segmentation by employing the convex relaxation approach. After relaxing the indicator function in GMM, flexible spatial regularization can be adopted and efficient segmentation can be achieved. To demonstrate the superiority of the proposed framework, the global, local intensity information and the spatial smoothness are integrated into a new model, and it can work well on images with inhomogeneous intensity and noise. Compared to classical GMM, numerical experiments have demonstrated that our algorithm can achieve promising segmentation performance for images degraded by intensity inhomogeneity and noise.     

Indoor scene understanding via RGB-D image segmentation employing depth-based CNN and CRFs

With the availability of low-cost depth-visual sensing devices, such as Microsoft Kinect, we are experiencing a growing interest in indoor environment understanding, at the core of which is semantic segmentation in RGB-D image. The latest research shows that the convolutional neural network (CNN) still dominates the image semantic segmentation field. However, down-sampling operated during the training process of CNNs leads to unclear segmentation boundaries and poor classification accuracy. To address this problem, in this paper, we propose a novel end-to-end deep architecture, termed FuseCRFNet, which seamlessly incorporates a fully-connected Conditional Random Fields (CRFs) model into a depth-based CNN framework. The proposed segmentation method uses the properties of pixel-to-pixel relationships to increase the accuracy of image semantic segmentation. More importantly, we formulate the CRF as one of the layers in FuseCRFNet to refine the coarse segmentation in the forward propagation, in meanwhile, it passes back the errors to facilitate the training. The performance of our FuseCRFNet is evaluated by experimenting with SUN RGB-D dataset, and the results show that the proposed algorithm is superior to existing semantic segmentation algorithms with an improvement in accuracy of at least 2%, further verifying the effectiveness of the algorithm.

     

High-resolution SAR images segmentation using NSCT denoising and QIGA based parameters selection of PCNN model

The high-resolution synthetic aperture radar (SAR) images usually contain inhomogeneous coherent speckle noises. For the high-resolution SAR image segmentation with such noises, the conventional methods based on pulse coupled neural networks (PCNN) have to face heavy parameters with a low efficiency. In order to solve the problems, this paper proposes a novel SAR image segmentation algorithm based on non-subsampling Contourlet transform (NSCT) denoising and quantum immune genetic algorithm (QIGA) improved PCNN models. The proposed method first denoising the SAR images for a pre-processing based on NSCT. Then, by using the QIGA to select parameters for the PCNN models, such models self-adaptively select the suitable parameters for segmentation of SAR images with different scenes. This method decreases the number of parameters in the PCNN models and improves the efficiency of PCNN models. At last, by using the optimal threshold to binary the segmented SAR images, the small objects and large scales from the original SAR images will be segmented. To validate the feasibility and effectiveness of the proposed algorithm, four different comparable experiments are applied to validate the proposed algorithm. Experimental results have shown that NSCT pre-processing has a better performance for coherent speckle noises suppression, and QIGA-PCNN model based on denoised SAR images has an obvious segmentation performance improvement on region consistency and region contrast than state-of-the-arts methods. Besides, the segmentation efficiency is also improved than conventional PCNN model, and the level of time complexity meets the state-of-the-arts methods. Our proposed NSCT+QIGA-PCNN model can be used for small object segmentation and large scale segmentation in high-resolution SAR images. The segmented results will be further used for object classification and recognition, regions of interest extraction, and moving object detection and tracking.

     

Optimizing MapReduce for energy efficiency

The efficient use of energy is essential to address concerns of cost and sustainability. Many data centers contain MapReduce clusters to process Big Data applications. A large number of machines and fault tolerance capabilities make MapReduce clusters energy inefficient. In this paper, we present a Configurator based on performance and energy models to improve the energy efficiency of MapReduce systems. Our solution is novel as it takes into account the dependence of the performance and energy consumption of a cluster on MapReduce parameters. While this dependence is known, we are the first to model it and design a Configurator to optimize these parameter settings for maximizing the energy efficiency of MapReduce systems. Our empirical evaluations show that the Configurator can result in up to 50% improvement in the energy efficiency of typical MapReduce applications in two architecturally different clusters.