密钥共享下跨用户密文数据去重挖掘方法

针对当前密文数据去重挖掘方法存在去重效果较差、特征聚合能力低的问题,提出一种密钥共享下跨用户密文数据去重挖掘方法.结合非线性统计序列分析方法对密钥共享下跨用户密文数据的统计特征进行采样,通过识别不同领域的统计特征进行密文数据的线性编码设计,抽取密钥共享下跨用户密文数据的平均互信息特征量.采用匹配滤波方法实现密钥共享下跨用户密文数据的去重处理.仿真结果表明,采用该方法的去重效果较好,特征聚合能力较强.     

A new content-defined chunking algorithm for data deduplication in cloud storage

Chunking is a process to split a file into smaller files called chunks. In some applications, such as remote data compression, data synchronization, and data deduplication, chunking is important because it determines the duplicate detection performance of the system. Content-defined chunking (CDC) is a method to split files into variable length chunks, where the cut points are defined by some internal features of the files. Unlike fixed-length chunks, variable-length chunks are more resistant to byte shifting. Thus, it increases the probability of finding duplicate chunks within a file and between files. However, CDC algorithms require additional computation to find the cut points which might be computationally expensive for some applications. In our previous work (Widodo et al., 2016), the hash-based CDC algorithm used in the system took more process time than other processes in the deduplication system. This paper proposes a high throughput hash-less chunking method called Rapid Asymmetric Maximum (RAM). Instead of using hashes, RAM uses bytes value to declare the cut points. The algorithm utilizes a fix-sized window and a variable-sized window to find a maximum-valued byte which is the cut point. The maximum-valued byte is included in the chunk and located at the boundary of the chunk. This configuration allows RAM to do fewer comparisons while retaining the CDC property. We compared RAM with existing hash-based and hash-less deduplication systems. The experimental results show that our proposed algorithm has higher throughput and bytes saved per second compared to other chunking algorithms.     

基于分片复用的多版本容器镜像加载方法

容器将应用和支持软件、库文件等封装为镜像，通过发布新版本镜像实现应用升级，导致不同版本之间存在大量相同数据.镜像加载消耗大量时间，使容器启动时间从毫秒级延迟为秒级甚至是分钟级.复用不同版本之间的相同数据，有利于减少容器加载时间.当前，容器镜像采用继承和分层加载机制，有效实现了支持软件、库文件等数据的复用，但对于应用内部数据还没有一种可靠的复用机制.提出一种基于分片复用的多版本容器镜像加载方法，通过复用不同版本镜像之间的相同数据，提升镜像加载效率.方法的核心思想是：利用边界匹配数据块切分方法将容器镜像切分为细粒度数据块，将数据块哈希值作为唯一标识指纹，借助B-树搜索重复指纹判断重复数据块，减少数据传输.实验结果表明，该方法可以提高5.8X以上容器镜像加载速度.     

Large-scale incremental processing with MapReduce

An important property of today’s big data processing is that the same computation is often repeated on datasets evolving over time, such as web and social network data. While repeating full computation of the entire datasets is feasible with distributed computing frameworks such as Hadoop, it is obviously inefficient and wastes resources. In this paper, we present HadUP (Hadoop with Update Processing), a modified Hadoop architecture tailored to large-scale incremental processing with conventional MapReduce algorithms. Several approaches have been proposed to achieve a similar goal using task-level memoization. However, task-level memoization detects the change of datasets at a coarse-grained level, which often makes such approaches ineffective. Instead, HadUP detects and computes the change of datasets at a fine-grained level using a deduplication-based snapshot differential algorithm (D-SD) and update propagation. As a result, it provides high performance, especially in an environment where task-level memoization has no benefit. HadUP requires only a small amount of extra programming cost because it can reuse the code for the map and reduce functions of Hadoop. Therefore, the development of HadUP applications is quite easy.     

基于压缩近邻的查重元数据去冗算法设计

随着重复数据删除次数的增加,系统中用于存储指纹索引的清单文件等元数据信息会不断累积,导致不可忽视的存储资源开销。因此,如何在不影响重复数据删除率的基础上,对重复数据删除过程中产生的元数据信息进行压缩,从而减小查重索引,是进一步提高重复数据删除效率和存储资源利用率的重要因素。针对查重元数据中存在大量冗余数据,提出了一种基于压缩近邻的查重元数据去冗算法Dedup2。该算法先利用聚类算法将查重元数据分为若干类,然后利用压缩近邻算法消除查重元数据中相似度较高的数据以获得查重子集,并在该查重子集上利用文件相似性对数据对象进行重复数据删除操作。实验结果表明,Dedup2可以在保持近似的重复数据删除比的基础上,将查重索引大小压缩50%以上。     

一种基于重复数据删除的网络文件备份系统设计与实现

针对广域网中网络备份系统存在备份速度低、网络带宽需求高和数据冗余等问题,提出并实现一种具有重复数据删除功能的网络文件备份系统。通过在客户端实现重复数据删除技术,对文件进行分块和在备份过程中去除重复数据块,减少客户端与服务期间需要传输的数据量,使得文件备份的速度获得较大提高,也较大地降低网络带宽要求,显著提高了网络备份系统的性能。此外,备份过程中客户端去除了重复的数据块使得服务器中存储的数据块都是唯一的,极大地降低了系统所需要的存储空间,同时,服务器端的数据块引用管理的设计很好地解决了空闲数据块耗费存储空间的问题。     

基于文件相似性分簇的重复数据消除模型

为解决现有提高重复数据消除系统吞吐量方法的局部性依赖和多节点依赖问题,提出了一种基于文件相似性分簇的重复数据消除模型。该模型将传统平面型索引结构拓展为空间结构,并依据Broder定理仅选择少量最具代表性的索引驻留在内存中;同时对索引进行横向分片并分布到完全自治的多个节点。实验结果表明,该方法能有效提高大规模云存储环境下重复数据消除性能和平均吞吐量,且各节点数据负载量均衡,故该模型可扩展性强。     

基于重复数据删除的快速文件归档方法

为了提高归档系统的存储效率及性能,提出了一种基于重复数据删除的快速文件归档方法（RAMBDD）,利用文件分块、比较数据块指纹、删除重复数据,实现了文件的数据块级归档。RAMBDD中给出了一种基于winnow-ing的重复数据删除方法LMCA,它在提高文件冗余检测率的同时也保证了文件分块的效率,并通过使用指纹快速检索方法和局部指纹缓存方法,减少了在查找不存在的数据块指纹时的磁盘读取次数,加速了查找重复数据块的过程。实验结果表明,与传统的文件归档方法相比,本方法大大节省了归档数据的存储空间和网络传输带宽,缩短了归档时间,提高了文件归档的效率。     

Research progress on secure data deduplication in cloud

In order to improve the efficiency of cloud storage and save the communication bandwidth, a deduplication mechanism for multi-duplicate of the same data in cloud environment was needed. However, the implement of the secure data deduplication was seriously hindered by the ciphertext in cloud. This issue has quickly aroused wide attention of academia and industry, and became a research hotspot. From a security standpoint, firstly the primary cause and the main challenges of secure data deduplication in cloud environment was analyzed, and then the deduplication system model as well as its security model was described. Furthermore, focusing on the realization mechanism of secure data deduplica-tion, the thorough analyses were carried on and reviews for the related research works in recent years from content-based encryption, proof of ownership and privacy protection for secure deduplication, then the advantages and common prob-lems of various key technologies and methods were summed up. Finally, the future research directions and development trends on secure data deduplication in cloud was given.     

云存储中密文数据的客户端安全去重方案

云存储环境下,客户端数据去重能在本地进行文件重复性检测,有效地节约存储空间和网络带宽.然而,客户端去重仍面临着很多安全挑战.首先,由于将文件哈希值作为重复性检测的证据,攻击者很可能通过一个文件的哈希值获得整个文件;其次,为了保护数据隐私,收敛加密被广泛运用于数据去重方案,但是由于数据本身是可预测的,所以收敛加密仍不可避免地遭受暴力字典攻击.为了解决上述问题,本文首次利用盲签名构造了一个安全的密钥生成协议,通过引入一个密钥服务器,实现了对收敛密钥的二次加密,有效地预防了暴力字典攻击;并进一步提出了一个基于块密钥签名的拥有权证明方法,能够有效预防攻击者通过单一的哈希值来获取文件,并能同时实现对密文文件的文件级和块级去重.同时,安全分析表明本文方案在随机预言模型下是可证明安全的,并能够满足收敛密钥安全、标签一致性和抗暴力字典攻击等更多安全属性.此外,与现有方案相比,实验结果表明本文方案在文件上传和文件去重方面的计算开销相对较小.