| 大数据下基于多CPU的两级指纹流水计算去重方法 |
| |
| 引用本文： | 贺建英,袁小艳,唐青松.大数据下基于多CPU的两级指纹流水计算去重方法[J].计算机系统应用,2015,24(8):206-211. |
| |
| 作者姓名： | 贺建英 袁小艳 唐青松 |
| |
| 作者单位： | 四川文理学院 计算机学院, 达州 635000;四川文理学院 计算机学院, 达州 635000;四川文理学院 计算机学院, 达州 635000 |
| |
| 基金项目： | 国家档案局项目(2014-X-65);四川省教育厅一般项目(14ZB0313) |
| |
| 摘 要： | 分析数据去重的重要意义, 根据现有的数据去重技术和算法, 改进MD5码指纹的计算算法并进行优化, 分析并重组指纹计算的流水化方法, 利用缓存组代替单个缓存的方式, 提出一种基于多CPU的两级指纹流水计算方法, 对该方法进行分析研究, 并通过相关试验和试验数据来支持该方法的有效性.
|
| 关 键 词： | 数据去重 多CPU 两级指纹 算法 缓存组 |
| 收稿时间： | 2015/1/28 0:00:00 |
| 修稿时间： | 2015/3/18 0:00:00 |
Duplicate Removal Method of Large Data under Two Level Fingerprins Flow Based on Multi CPU Calculation |
| |
| HE Jian-Ying,YUAN Xiao-Yan and TANG Qing-Song.Duplicate Removal Method of Large Data under Two Level Fingerprins Flow Based on Multi CPU Calculation[J].Computer Systems& Applications,2015,24(8):206-211. |
| |
| Authors: | HE Jian-Ying YUAN Xiao-Yan and TANG Qing-Song |
| |
| Affiliation: | College of Computer, Sichuan University of Arts and Science, Dazhou 635000, China;College of Computer, Sichuan University of Arts and Science, Dazhou 635000, China;College of Computer, Sichuan University of Arts and Science, Dazhou 635000, China |
| |
| Abstract: | This paper analyzes the importance of data removal. According to the existing data of the removal techniques and algorithms, it improves MD5 code fingerprint algorithm to calculate and optimize it, analyzes and recombines the fingerprint calculation of water level. Using the cache group to replace the single cache, we propose a new method to calculate the two water level fingerprints based on multi CPU to study and analyze the method. At the same time, it supports effectiveness of the method through the relevant tests and test data. |
| |
| Keywords: | data duplicate removal multi CPU the two level of fingerprint algorithm cache group |
| 本文献已被 万方数据 等数据库收录！ |
| 点击此处可从《计算机系统应用》浏览原始摘要信息 |
|
点击此处可从《计算机系统应用》下载全文 |
|
