一种面向HDFS中海量小文件的存取优化方法*

为了解决HDFS(Hadoop Distributed File System)在存储海量小文件时遇到的NameNode内存瓶颈等问题,提高HDFS处理海量小文件的效率,提出一种基于小文件合并与预取的存取优化方案。首先通过分析大量小文件历史访问日志,得到小文件之间的关联关系,然后根据文件相关性将相关联的小文件合并成大文件后再存储到HDFS。从HDFS中读取数据时,根据文件之间的相关性,对接下来用户最有可能访问的文件进行预取,减少了客户端对NameNode节点的访问次数,提高文件命中率和处理速度。实验结果证明,该方法有效提升了Hadoop对小文件的存取效率,降低了NameNode节点的内存占用率。     

面向Hadoop分布式文件系统的小文件存取优化方法

为提高Hadoop分布式文件系统(HDFS)的小文件处理效率,提出了一种面向HDFS的智能小文件存取优化方法--SmartFS。SmartFS通过分析小文件访问日志,获取用户访问行为,建立文件关联概率模型,并根据基于文件关联关系的合并算法将小文件组装成大文件之后存至HDFS;当从HDFS获取文件时,根据基于文件关联关系的预取算法来提高文件访问效率,并提出基于预取的缓存替换算法来管理缓存空间,从而提高文件的命中率。实验结果表明,SmartFS有效减少了HDFS中NameNode的元数据空间,减少了用户与HDFS的交互次数,提高了小文件的存储效率和访问速度。     

HDFS中高效存储小文件的方法

为改善应用Hadoop分布式文件系统存储大量小文件时效率低下的问题,将NameNode职责分离,使用单独的NFS服务器同步存储元数据信息,以降低Client数据请求压力,提供大吞吐量数据访问并改善访问延迟;设计文件与数据块的对应模式,允许在同一块中存储多个小文件,并对系统加以实现,为海量小文件的存储提供了一个有效的解决方案。实验结果表明,该机制可以在数据迅速增长的背景下实现海量小文件的高效存取。     

基于小文件的内存云存储优化策略

由于内存云RAMCloud采用日志段的方式存储数据,因此当大量小文件存储于RAMCloud集群时,每个小文件独占整个段,会产生较多的段内碎片,从而导致内存的有效利用率较低以及大量的内存空间浪费。为了解决这个问题,提出基于文件分类的RAMCloud小文件存储优化策略。该策略首先根据文件的相关特性将小文件分为结构相关文件、逻辑相关文件以及相互独立文件三类;然后在存储时对结构相关的文件使用文件合并算法,逻辑相关和相互独立的小文件则使用分组算法。实验结果表明:同未进行优化的RAMCloud存储策略相比,该策略能有效提高集群内存利用率。     

基于云环境下一种小文件传输策略研究

基于互联网的文件传输策略将直接影响到文件传输效率和网络资源的使用效率,这一情况在各种资源集中的云计算环境表现得更为明显,特别是当传输大量小文件时尤为突出。针对这种现状,提出了一种基于打包策略的文件传输策略,给出了文件包优化处理机制,基于该策略,文件传输之前对小文件进行打包处理,然后进行传输,在接收端收到文件时首先进行解包再进行存储,从而减小传输过程中大量的IO操作,提升文件传输效率。大量的实验表明了本文所提策略的正确性,能够极大地提升云环境下小文件的传输效率,提高云资源的利用率。     

一种区块链数据的云存储与共享方法

为了在不信任的环境中达成共识,区块链节点需要冗余地存储完整的区块链数据,对节点的存储要求很高.提出一种区块链数据的存储方法,设置M个云存储共享模块用于存储完整的区块链以及节点数字签名,节点将区块及其数字签名存储到若干个云存储共享模块.当区块链节点数量N足够大时,能够选择合适的M(M<相似文献   

An optimization approach for cloud composite services

Recently, a considerable literature has grown up around the theme of composite services verification. Namely, the verification of the non-functional aspect generally consisting of optimizing the quality of service (QoS) of the composite service. Great efforts have been devoted to the study of several optimization methods and their impact on the QoS of the composite service. Guaranteeing the service level agreements established with users remains one of the greatest challenges in this field. This essay explores a new composition approach based on a linear programming algorithm and compares the obtained results with existing works. Our approach aims to guarantee an efficient and optimal solution to the Cloud composite service problem. For evaluation, we have developed the CR-SIM simulator that selects and composes services in the Cloud context.

     

Zero knowledge based client side deduplication for encrypted files of secure cloud storage in smart cities

As typical applications in the field of the cloud computing, cloud storage services are popular in the development of smart cities for their low costs and huge storage capacity. Proofs-of-ownership (PoW) is an important cryptographic primitive in cloud storage to ensure that a client holds the whole file rather than part of it in secure client side data deduplication. The previous PoW schemes worked well when the file is in plaintext. However, the privacy of the clients’ data may be vulnerable to honest-but-curious attacks. To deal with this issue, the clients tend to encrypt files before outsourcing them to the cloud, which makes the existing PoW schemes inapplicable any more. In this paper, we first propose a secure zero-knowledge based client side deduplication scheme over encrypted files. We prove that the proposed scheme is sound, complete and zero-knowledge. The scheme can achieve a high detection probability of the clients’ misbehavior. Then we introduced a proxy re-encryption based key distribution scheme. This scheme ensures that the server knows nothing about the encryption key even though it acts as a proxy to help distributing the file encryption key. It also enables the clients who have gained the ownership of a file to share the file with the encryption key generated without establishing secure channels among them. It is proved that the clients’ private key cannot be recovered by the server or clients collusion attacks during the key distribution phase. Our performance evaluation shows that the proposed scheme is much more efficient than the existing client side deduplication schemes.     

面向云存储的非结构化数据存储研究

云存储是网格、并行和分布式计算等众多技术发展和延伸,云存储实现了存储的完全虚拟化,提供更强大的存储及共享功能[1].非机构化数据包括文本、图像、音频、视频、PDF、电子表格等.非结构化数据的存储通常有两种方式,一种是使用文件系统以文件的方式存储,将文件的路径或者链接存储在关系型数据库表中;另一种是将这些数据存储在传统的数据库表的大对象字段中.文章主要研究非结构化数据的存储方式,结合非结构化数据的特点,云存储的优势以及MongoDB的数据存储特性,提出非结构化数据云存储的必要性.     

Proxy re-encryption architect for storing and sharing of cloud contents

Internet-based online cloud services provide enormous volumes of storage space, tailor-made computing resources and eradicate the obligation of native machines for data maintenance as well. Cloud storage service providers claim to offer the ability of secure and elastic data-storage services that can adapt to various storage necessities. Most of the security tools have a finite rate of failure, and intrusion comes with more complex and sophisticated techniques; the security failure rates are skyrocketing. Once we upload our data into the cloud, we lose control of our data, which certainly carries new security hazards toward integrity and privacy of our information. In this paper, we discuss a secure file sharing mechanism for the cloud with proxy re-encryption (PRE). PRE-scheme is implemented with the Disintegration Protocol to secure storage data in storage and in the flight. The paper introduces a new contribution of a seamless file sharing technique among different clouds without sharing an encryption key.     

An IBE-based design for assured deletion in cloud storage

Assured deletion of outsourced data in cloud storage is one of the most important issues in cloud storage. The use of encryption is a promising approach to address this issue. In this paper, we propose a design to achieve assured deletion in cloud storage built upon identity based cryptography (IBE). Compared with other designs known in the literature, foremost among them being FADE, the proposed design comes with a lightweight key management infrastructure. Furthermore, it offers other key management benefits inherent in identity-based cryptography such as key delegation and revocation. To demonstrate the practicality of the proposed design, we implement a fully featured prototype and evaluate its performance. Results analysis shows that the overhead time cost of the proposed design does not exceed 3 percent of the plaintext design. This rate decreases significantly for files with larger sizes, which makes it a relevant choice for use in real applications.     

高机密性数据安全存储与访问系统设计

研究并设计了一套安全模型的原型,主要用于为高机密性的数据提供安全保护.数据保护与访问方案可以分为静态保护与动态保护两种,本原型系统在加强静态保护的同时更加注重数据访问时的动态安全性.基于此提出了密码保险箱结构,用于保护运行时的机密数据,并且设计了一系列安全措施用以保障程序本身与用户数据的安全性与完整性,防御重放攻击,替换攻击等诸多攻击手段.对该原型系统的安全性进行了理论分析,并通过实际测试分析表明了系统的安全性与有效性.     

An optimized approach to histogram computation on GPU

A histogram is a compact representation of the distribution of data in an image with a full range of applications in diverse fields. Histogram generation is an inherently sequential operation where every pixel votes in a reduced set of bins. This makes finding efficient parallel implementations very desirable but challenging, because on graphics processing units thousands of threads may be atomically updating a short number of histogram bins. Under these circumstances, collisions among threads will be very frequent and such collisions will serialize thread execution, seriously damaging the performance. In this paper we propose a highly optimized approach to histogram calculation, which tackles such performance bottlenecks. It uses histogram replication for eliminating position conflicts, padding to reduce bank conflicts, and an improved access to input data called interleaved read access. Our so-called ${\mathcal{R}}$ -per-block approach to histogram calculation has been successfully compared to the main state-of-the-art works using four histogram-based image processing kernels and two real image databases. Results show that our proposal is between 1.4 and 15.7 faster than every previous implementation for histograms of up to 4,096 bins.     

An approach for detecting LDoS attack based on cloud model

Cybersecurity has always been the focus of Internet research. An LDoS attack is an intelligent type of DoS attack, which reduces the quality of network service by periodically sending high-speed but short-pulse attack traffic. Because of its concealment and low average rate, the traditional DoS attack detection methods are challenging to be effective. The existing LDoS attack detection methods generally have the problems of high FPR and FNR. A cloud model-based LDoS attack detection method is proposed, and a classifier based on SVM is used to train and classify the feature parameters. The detection method is verified and tested in the NS2 simulation platform and Test-bed network environment. Compared with the existing research results, the proposed method requires fewer samples, and it has lower FPR and FNR.     

An SSD-based accelerator for directory parsing in storage systems containing massive files

Data explosion introduces new challenges to storage systems. In a file system for big data, a large number of directories and files exist, which are usually organized in a large tree. Parsing directories in a large tree is difficult. In this paper, we propose an accelerator, which helps file systems to fetch the metadata of files rapidly. Contributions of this work include two aspects. First, we propose an accelerator for directory parsing. The accelerator is actually an SSD-based (Solid State Drive-based) cache, which keeps the metadata of frequently or recently accessed files and directories. When a file is demanded, the accelerator attempts to obtain its metadata directly from SSD. If the metadata is kept in SSD, the file system can rapidly obtain the metadata. However, if the metadata is not in SSD, the accelerator consumes a long time to access SSD, but to no avail. In order to avoid non-beneficial SSD accesses, the accelerator predicts whether the metadata is kept by SSD before issuing a read request. Only if the metadata has a high probability of being kept in SSD, the accelerator issues a request to the SSD. The second contribution of this paper is a new bloom filter used to predict whether a piece of data is kept in SSD. Bloom filter is a space-efficient data structure supporting membership query. But, the standard bloom filter cannot support element deletion. Whereas, our accelerator is a cache, which evicts items periodically. The standard bloom filter is not suitable for our accelerator. In this work, we designed a new bloom filter with low overhead, which supports element deletion. The new bloom filter perfectly suits the proposed accelerator. With the prediction of our bloom filter, the accelerator can accelerate the process of directory parsing with nearly no negative impact. We evaluated the accelerator by using a prototype. Experimental results demonstrate that, the accelerator can speed up the directory parsing process by nearly four times compared with a file system without an accelerator.     

An efficient and secure approach for a cloud collaborative editing

The demand for cloud-based collaborative editing service is rising along with the tremendously increased popularity in cloud computing. In the cloud-based collaborative editing environment, the data are stored in the cloud and able to be accessed from everywhere through every compatible device with the Internet. The information is shared with every accredited user in a group. In other words, multiple authorized users of the group are able to work on the same document and edit the document collaboratively and synchronously online. Meanwhile, during the whole collaborative editing process, the encryption technique is eventually applied to protect and secure the data. The encryption for the collaborative editing, however, could require much time to operate. To elevate the efficiency of the encryption, this study first analyzes the text editing in the collaborative service and presents a framework of the Red–Black tree, named as rbTree-Doc. The rbTree-Doc can reduce the amount of data to be encrypted. Although the trade-off for creating the Red–Black tree introduces extra cost, the experimental results of using rbTree-Doc in text editing operations, such as insertion and removal, show improved efficiency compared with other whole-document encryption strategy. Using rbTree-Doc, the efficiency is improved by 31.04% compared to that 3DES encryption is applied and by 23.94% compared to that AES encryption is applied.     

An approach to accessing product data across system and software revisions

Long-term users of engineering product data are hampered by the ephemeral nature of CAD file formats and the applications that work with them. STEP, the Standard for the Exchange of Product Model Data (ISO 10303), promises to help with meeting this challenge, but is not without problems of its own. We present a complementary solution based on the use of lightweight file formats to preserve specific aspects of the product data, in conjunction with a registry of relevant representation information as defined by the Open Archival Information System Reference Model (ISO 14721). This registry is used to identify suitable destination file formats for different purposes, and provides a resource to aid in the recovery of information from these formats in the future.     

基于NoSQL的海量航空物流小文件分布式多级存储方法

为了解决航空物流领域海量小文件存储效率和访问效率不高的问题,提出一种基于Nosql的海量小文件分布式多级存储方法,充分考虑到数据的时效性、本地性、操作的并发性以及文件之间的相关性,先根据相关性将文件合并,然后采用分布式多级存储,使用内存式Redis数据库做缓存,HDFS做数据的持久化存储,其过程采用预取机制。实验结果表明,该方法有效提高了小文件的存取效率和磁盘的利用率,显著地降低了网络的带宽占用和集群NameNode的内存消耗,适合解决航空领域海量小文件存储问题。     

Dynamic searchable symmetric encryption for storing geospatial data in the cloud

International Journal of Information Security - We present a dynamic searchable symmetric encryption scheme allowing users to securely store geospatial data in the cloud. Geospatial data sets often...