LZW算法优化及在雷达数据压缩中的应用

LZW算法是一种性能优异的字典压缩算法,具有通用性强、字典在编解码过程中动态形成等优点,在无损压缩领域应用广泛。介绍了其算法原理,给出了程序实现的编码步骤,并选取一个实例进行详细分析。设计了一种哈希表对程序进行优化,显著降低检索字典时间,分别选取图片、雷达数据、文本文件进行编码速度对比,获得了较好的效果。最后,使用不同的数据分段选取若干典型的真实雷达数据进行试验,并与游程编码进行了对比,得出若干结论。     

数字音频压缩编码及在软件中播放音频文件

模拟音频信号经过数字化后,会产生很大的数据量,不适合存储和传输的需要。数字音频的压缩编码已经成为一门非常重要的技术。该文主要介绍数字音频信号的常用压缩编码方法,如MPEG-1、MPEG-2、AC-3数字音频压缩编码标准,并阐述了各自的性能特点及应用领域。然后结合应用软件——Delphi,介绍制作音频播放器的过程,并利用音频播放器实现mp3等格式音频文件的播放。     

Stochastic automata-based estimators for adaptively compressing files with nonstationary distributions.

This correspondence shows that learning automata techniques, which have been useful in developing weak estimators, can be applied to data compression applications in which the data distributions are nonstationary. The adaptive coding scheme utilizes stochastic learning-based weak estimation techniques to adaptively update the probabilities of the source symbols, and this is done without resorting to either maximum likelihood, Bayesian, or sliding-window methods. The authors have incorporated the estimator in the adaptive Fano coding scheme and in an adaptive entropy-based scheme that "resembles" the well-known arithmetic coding. The empirical results obtained for both of these adaptive methods are obtained on real-life files that possess a fair degree of nonstationarity. From these results, it can be seen that the proposed schemes compress nearly 10% more than their respective adaptive methods that use maximum-likelihood estimator-based estimates.     

中文文本压缩的LZSSCH算法

本文结合中文的特点,从建模编码、自适应索引扩位和最大索引位长等方面对LZSS 算法进行了修改,得到的LZSSCH 算法对以中文为主的中西文混合文本文件的压缩比平均与LZSS 算法高出约8 % ,而其压缩和扩展速度以及可执行程序的大小均与LZSS算法相当。算法无须任何预处理,还可用于压缩其它非拼音文字文本文件。     

Sequential neural text compression

The purpose of this paper is to show that neural networks may be promising tools for data compression without loss of information. We combine predictive neural nets and statistical coding techniques to compress text files. We apply our methods to certain short newspaper articles and obtain compression ratios exceeding those of the widely used Lempel-Ziv algorithms (which build the basis of the UNIX functions "compress" and "gzip"). The main disadvantage of our methods is that they are about three orders of magnitude slower than standard methods.     

Evolution of human-competitive lossless compression algorithms with GP-zip2

We propose GP-zip2, a new approach to lossless data compression based on Genetic Programming (GP). GP is used to optimally combine well-known lossless compression algorithms to maximise data compression. GP-zip2 evolves programs with multiple components. One component analyses statistical features extracted by sequentially scanning the data to be compressed and divides the data into blocks. These blocks are projected onto a two-dimensional Euclidean space via two further (evolved) program components. K-means clustering is then applied to group similar data blocks. Each cluster is labelled with the optimal compression algorithm for its member blocks. After evolution, evolved programs can be used to compress unseen data. The compression algorithms available to GP-zip2 are: Arithmetic coding, Lempel-Ziv-Welch, Unbounded Prediction by Partial Matching, Run Length Encoding, and Bzip2. Experimentation shows that the results produced by GP-zip2 are human-competitive, being typically superior to well-established human-designed compression algorithms in terms of the compression ratios achieved in heterogeneous archive files.     

Efficient method for lossless LIDAR data compression

Light Detection and Ranging (LIDAR) has become one of the prime technologies for rapid collection of vast spatial data, usually stored in a LAS file format (LIDAR data exchange format standard). In this article, a new method for lossless LIDAR LAS file compression is presented. The method applies three consequent steps: a predictive coding, a variable-length coding and an arithmetic coding. The key to the method is the prediction schema, where four different predictors are used: three predictors for x, y and z coordinates and a predictor for scalar values, associated with each LIDAR point. The method has been compared with the popular general-purpose methods and with a method developed specially for compressing LAS files. The proposed method turns out to be the most efficient in all test cases. On average, the LAS file is losslessly compressed to 12% of its original size.     

Compression techniques for fast external sorting

External sorting of large files of records involves use of disk space to store temporary files, processing time for sorting, and transfer time between CPU, cache, memory, and disk. Compression can reduce disk and transfer costs, and, in the case of external sorts, cut merge costs by reducing the number of runs. It is therefore plausible that overall costs of external sorting could be reduced through use of compression. In this paper, we propose new compression techniques for data consisting of sets of records. The best of these techniques, based on building a trie of variable-length common strings, provides fast compression and decompression and allows random access to individual records. We show experimentally that our trie-based compression leads to significant reduction in sorting costs; that is, it is faster to compress the data, sort it, and then decompress it than to sort the uncompressed data. While the degree of compression is not quite as great as can be obtained with adaptive techniques such as Lempel-Ziv methods, these cannot be applied to sorting. Our experiments show that, in comparison to approaches such as Huffman coding of fixed-length substrings, our novel trie-based method is faster and provides greater size reductions. Preliminary versions of parts of this paper, not including the work on vargram compression” [41]     

小波图像压缩进展

小波分析由于具有多分辨分析特性而在图像压缩中得到了广泛应用,产生了许多行之有效的方法和技术。文中综述了小波图像压缩方法及其进展,如零树编码、小波包图像压缩和多小波图像压缩等,并对这些方法的优缺点进行了总结和对比。实践证明,小波包、多小波的压缩性能都比传统的单小波优越。但是,小波包、多小波图像压缩方法还很不完善,具有自己的不足之处,需要对其进行改进和完善以提高运算速度,改进的方法之一就是和其它压缩方法(如分形)相结合。     

Web-based delivery of medical multimedia contents using an MPEG-4 system

Moving picture expert group compression standard version 4 (MPEG-4) is a standard for video coding aimed at multimedia applications. MPEG-4 was developed to enable high compression rate in a low bitrate transmission via the Internet or mobile telecommunications. Although these characteristics of MPEG-4 are suitable for telemedicine, little is known about the possibility of using this technology in the field of telemedicine. We evaluated the quality of MPEG-4-encoded medical video streams and compared them with original analogue videos and audio-video-interleave (AVI) files. Although MPEG-4 video streams have the advantage of small file size, they were found to be inferior to original videos and AVI files in terms of smoothness of motion pictures, sharpness of images and clearness of sound. Illegibility of characters was a major problem in MPEG-4 files. The score for total impression of MPEG-4 files was significantly lower than those for AVI files. The results of this study suggest that the quality of MPEG-4-encoded video streams is not adequate for telemedicine.     

A large‐alphabet‐oriented scheme for Chinese and English text compression

In this paper, a large‐alphabet‐oriented scheme is proposed for both Chinese and English text compression. Our scheme parses Chinese text with the alphabet defined by Big‐5 code, and parses English text with some rules designed here. Thus, the alphabet used for English is not a word alphabet. After a token is parsed out from the input text, zero‐, first‐, and second‐order Markov models are used to estimate the occurrence probabilities of this token. Then, the probabilities estimated are blended and accumulated in order to perform arithmetic coding. To implement arithmetic coding under a large alphabet and probability‐blending condition, a way to partition count‐value range is studied. Our scheme has been programmed and can be executed as a software package. Then, typical Chinese and English text files are compressed to study the influences of alphabet size and prediction order. On average, our compression scheme can reduce a text file's size to 33.9% for Chinese and to 23.3% for English text. These rates are comparable with or better than those obtained by popular data compression packages. Copyright © 2005 John Wiley & Sons, Ltd.     

面向同步辐射光源图像的可并行智能压缩方法

在建的高能同步辐射光源预计会产生海量原始数据,其中硬X射线实验线站产生的图像数据占比最高且具有高分辨率和高帧率的特点,亟需有效的无损压缩方法缓解存储和传输压力,然而现有通用无损压缩方法对该类图像压缩效果不佳,基于深度学习的无损压缩方法又耗时较长。结合同步辐射光源图像的特点,提出一种在保证图像压缩比前提下的可并行智能无损图像压缩方法。通过参数自适应的可逆分区量化方法,大幅缩小图像经过时间差分后的像素值分布范围,能够节省20%以上的存储空间。将以CNN为基础架构的时空学习网络C-Zip作为概率预测器,同时以数据集为单位过拟合训练模型进一步优化图像压缩比。针对压缩过程中耗时较长的算术编码过程,利用概率距离量化代替算术编码,结合深度学习进行无损编码,增加编码过程的并行度。实验结果表明,该方法的图像压缩比相比于PNG、FLIF等传统图像无损压缩方法提升了0.23~0.58,对于同步辐射光源图像具有更好的压缩效果。     

文本压缩的实现原理及方法

无论是在存储设备中存储文件,还是用因特网传输文件,经常会用到一些压缩算法来节省空间,提高传输速度。文中利用聚焦最有效的压缩算法Huffman编码,并用Java来实现这个过程。     

Tailored compression of Java class files

Java class files can be transmitted more efficiently over a network if they are compressed. After an examination of the class file structure and obtaining statistics from a large collection of class files, we propose a compression scheme that is tailored to class files. Our scheme achieves significantly better compression than commonly used methods such as ZIP. © 1998 John Wiley & Sons, Ltd.     

A practical design of high-volume steganography in digital video files

In this research, we consider exploiting the large volume of audio/video data streams in compressed video clips/files for effective steganography. By observing that most of the distributed video files employ H.264 Advanced Video Coding (AVC) and MPEG Advanced Audio Coding (AAC) for video/audio compression, we examine the coding features in these data streams to determine appropriate data for modification so that the reliable high-volume information hiding can be achieved. Such issues as the perceptual quality, compressed bit-stream length, payload of embedding, effectiveness of extraction and efficiency of execution will be taken into consideration. First, the effects of using different coding features are investigated separately and three embedding profiles, i.e. High, Medium and Low, which indicate the amount of payload, will then be presented. The High profile is used to embed the maximum amount of hidden information when the high payload is the only major concern in the target application. The Medium profile is recommended since it is designed to achieve a good balance among several requirements. The Low profile is an efficient implementation for faster information embedding. The performances of these three profiles are reported and the suggested Medium profile can hide more than 10% of the compressed video file size in common Flash Video (FLV) files.     

An investigation of byte n-gram features for malware classification

Malware classification using machine learning algorithms is a difficult task, in part due to the absence of strong natural features in raw executable binary files. Byte n-grams previously have been used as features, but little work has been done to explain their performance or to understand what concepts are actually being learned. In contrast to other work using n-gram features, in this work we use orders of magnitude more data, and we perform feature selection during model building using Elastic-Net regularized Logistic Regression. We compute a regularization path and analyze novel multi-byte identifiers. Through this process, we discover significant previously unreported issues with byte n-gram features that cause their benefits and practicality to be overestimated. Three primary issues emerged from our work. First, we discovered a flaw in how previous corpora were created that leads to an over-estimation of classification accuracy. Second, we discovered that most of the information contained in n-grams stem from string features that could be obtained in simpler ways. Finally, we demonstrate that n-gram features promote overfitting, even with linear models and extreme regularization.     

多模态医疗数据中海量小文件存储优化方法

Hadoop分布式文件系统(HDFS)通常用于大文件的存储和管理,当进行海量小文件的存储和计算时,会消耗大量的NameNode内存和访问时间,成为制约HDFS性能的一个重要因素.针对多模态医疗数据中海量小文件问题,提出一种基于双层哈希编码和HBase的海量小文件存储优化方法.在小文件合并时,使用可扩展哈希函数构建索引文件存储桶,使索引文件可以根据需要进行动态扩展,实现文件追加功能.在每个存储桶中,使用MWHC哈希函数存储每个文件索引信息在索引文件中的位置,当访问文件时,无须读取所有文件的索引信息,只需读取相应存储桶中的索引信息即可,从而能够在O(1)的时间复杂度内读取文件,提高文件查找效率.为了满足多模态医疗数据的存储需求,使用HBase存储文件索引信息,并设置标识列用于标识不同模态的医疗数据,便于对不同模态数据的存储管理,并提高文件的读取速度.为了进一步优化存储性能,建立了基于LRU的元数据预取机制,并采用LZ4压缩算法对合并文件进行压缩存储.通过对比文件存取性能、NameNode内存使用率,实验结果表明,所提出的算法与原始HDFS、HAR、MapFile、TypeStorage以及...     

An intelligent tutor for intrusion detection on computer systems

Intrusion detection is the process of identifying unauthorized usage of a computer system. It is an important skill for computer-system administrators. It is difficult to learn on the job because it is needed only occasionally but can be critical. We describe a tutor incorporating two programs. The first program uses artificial-intelligence planning methods to generate realistic audit files reporting actions of a variety of simulated users (including intruders) of a Unix computer system. The second program simulates the system afterwards, and asks the student to inspect the audit and fix the problems caused by the intruders. This program uses intrusion-recognition rules to itself infer the problems, planning methods to figure how best to fix them, plan-inference methods to track student actions, and tutoring rules to tutor intelligently. Experiments show that students using the tutor learn a significant amount in a short time.     

视频显示流压缩的技术与标准

随着计算机、电视机和手机等显示设备分辨率的迅速增加，采用显示流压缩（Display stream compression,DSC）技术解决显示链路带宽不足已成业界的共识。近年来陆续出现了JEPG-XS，H.264/AVC和Dirac（VC-2）的纯帧内编码、HEVC-SCC屏幕内容编码和VESA的DSC等显示链路压缩技术。其中最为典型的是视频电子标准协会(Video electronics standands association, VESA)的 DSC标准，这是一个被广泛接受的用于显示链路的低成本、短延时和视觉无损的轻量级编解码标准。DSC的高效编码技术包括先进的预测、历史彩色索引、简捷的熵编码以及良好的速率控制等。本文综述了DSC标准的新特性和主要技术概况。     

Optimizations of a GPU accelerated heat conduction equation by a programming of CUDA Fortran from an analysis of a PTX file

The Fortran language has been commonly used for many kinds of scientific computation. In this paper, we focus on the solution of an unsteady heat conduction equation, which is one of the simplest problems for thermal dynamics. Recently, a GPU (graphics processing unit) has been enhanced with a Fortran programming language capability employing CUDA (compute unified device architecture), known as CUDA Fortran. We find that the speed performance of a system using an ordinary program coding of CUDA Fortran is lower than that of systems using a program coding of CUDA C. We also find that intermediate assembly files PTX (parallel thread execution) of the two languages are not coincident. Therefore, by comparing the PTX files from the two coding programs we could detect the bottleneck that causes the speed reduction. We propose three optimization techniques that can enable the calculated speeds using CUDA Fortran and CUDA C to be coincident. The optimizations can be performed by the Fortran language when improved by an analyzed PTX file. It is thus possible to improve the performance of CUDA Fortran by adding a correction to it, which happens to be at a programming language level.