On the detection of custom memory allocators in C binaries

Many reverse engineering techniques for data structures rely on the knowledge of memory allocation routines. Typically, they interpose on the system’s malloc and free functions, and track each chunk of memory thus allocated as a data structure. However, many performance-critical applications implement their own custom memory allocators. Examples include webservers, database management systems, and compilers like gcc and clang. As a result, current binary analysis techniques for tracking data structures fail on such binaries. We present MemBrush, a new tool to detect memory allocation and deallocation functions in stripped binaries with high accuracy. We evaluated the technique on a large number of real world applications that use custom memory allocators. We demonstrate that MemBrush can detect allocators/deallocators with a high accuracy which is 52 out of 59 for allocators, and 29 out of 31 for deallocators in SPECINT 2006. As we show, we can furnish existing reverse engineering tools with detailed information about the memory management API, and as a result perform an analysis of the actual application specific data structures designed by the programmer. Our system uses dynamic analysis and detects memory allocation and deallocation routines by searching for functions that comply with a set of generic characteristics of allocators and deallocators.     

Verification of complex dynamic data tree with mu-calculus

The problem of verifying software systems that use dynamic data structures (such as linked lists, queues, or binary trees) has attracted increasing interest over the last decade. Dynamic structures are not easily supported by verification techniques because, among other reasons, it is difficult to efficiently manage the pointer-based internal representation. This is a key aspect when, for instance, the goal is to construct a verification tool based on model checking techniques. In addition, since new nodes can be dynamically inserted or extracted from the structure, the shape of the dynamic data (and other more specific properties) may vary at runtime, with errors such as the non desirable sharing between two nodes being difficult to detect. In this paper, we propose to use mu-calculus to describe and analyze with model checking techniques dynamic data structures such as lists and trees. The expressiveness of mu-calculus makes it possible to naturally describe these structures. In addition, following the ideas of separation logic, the logic has been extended with a new operator capable of describing the non-sharing property, which is essential when analyzing dynamic data structures.     

试谈C语言程序设计教学

C语言凭借其丰富的数据结构、优良的机构性灵活性为电脑编程工作的人推崇学习。C语言作为一门基础课程,在高等院校计算机专业和很多的非计算机专业中开设,具有十分重要的作用和意义,它不仅能锻炼学生的思维逻辑能力,还为很多课程,如数据结构、单片机和其他程序设计语言打下良好的基础,具有引导性的作用。根据教学过程中学生常出现的问题,总结出了在C语言教学过程中的几点教学方法。     

随机森林的可解释性可视分析方法研究

由于随机森林算法在很多情况下都以“黑盒”的方式存在,对于用户而言,参数调整,训练甚至最终构建的模型细节是隐蔽的,这导致了随机森林模型的可解释性非常差,在一定程度上阻碍了该模型在一些诸如医学诊断、司法、安全领域等需要透明化和可解释需求比较高的领域使用。影响该模型可解释性挑战主要来源于特征选择和数据的随机性。同时随机森林包含许多决策树,用户很难理解和比较所有决策树的结构和属性。为了解决上述问题,设计并实现了可视分析系统FORESTVis,该系统包括树视图、部分依赖图、t-SNE投影图、特征视图等多个交互式可视化组件,借助该系统,相关研究人员和从业人员可以直观地了解随机森林的基本结构和工作机制,并协助用户对模型的性能进行评估。使用Kaggle公开数据集上进行案例分析,验证了该方法的可行性和有效性。     

A binary decision diagram based approach for mining frequent subsequences

Sequential pattern mining is an important problem in data mining. State of the art techniques for mining sequential patterns, such as frequent subsequences, are often based on the pattern-growth approach, which recursively projects conditional databases. Explicitly creating database projections is thought to be a major computational bottleneck, but we will show in this paper that it can be beneficial when the appropriate data structure is used. Our technique uses a canonical directed acyclic graph as the sequence database representation, which can be represented as a binary decision diagram (BDD). In this paper, we introduce a new type of BDD, namely a sequence BDD (SeqBDD), and show how it can be used for efficiently mining frequent subsequences. A novel feature of the SeqBDD is its ability to share results between similar intermediate computations and avoid redundant computation. We perform an experimental study to compare the SeqBDD technique with existing pattern growth techniques, that are based on other data structures such as prefix trees. Our results show that a SeqBDD can be half as large as a prefix tree, especially when many similar sequences exist. In terms of mining time, it can be substantially more efficient when the support is low, the number of patterns is large, or the input sequences are long and highly similar.     

Software Caching and Computation Migration in Olden

Software caching and computation migration are mechanisms that satisfy remote references by either bringing a copy of the data to the computation or moving the computation to the data. We evaluate these mechanisms usingOlden, a system that, with minimal programmer annotations, provides parallelism for C programs that use recursively defined structures, such as trees, lists, and DAGs. We demonstrate that providing both software caching and computation migration can improve the performance of these programs, and provide a compile-time heuristic that selects between them for each pointer dereference. We have implemented the heuristic in Olden on the Thinking Machines CM-5. We describe our implementation and report on experiments with eleven benchmarks.     

基于JavaCC的抽象语法树生成错误处理技术研究

随着C++语言标准的不断演进,词法语法解析工具如JavaCC等对于很多扩充的新特性以及复杂的语法结构不能做到完全支持,这可能会导致抽象语法树生成错误且不完整;针对这一问题,提出一个针对抽象语法树生成错误的处理框架;首先,通过对JavaCC的扩充,实现一套可以解析C++语言的词法语法分析器,生成抽象语法树并记录报错行;其次,根据报错行寻找所在函数区间即不支持或不匹配的语法片段;最后,通过注释函数区间的方式来跳过不支持或不匹配的语法片段进行错误处理并迭代生成抽象语法树;实验结果表明,对抽象语法树生成进行错误处理后可以更全面的分析代码,抽象语法树完成率上升37.8%,分析行数提高3.9倍。     

A hybrid decision tree training method using data streams

Classical classification methods usually assume that pattern recognition models do not depend on the timing of the data. However, this assumption is not valid in cases where new data frequently become available. Such situations are common in practice, for example, spam filtering or fraud detection, where dependencies between feature values and class numbers are continually changing. Unfortunately, most classical machine learning methods (such as decision trees) do not take into consideration the possibility of the model changing, as a result of so-called concept drift and they cannot adapt to a new classification model. This paper focuses on the problem of concept drift, which is a very important issue, especially in data mining methods that use complex structures (such as decision trees) for making decisions. We propose an algorithm that is able to co-train decision trees using a modified NGE (Nested Generalized Exemplar) algorithm. The potential for adaptation of the proposed algorithm and the quality thereof are evaluated through computer experiments, carried out on benchmark datasets from the UCI Machine Learning Repository.     

Mining closed and maximal frequent subtrees from databases of labeled rooted trees

Tree structures are used extensively in domains such as computational biology, pattern recognition, XML databases, computer networks, and so on. One important problem in mining databases of trees is to find frequently occurring subtrees. Because of the combinatorial explosion, the number of frequent subtrees usually grows exponentially with the size of frequent subtrees and, therefore, mining all frequent subtrees becomes infeasible for large tree sizes. We present CMTreeMiner, a computationally efficient algorithm that discovers only closed and maximal frequent subtrees in a database of labeled rooted trees, where the rooted trees can be either ordered or unordered. The algorithm mines both closed and maximal frequent subtrees by traversing an enumeration tree that systematically enumerates all frequent subtrees. Several techniques are proposed to prune the branches of the enumeration tree that do not correspond to closed or maximal frequent subtrees. Heuristic techniques are used to arrange the order of computation so that relatively expensive computation is avoided as much as possible. We study the performance of our algorithm through extensive experiments, using both synthetic data and data sets from real applications. The experimental results show that our algorithm is very efficient in reducing the search space and quickly discovers all closed and maximal frequent subtrees.     

Regular Languages of Thin Trees

An infinite tree is called thin if it contains only countably many infinite branches. Thin trees can be seen as intermediate structures between infinite words and infinite trees. In this work we investigate properties of regular languages of thin trees. Our main tool is an algebra suitable for thin trees. Using this framework we characterize various classes of regular languages: commutative, open in the standard topology, and definable in weak MSO logic among all trees. We also show that in various meanings thin trees are not as rich as all infinite trees. In particular we observe a collapse of the parity index to the level (1, 3) and a collapse of the topological complexity to co-analytic sets. Moreover, a gap property is shown: a regular language of thin trees is either weak MSO-definable among all trees or co-analytic-complete.     

Extending metric index structures for efficient range query processing

Databases are getting more and more important for storing complex objects from scientific, engineering, or multimedia applications. Examples for such data are chemical compounds, CAD drawings, or XML data. The efficient search for similar objects in such databases is a key feature. However, the general problem of many similarity measures for complex objects is their computational complexity, which makes them unusable for large databases. In this paper, we combine and extend the two techniques of metric index structures and multi-step query processing to improve the performance of range query processing. The efficiency of our methods is demonstrated in extensive experiments on real-world data including graphs, trees, and vector sets.     

A Shape Graph Logic and A Shape System

Analysis and verification of pointer programs are still difficult problems so far. This paper uses a shape graph logic and a shape system to solve these problems in two stages. First, shape graphs at every program point are constructed using an analysis tool. Then, they are used to support the verification of other properties （e.g., orderedness）. Our prototype supports automatic verification of programs manipulating complex data structures such as splay trees, treaps, AVL trees and AA trees, etc. The proposed shape graph logic, as an extension to Hoare logic, uses shape graphs directly as assertions. It can be used in the analysis and verification of programs manipulating mutable data structures. The benefit using shape graphs as assertions is that it is convenient for acquiring the relations between pointers in the verification stage. The proposed shape system requires programmers to provide lightweight shape declarations in recursive structure type declarations. It can help rule out programs that construct shapes deviating from what programmers expect （reflected in shape declarations） in the analysis stage. As a benefit, programmers need not provide specifications （e.g., pre-/post-conditions, loop invariants） about pointers. Moreover, we present a method doing verification in the second stage using traditional Hoare logic rules directly by eliminating aliasing with the aid of shape graphs. Thus, verification conditions could be discharged by general theorem provers.     

New Shape Analysis and Interprocedural Techniques for Automatic Parallelization of C Codes

Automatic parallelization of codes with complex data structures is becoming very important. These complex, and often recursive, data structures are widely used in scientific computing. Shape analysis is one of the key steps in the automatic parallelization of such codes. In this paper we extend the Static Shape Graph (SSG) method to enable the successful and accurate detection of complex doubly-linked structures. We have also included an interprocedural analysis in our framework. These techniques have been implemented in a compiler, which has been validated for several C codes. In particular, we present the results the compiler achieves for the C sparse LU factorization algorithm and for a recursive code generating a tree data structure. The output SSG perfectly describes the complex data structure used in these codes.     

Programs to process trees,representing program structures and data structures

Utility programs discussed in this paper were initially developed to automate the ‘drawing’ of the non-binary trees which occur when using a Jackson type methodology for program design; these trees represent either data structures or program structures, showing the selective, sequential and iterative relationships between nodes First a language was developed to allow information about tree structures to be input to the programs. Techniques and algorithms are described which enabled utility programs to be coded to process this data; transforming it into printable diagrammatic representation of trees; pseudo-code representation of program trees; and PL/I or Cobol programs Secondly a merge operation for trees had to be defined, reflecting the step in Warmer's and Jackson's methodologies of combining data tree structures together to provide a program structure. Further algorithms are described which enable this step to be performed automatically, they include checks that each particular merge can be performed Because the aims and concepts have been fluid some of the ideas have been discarded, and programs have been rewritten. Valuable insights were gained from persuing both the rejected and the acceptable ideas, these are described in this paper.     

A Generic Implementation of Tree Skeletons

In data-parallel skeleton libraries, the implementation of skeletons is usually tightly-coupled with that of data structures. However, loose coupling between them like C++ STL will improve modularity and flexibility of skeletons and data structures. This flexibility is particularly valuable for tree skeletons. To achieve such loose coupling, we present an iterator-based interface of trees for tree skeletons. We have implemented tree skeletons on the basis of our interface; we present their design and implementation. This paper also reports the results of preliminary experiments.     

一种基于变精度粗糙集的C4.5决策树改进算法

针对C4.5决策树构造复杂、分类精度不高等问题,提出了一种基于变精度粗糙集的决策树构造改进算法.该算法采用近似分类质量作为节点选择属性的启发函数,与信息增益率相比,该标准更能准确地刻画属性分类的综合贡献能力,同时对噪声有一定的抑制能力.此外还针对两个或两个以上属性的近似分类质量相等的特殊情形,给出了如何选择最优的分类属...     

空间剖分树形查找结构的效率分析

空间剖分是构造快速空间查找数据结构的有效方法,四叉树、八叉树、Kd-树是典型的基于空间剖分思想的树形空间查找结构。选择合适的参数来构造实际点集数据的树形查找结构,对提高相关算法的效率具有重要意义。在分析三种树形查找结构基本原理的基础上,通过构造具有不同空间分布特征的实验数据,设置不同的树形空间剖分结构参数,来分析三种结构支持下搜索算法的时间消耗,确定使查找效率达到最优的树形结构构造参数。相关研究结论对于优化空间剖分树形查找结构的效率、提高相关算法的性能等,有一定的参考价值。     

Creating Decision Trees from Rules using RBDT‐1

Most of the methods that generate decision trees for a specific problem use the examples of data instances in the decision tree–generation process. This article proposes a method called RBDT‐1—rule‐based decision tree—for learning a decision tree from a set of decision rules that cover the data instances rather than from the data instances themselves. The goal is to create on demand a short and accurate decision tree from a stable or dynamically changing set of rules. The rules could be generated by an expert, by an inductive rule learning program that induces decision rules from the examples of decision instances such as AQ‐type rule induction programs, or extracted from a tree generated by another method, such as the ID3 or C4.5. In terms of tree complexity (number of nodes and leaves in the decision tree), RBDT‐1 compares favorably with AQDT‐1 and AQDT‐2, which are methods that create decision trees from rules. RBDT‐1 also compares favorably with ID3 while it is as effective as C4.5 where both (ID3 and C4.5) are well‐known methods that generate decision trees from data examples. Experiments show that the classification accuracies of the decision trees produced by all methods under comparison are indistinguishable.