基于共享缓冲区的数据流处理框架设计与实现

为解决数据流处理应用程序依赖数据流传输,分布式环境部署困难,应用开发难度大等问题,设计了一种基于共享缓冲区的数据流处理框架。该框架利用共享缓冲区为应用提供数据代理接口,实现数据流推动和分布式处理,并引入可自定义的负载均衡,解决了应用对数据流的依赖度和开发部署的难度系数,使其专注数据分析处理,无需关心数据的地理位置,提高了应用系统的稳定性和扩展性。通过在互联网病毒监控系统中应用,验证了该框架在分布式环境下的实时性和灵活性,使其可移植到云计算环境中。     

1种基于遗失数据重构的软测量方法

提出1种遗失数据重构思想下的软测量方法:先采用主元分析(PCA)离线建立所有变量(包括难测变量)的主元模型,实际应用时,将实时的难测变量看作遗失数据,通过遗失数据重构方法估计出难测变量,增加了软测量方法的灵活性.更进一步,在重构遗失数据时,使用马氏距离取代欧几里德距离作为指标,更准确地反映了过程变量之间的相关关系,由此...     

Multimodal Earth observation data fusion: Graph-based approach in shared latent space

Multiple and heterogenous Earth observation (EO) platforms are broadly used for a wide array of applications, and the integration of these diverse modalities facilitates better extraction of information than using them individually. The detection capability of the multispectral unmanned aerial vehicle (UAV) and satellite imagery can be significantly improved by fusing with ground hyperspectral data. However, variability in spatial and spectral resolution can affect the efficiency of such dataset's fusion. In this study, to address the modality bias, the input data was projected to a shared latent space using cross-modal generative approaches or guided unsupervised transformation. The proposed adversarial networks and variational encoder-based strategies used bi-directional transformations to model the cross-domain correlation without using cross-domain correspondence. It may be noted that an interpolation-based convolution was adopted instead of the normal convolution for learning the features of the point spectral data (ground spectra). The proposed generative adversarial network-based approach employed dynamic time wrapping based layers along with a cyclic consistency constraint to use the minimal number of unlabeled samples, having cross-domain correlation, to compute a cross-modal generative latent space. The proposed variational encoder-based transformation also addressed the cross-modal resolution differences and limited availability of cross-domain samples by using a mixture of expert-based strategy, cross-domain constraints, and adversarial learning. In addition, the latent space was modelled to be composed of modality independent and modality dependent spaces, thereby further reducing the requirement of training samples and addressing the cross-modality biases. An unsupervised covariance guided transformation was also proposed to transform the labelled samples without using cross-domain correlation prior. The proposed latent space transformation approaches resolved the requirement of cross-domain samples which has been a critical issue with the fusion of multi-modal Earth observation data. This study also proposed a latent graph generation and graph convolutional approach to predict the labels resolving the domain discrepancy and cross-modality biases. Based on the experiments over different standard benchmark airborne datasets and real-world UAV datasets, the developed approaches outperformed the prominent hyperspectral panchromatic sharpening, image fusion, and domain adaptation approaches. By using specific constraints and regularizations, the network developed was less sensitive to network parameters, unlike in similar implementations. The proposed approach illustrated improved generalizability in comparison with the prominent existing approaches. In addition to the fusion-based classification of the multispectral and hyperspectral datasets, the proposed approach was extended to the classification of hyperspectral airborne datasets where the latent graph generation and convolution were employed to resolve the domain bias with a small number of training samples. Overall, the developed transformations and architectures will be useful for the semantic interpretation and analysis of multimodal data and are applicable to signal processing, manifold learning, video analysis, data mining, and time series analysis, to name a few.     

Parallelizing a GIS on a shared address space architecture

We are developing a high-performance GIS (our term for a parallel GIS) on an SGI Challenge, a 16-processor machine with a shared address space architecture (SASA). We describe how we parallelized a key GIS operation using a message-passing algorithm. We advocate the linking of two diverse approaches to the design of parallel architectures and algorithms. As part of our project, we evaluated the effect of parallelizing an important GIS operation: range query. We parallelized a range query using data partitioning (to reduce synchronization) and dynamic load balancing (to improve speedup). We found that these approaches do achieve the performance required for many GIS applications     

Towards a distributed multi-agent framework for shared resources scheduling

Nowadays, manufacturers have to share some of their resources with partners due to the competitive economic environment. The management of the availability periods of shared resources causes a problem because it is achieved by the scheduling systems, which assume a local environment where all resources are on the same site. Therefore, distributed scheduling with shared resources is an important research topic. In this communication, we introduce the architecture and behaviour of DSCEP framework (Distributed, Supervisor, Customer, Environment, and Producer) under shared resources situation with disturbances. We are using a simple example of manufacturing system to illustrate the ability of DSCEP framework to solve the shared resources scheduling problem in complex systems.     

Modeling self-deception within a decision-theoretic framework

Computational modeling of human belief maintenance and decision-making processes has become increasingly important for a wide range of applications. In this paper, we present a framework for modeling the human capacity for self-deception from a decision-theoretic perspective in which we describe an integrated process of wishful thinking which includes the determination of a desired belief state, the biasing of internal beliefs towards or away from this desired belief state, and the final decision-making process. Finally, we show that in certain situations self-deception can be beneficial.     

Scalable framework for blockchain-based shared manufacturing

Shared Manufacturing is a new mode of social manufacturing based on the principles of a sharing economy. This paper presents a scalable framework for blockchain-based Shared Manufacturing that preserves the transparency and immutability characteristics of transaction records, which is critical to building trust between entities in blockchain-based systems. We define a blockchain-based protocol for the service execution according to the design principles of the sharing economy. We present a scalable integration of blockchain technology into the concept of Shared Manufacturing by employing cross-chain solutions. We discuss existing cross chain technologies regarding the requirements of Shared Manufacturing and propose hybrid approach. We compare implementations of the proposed framework on two different blockchain networks: Ethereum public network and Xdai sidechain network. We conduct user-oriented test to explore the performance (cost and time) of the implementations in realistic situations in order to justify the use of the sidechain technology. Results indicate that the implementation on the sidechains provides greater scalability than the implementation on the public blockchain network.     

Distributed shared memory: a survey of issues and algorithms

An overview of distributed shared memory (DSM) issues is presented. Memory coherence, design choices, and implementation methods are included. The discussion of design choices covers structure and granularity, coherence semantics, scalability, and heterogeneity. Implementation issues concern data location and access, the coherence protocol, replacement strategy, and thrashing. Algorithms that support process synchronization and memory management are discussed     

分布式入侵检测框架

现有的网络安全系统往往功能比较单一,难以完成网络系统的整体防护要求,难以保证可靠地提供所需的业务,难以保证业务信息的安全可靠.因此,提出了一个分布式入侵检测框架DIDF,将防火墙、入侵检测和安全事件响应结合起来,实现了一个以管理为核心,具有防范、检测和响应能力的综合安全体系.     

Scheduling big data applications within advance reservation framework in optical grids

Providing QoS for big data applications requires a way to reserve computing and networking resources in advance. Within advance reservation framework, a multi-domain scheduling process is carried out in a top down hierarchical way across multiple hierarchical levels. This ensures that each domain executes intra-domain scheduling algorithm to co-schedule its own computing and networking resources while coordinating the scheduling at the inter-domain level. Within this process, we introduce two algorithms: iterative scheduling algorithm and K-shortest paths algorithm. We conducted a comprehensive performance evaluation study considering several metrics that reflect both grid system and grid user goals. The results demonstrated the advantages of the proposed scheduling process. Moreover, the results highlight the importance of using the iterative scheduling and K-shortest paths algorithms especially for data intensive applications.     

A framework for virtual organization creation in a breeding environment

Effectiveness in the process of creating virtual organizations (VO) is an important pre-condition for having a truly dynamic VOs, in response to collaboration opportunities in fast changing market contexts. A realistic approach to materialize agility in VO creation is defined under the assumption of a VO Breeding Environment (VBE), that guarantees the preparedness of its members to quickly get engaged in collaboration processes. After a survey of past approaches and a characterization of the VBE concept, a discussion of the process and suggested functionalities towards a VO creation framework are presented in this context. Finally a list of supporting tools is described and future research challenges are pointed out.     

Enabling shared audit data

Audit is an important aspect of good security and business practice; however, current solutions are not supportive of electronic data and processes. This paper describes an audit service that both acts as a central place for logging from heterogeneous IT systems and a place to search and check the audit data. Notarisation structures enabling a user to check the integrity of audit records and subsets of the audit chain relating to their transactions have been developed. The audit system uses a secure hardware device to create an alternative trust domain in which to run processes, maintaining the integrity of the audit trail whilst allowing it to be tightly integration and co-located with the overall IT infrastructure.     

Distributed dictionary learning for industrial process monitoring with big data

Applied Intelligence - With the development of sensor and communication technology, industrial systems have accumulated a large amount of data. This data has provided new perspectives and methods...     

Distributed shared memory for roaming large volumes

We present a cluster-based volume rendering system for roaming very large volumes. This system allows to move a gigabyte-sized probe inside a total volume of several tens or hundreds of gigabytes in real-time. While the size of the probe is limited by the total amount of texture memory on the cluster, the size of the total data set has no theoretical limit. The cluster is used as a distributed graphics processing unit that both aggregates graphics power and graphics memory. A hardware-accelerated volume renderer runs in parallel on the cluster nodes and the final image compositing is implemented using a pipelined sort-last rendering algorithm. Meanwhile, volume bricking and volume paging allow efficient data caching. On each rendering node, a distributed hierarchical cache system implements a global software-based distributed shared memory on the cluster. In case of a cache miss, this system first checks page residency on the other cluster nodes instead of directly accessing local disks. Using two Gigabit Ethernet network interfaces per node, we accelerate data fetching by a factor of 4 compared to directly accessing local disks. The system also implements asynchronous disk access and texture loading, which makes it possible to overlap data loading, volume slicing and rendering for optimal volume roaming.     

A framework for a feedback process to analyze and personalize a document vector space in a feature extraction model

In this paper, we present a framework for a feedback process to implement a highly accurate document retrieval system. In the system, a document vector space is created dynamically to implement retrieval processing. The retrieval accuracy of the system depends on the vector space. When the vector space is created based on a specific purpose and interest of a user, highly accurate retrieval results can be obtained. In this paper, we present a method for analyzing and personalizing the vector space according to the purposes and interests of users. In order to optimize the document vector space, we defined and implemented functions for the operations of adding, deleting and weighting the terms that were used to create the vector space. By exploiting effectively and dynamically the classified-document information related to the queries, our methods allow users to retrieve relevant documents for their interests and purposes. Even if the search results of the initial retrieval space are not appropriate, by applying the proposed feedback operations, our proposed method effectively improves the search results. We also implemented an experimental search system for semantic document retrieval. Several experimental results including comparisons of our method with the traditional relevance feedback method is presented to clarify how retrieval accuracy was improved by the feedback process and how accurately documents that satisfied the purpose and interests of users were extracted.     

MFC应用程序架构与动态创建对象

通过对Win32SDK程序中入口函数(WinMain)、初始化函数和窗口函数的分析,阐述了Windows应用程序的运行机理。在此基础上,进一步说明了MFC应用程序如何实现Windows应用程序运行的过程。另外也详细解析了MFC应用程序动态创建对象机制。     

Distributed state space minimization

We present a new algorithm, and its distributed implementation, for reducing labeled transition systems modulo strong bisimulation. The base of this algorithm is the Kanellakis–Smolka “naive method”, which has a high theoretical complexity but is successful in practice and well suited to parallelization. This basic approach is combined with optimizations inspired by the Kanellakis–Smolka algorithm for the case of bounded fanout, which has the best known time complexity. The distributed implementation is improved with respect to previous attempts by a better overlap between communication and computation, which results in an efficient usage of both memory and processing power. We also discuss the time complexity of this algorithm and show experimental results with sequential and distributed prototype tools.     

Distributed data mining: a survey

Most data mining approaches assume that the data can be provided from a single source. If data was produced from many physically distributed locations like Wal-Mart, these methods require a data center which gathers data from distributed locations. Sometimes, transmitting large amounts of data to a data center is expensive and even impractical. Therefore, distributed and parallel data mining algorithms were developed to solve this problem. In this paper, we survey the-state-of-the-art algorithms and applications in distributed data mining and discuss the future research opportunities.     

Facilitating the legislation process using a shared conceptual model

The Power (Program for an Ontology-based Working Environment for Rules and regulations) research program combines a knowledge-capitalization/knowledge-codification approach with an organization-dynamics approach. Initial results indicate that this method will help to improve the quality of law enforcement and decrease the time needed for implementing legislation and regulation changes.