Query processing of multi-way stream window joins

This paper introduces a class of join algorithms, termed W-join, for joining multiple infinite data streams. W-join addresses the infinite nature of the data streams by joining stream data items that lie within a sliding window and that match a certain join condition. In addition to its general applicability in stream query processing, W-join can be used to track the motion of a moving object or detect the propagation of clouds of hazardous material or pollution spills over time in a sensor network environment. We describe two new algorithms for W-join and address variations and local/global optimizations related to specifying the nature of the window constraints to fulfill the posed queries. The performance of the proposed algorithms is studied experimentally in a prototype stream database system, using synthetic data streams and real time-series data. Tradeoffs of the proposed algorithms and their advantages and disadvantages are highlighted, given variations in the aggregate arrival rates of the input data streams and the desired response times per query. This is an extended version of the paper published in the Proceedings of the 15th International Conference on Scientific and Statistical Database Management, SSDBM 2003, Boston, U.S.A., pp. 75–84.     

CPU load shedding for binary stream joins

We present an adaptive load shedding approach for windowed stream joins. In contrast to the conventional approach of dropping tuples from the input streams, we explore the concept ofselective processing for load shedding. We allow stream tuples to be stored in the windows and shed excessive CPU load by performing the join operations, not on the entire set of tuples within the windows, but on a dynamically changing subset of tuples that are learned to be highly beneficial. We support such dynamic selective processing through three forms of runtimeadaptations: adaptation to input stream rates, adaptation to time correlation between the streams and adaptation to join directions. Our load shedding approach enables us to integrateutility-based load shedding withtime correlation-based load shedding. Indexes are used to further speed up the execution of stream joins. Experiments are conducted to evaluate our adaptive load shedding in terms of output rate and utility. The results show that our selective processing approach to load shedding is very effective and significantly outperforms the approach that drops tuples from the input streams. Bugra Gedik received the B.S. degree in C.S. from the Bilkent University, Ankara, Turkey, and the Ph.D. degree in C.S. from the College of Computing at the Georgia Institute of Technology, Atlanta, GA, USA. He is with the IBM Thomas J. Watson Research Center, currently a member of the Software Tools and Techniques Group. Dr. Gedik's research interests lie in data intensive distributed computing systems, spanning data-centric peer-to-peer overlay networks, mobile and sensor-based distributed data management systems, and distributed data stream processing systems. His research focus is on developing system-level architectures and techniques to address scalability problems in distributed continual query systems and applications. He is the recipient of the ICDCS 2003 best paper award. He has served in the program committees of several international conferences, such as ICDE, MDM, and CollaborateCom. Kun-Lung Wu received the B.S. degree in E.E. from the National Taiwan University, Taipei, Taiwan, the M.S. and Ph.D. degrees in C.S. both from the University of Illinois at Urbana-Champaign. He is with the IBM Thomas J. Watson Research Center, currently a member of the Software Tools and Techniques Group. His recent research interests include data streams, continual queries, mobile computing, Internet technologies and applications, database systems and distributed computing. He has published extensively and holds many patents in these areas. Dr. Wu is a Senior Member of the IEEE Computer Society and a member of the ACM. He is the Program Co-Chair for the IEEE Joint Conference on e-Commerce Technology (CEC 2007) and Enterprise Computing, e-Commerce and e-Services (EEE 2007). He was an Associate Editor for the IEEE Trans. on Knowledge and Data Engineering, 2000–2004. He was the general chair for the 3rd International Workshop on E-Commerce and Web-Based Information Systems (WECWIS 2001). He has served as an organizing and program committee member on various conferences. He has received various IBM awards, including IBM Corporate Environmental Affair Excellence Award, Research Division Award, and several Invention Achievement Awards. He received a best paper award from IEEE EEE 2004. He is an IBM Master Inventor. Philip S. Yu received the B.S. Degree in E.E. from National Taiwan University, the M.S. and Ph.D. degrees in E.E. from Stanford University, and the M.B.A. degree from New York University. He is with the IBM Thomas J. Watson Research Center and currently manager of the Software Tools and Techniques group. His research interests include data mining, Internet applications and technologies, database systems, multimedia systems, parallel and distributed processing, and performance modeling. Dr. Yu has published more than 430 papers in refereed journals and conferences. He holds or has applied for more than 250 US patents. Dr. Yu is a Fellow of the ACM and a Fellow of the IEEE. He is associate editors of ACM Transactions on the Internet Technology and ACM Transactions on Knowledge Discovery in Data. He is a member of the IEEE Data Engineering steering committee and is also on the steering committee of IEEE Conference on Data Mining. He was the Editor-in-Chief of IEEE Transactions on Knowledge and Data Engineering (2001–2004), an editor, advisory board member and also a guest co-editor of the special issue on mining of databases. He had also served as an associate editor of Knowledge and Information Systems. In addition to serving as program committee member on various conferences, he will be serving as the general chair of 2006 ACM Conference on Information and Knowledge Management and the program chair of the 2006 joint conferences of the 8th IEEE Conference on E-Commerce Technology (CEC' 06) and the 3rd IEEE Conference on Enterprise Computing, E-Commerce and E-Services (EEE' 06). He was the program chair or co-chairs of the 11th IEEE Intl. Conference on Data Engineering, the 6th Pacific Area Conference on Knowledge Discovery and Data Mining, the 9th ACM SIGMOD Workshop on Research Issues in Data Mining and Knowledge Discovery, the 2nd IEEE Intl. Workshop on Research Issues on Data Engineering: Transaction and Query Processing, the PAKDD Workshop on Knowledge Discovery from Advanced Databases, and the 2nd IEEE Intl. Workshop on Advanced Issues of E-Commerce and Web-based Information Systems. He served as the general chair of the 14th IEEE Intl. Conference on Data Engineering and the general co-chair of the 2nd IEEE Intl. Conference on Data Mining. He has received several IBM honors including 2 IBM Outstanding Innovation Awards, an Outstanding Technical Achievement Award, 2 Research Division Awards and the 84th plateau of Invention Achievement Awards. He received an Outstanding Contributions Award from IEEE Intl. Conference on Data Mining in 2003 and also an IEEE Region 1 Award for “promoting and perpetuating numerous new electrical engineering concepts” in 1999. Dr. Yu is an IBM Master Inventor. Ling Liu is an associate professor at the College of Computing at Georgia Tech. There, she directs the research programs in Distributed Data Intensive Systems Lab (DiSL), examining research issues and technical challenges in building large scale distributed computing systems that can grow without limits. Dr. Liu and the DiSL research group have been working on various aspects of distributed data intensive systems, ranging from decentralized overlay networks, exemplified by peer to peer computing, data grid computing, to mobile computing systems and location based services, sensor network computing, and enterprise computing systems. She has published over 150 international journal and conference articles. Her research group has produced a number of software systems that are either open sources or directly accessible online, among which the most popular ones are WebCQ and XWRAPElite. Dr. Liu is currently on the editorial board of several international journals, including IEEE Transactions on Knowledge and Data Engineering, International Journal of Very large Database systems (VLDBJ), International Journal of Web Services Research, and has chaired a number of conferences as a PC chair, a vice PC chair, or a general chair, including IEEE International Conference on Data Engineering (ICDE 2004, ICDE 2006, ICDE 2007), IEEE International Conference on Distributed Computing (ICDCS 2006), IEEE International Conference on Web Services (ICWS 2004). She is a recipient of IBM Faculty Award (2003, 2006). Dr. Liu's current research is partly sponsored by grants from NSF CISE CSR, ITR, CyberTrust, a grant from AFOSR, an IBM SUR grant, and an IBM faculty award.     

流系统的降载技术

流数据连续查询及优化研究已成为当前国际数据库研究领域的一个热点。数据流的到达速率经常是不可预测的且具有很高的突发性,数据流速这种不规则的变化会引起系统负载急剧的波动。当输入速率超过系统处理能力时,系统会发生过载并且导致系统性能的恶化,降载技术是解决此问题最有效的途径之一。对降载技术中系统负载估计、降载器的最佳的放置位置、降载量的大小、降载器合并等关键问题进行了讨论。     

Load shedding and distributed resource control of stream processing networks

Recent advances in networking and information technology boost the development of new and advanced services offered over communication systems that integrate a widely heterogeneous mix of applications and computer devices. Without careful traffic control and resource management, the implied dramatic increase in the demand for networking resources and remote application services may lead to substantial degradation of the Quality of Service as experienced by the end users. In this paper, we consider the problem of joint admission control and dynamic resource allocation in a stream processing network so as to optimize the overall system utility. With a primal-dual-based optimization approach, we show that the resource allocation problem and the admission control problem can be decomposed. We then present a distributed algorithm which incorporates a push-and-pull-based admission control mechanism, and a pressure-based cμ rule for resource allocation. We show that the algorithm guarantees the stability of the network and converges to the optimal solution. Various numerical experiments are then presented to demonstrate the quality of the solution and the speed of convergence.     

以订单流为主线的进销存系统设计与实现

以纺织企业中一类企业面临的实际问题为背景,为帮助企业提升管理水平和提高核心竞争力,设计了一个以订单流为主线的进销存系统。此系统具有平台无关、延展性性好、柔韧性强等特点。从系统体系结构、功能和关键技术几个方面详细介绍了此系统的设计方案,并给出了以模版开发的实现方法。应用结果数据显示此系统有着较好的应用效果。     

基于多路交叉的用户金融行为预测

针对通过挖掘用户的金融行为来改善金融领域的服务模式和服务质量的问题,本文提出了一种基于多路交叉特征的用户金融行为预测算法.根据数据包含的属性构建训练的特征,基于因子分解机模型(FM)利用下游行为预测任务对金融数据的特征进行预训练,获取数据特征的隐含向量.引入特征交叉层对金融数据的高阶特征进行提取,解决FM线性模型只能提...     

双向过滤的字符串相似连接验证方法

字符串相似连接是指在字符串集合中找出相似的字符串对,是许多应用的关键操作,寻找高效的字符串相似连接算法已成为研究热点。基于划分的过滤-验证方法（Pass-Join）与其他方法相比具有较高的效率。它按照字符串长度递增的顺序访问字符串集合,通过查找一个字符串的划分块是否存在于另一个字符串中,快速筛选出可能相似的字符串对（候选集）,然后利用编辑距离进行相似性验证。研究发现,按照字符串长度递减的顺序进行过滤（长度递减过滤）的效果优于按照长度递增的顺序过滤（长度递增过滤）的效果,基于此,提出双向过滤-验证机制：在过滤阶段对长度递减过滤的结果再进行一次长度递增过滤,进一步减小候选集大小;在验证阶段利用双向过滤产生的两对划分块和其匹配子串分隔字符串对,从而减小需要验证的字符串的长度,加速验证过程。实验证明,双向过滤-验证算法在真实数据集上优于原算法。     

数据流滑动窗口聚集查询降载策略研究*

滑动窗口聚集查询在数据流管理系统中应用广泛,数据流到达高峰期,必须考虑滑动窗口聚集查询中出现的降载问题。分析了子集模型的特点和已有降载策略的不足,给出了数据流滑动窗口聚集查询降载问题的约束条件,提出了能保证子集结果产生的基于丢弃窗口更新策略的降载算法。理论分析和实验结果表明,该算法对数据流滑动窗口聚集查询降载问题的处理具有较高的有效性和实用性。     

基于自适应多向独立成分分析的间歇过程监控的研究

针对间歇过程批次与批次之间,操作条件缓慢变化的特性,提出一种基于自适应多向独立成分分析(MICA)的监控算法。该方法首先用MICA法建模,然后在历史数据集中加入新的正常批次并剔除最早批次,逐渐更新模型,同时引入遗忘因子,提高对新过程特性的适应性。青霉素发酵过程的仿真结果表明,自适应MICA比MICA更准确地描述过程行为,并有效减少检测故障时的误报。     

基于EXN-Tree编码的XML结构连接算法研究

提出了EXN-Tree的概念,将XML文档树的节点映射到EXN-Tree,依据EXN-Tree的节点编码生成XML文档树节点数据结构。基于此新型的节点编码结构,就无序无索引节点集和有序有索引节点集两种情况下的XML结构连接算法展开研究,提出了一系列的结构连接算法,解决了无序无索引节点集和有序有索引节点集两种情况下的XML结构连接。分析表明该算法的I/O复杂性优于已有算法,具有良好的性能。     

一种基于演化计算的序列密码分析方法

序列密码是一类重要的密码,演化计算是一种重要的智能计算。在研究利用演化计算进行序列密码分析方法的基础上,具体给出了一种利用演化计算对非线性滤波型序列密码体制进行分析的方法。分别在移位器初态未知和抽头位置未知两种情况下,对滤波流密码体制进行了密码分析。实验结果表明,该算法的攻击复杂度远远小于穷举攻击的复杂度。     

基于多阶段MPLS法的发酵过程质量预测

由于发酵过程中,系统的非线性特性与发酵阶段密切相关,而应用常规MPLS时的静态单一模型、模型失配、不能充分有效压缩和抽取非线性信息、估计未来测量变量引入模型误差等问题,提出1种多阶段MPLS法.先采用ISODATA动态分类算法聚类分析过程数据,划分子阶段,针对各批次子阶段不等长的特点,再用DTW算法同步阶段轨迹,然后在...     

基于时间反演的到达时间定位

针对传统算法在室内超宽带(UWB)到达时间(TOA)定位系统里很难准确搜寻出第一条直射路径,从而导致定位精度不高的问题,提出了基于时间反演(TR)的TOA室内UWB定位算法.首先,利用TR处理的空时聚焦特性确定第一条直射路径,从而估计这条路径的TOA;其次,通过加权最小二乘(WLS)定位算法对不同的估计分量赋予相应的权...     

基于时变向量场的多无人机编队集结控制方法

编队集结是无人机编队飞行的必要条件.固定翼无人机由于其速度可变范围有限,难以通过单一的速度调整的方式构成编队.本文提出了一种以路径调节为主、速度调节为辅的无人机编队集结策略,根据各无人机到达集结位置的最短预估时间确定集结时间.并提出一种时变向量场的构造方式,将水平机动的思想与时变向量场相结合,根据无人机的飞行状况通过时变向量场实时调整路径长度,同时调整自身速度,使各无人机都能在集结时间到达集结位置,且速度和航向角保持一致,从而实现编队集结.随后证明了系统的收敛性,并分析了实际情况中避撞问题的处理.论文用某型无人机的大偏差非线性数学模型对所提的编队集结方法进行了验证,证明了所提方法具有较好的实用性.     

基于距离的数据流离群点挖掘算法

传统的离群点挖掘算法无法有效挖掘数据流中的离群点。针对数据流的无限输入和动态变化等特点,提出一种新的基于距离的数据流离群点挖掘算法。通过Hoeffding定理及独立同分布中心极限定理,对数据流概率分布变化进行动态检测,利用检测结果自适应调整滑动窗口大小对数据流离群点进行挖掘。实验结果表明,该算法在人工数据集和真实数据集KDD-CUP99中可以对数据流中的离群点进行有效挖掘。     

一种基于无线传感器阵列的到达方向检测方法

以线性等距无线传感器阵列为例,提出一种有效的到达方向检测算法.列堆栈两个平移不变子阵的相关矩阵,给出一种奇异值分解和特征值分解相结合的两步算法,估计传感器阵列的导向矢量矩阵,通过分析导向矢量矩阵的结构化信息,构造估计导向矢量和理想导向矢量的相关函数,进而求解相关函数的驻点,搜索有限个驻点中使相关函数最大的驻点对应的角度估计到达方向,避免了穷尽搜索.仿真结果表明：所提算法在相同信噪比下分辨成功率高于著名的ESPRIT算法、同一分辨成功率下要求的信噪比更低.在信噪比、快拍数、阵元个数变化下对目标定位的均方根误差均优于ESPRIT算法,更接近于理论最优值.     

Genetic algorithm based on receding horizon control for arrival sequencing and scheduling

The concept of Receding Horizon Control (RHC) is introduced into Genetic Algorithm (GA) in this paper to solve the problem of arrival scheduling and sequencing (ASS) at a busy hub airport. A GA-based method is proposed for solving the dynamic ASS problem, and the focus is put on the methodology of integrating the RHC strategy into the GA for real-time implementations in a dynamic environment of air traffic control. Receding horizon and terminal penalty are investigated in depth as two key techniques of this novel RHC-based GA. Simulation results show that the new method proposed in this paper is effective and efficient to solve the ASS problem in a dynamic environment.     

基于时空混沌的视频流加密算法

基于视频具有数据量大、实时性要求高等特点,在一维耦合映像格子模型的基础上,对耦合格点结构和动力学模型这两个方面进行了改进,在系统结构上采用一维链加二维网格再加上一维链的组合模型来构造驱动和响应系统及多级加密的方案.实验结果表明,系统在增强密钥序列复杂度的同时也提高了加密效率,使得系统的抗破译能力大大加强.     

基于MSP430单片机的远程束流诊断系统编程

通过RS-485总线和Intranet网络,来实现对柬流的远程测量与控制,并给出软件的C程序实现和部分硬件功能模块的程序流程图。系统功能模块VAC401采用TI公司的MSP430F169混合信号处理器,具有超低功耗和高集成度等优点。利用它构建的控制网络功能强大,结构简单,可靠性高,抗干扰能力强,一般不需扩展外围器件。本来流诊断系统能够完成对加速器中柬流的远距离测量与控制。