Real-time data mining of massive data streams from synoptic sky surveys

The nature of scientific and technological data collection is evolving rapidly: data volumes and rates grow exponentially, with increasing complexity and information content, and there has been a transition from static data sets to data streams that must be analyzed in real time. Interesting or anomalous phenomena must be quickly characterized and followed up with additional measurements via optimal deployment of limited assets. Modern astronomy presents a variety of such phenomena in the form of transient events in digital synoptic sky surveys, including cosmic explosions (supernovae, gamma ray bursts), relativistic phenomena (black hole formation, jets), potentially hazardous asteroids, etc. We have been developing a set of machine learning tools to detect, classify and plan a response to transient events for astronomy applications, using the Catalina Real-time Transient Survey (CRTS) as a scientific and methodological testbed. The ability to respond rapidly to the potentially most interesting events is a key bottleneck that limits the scientific returns from the current and anticipated synoptic sky surveys. Similar challenge arises in other contexts, from environmental monitoring using sensor networks to autonomous spacecraft systems. Given the exponential growth of data rates, and the time-critical response, we need a fully automated and robust approach. We describe the results obtained to date, and the possible future developments.     

Simultaneous heat exchanger network synthesis for direct and indirect heat transfer inside and between processes

Process plants are typically divided into different process parts having specific processing tasks with possibly different ownership. Heat integration between these processes can increase the energy- and economic efficiency for both the overall plant and the individual processes. In this paper we present a heat exchanger network synthesis MINLP-model that allows simultaneous heat integration directly between streams in the same process and both directly and indirectly between streams in different processes. The indirect heat transfer is accomplished by using intermediate streams. Two examples, one small explanatory one and one from the literature, are optimized. The results verify that the model works logically.     

液柱冲击塔湿法烟气脱硫的试验研究

设计了一种新型的液柱冲击塔脱硫装置，通过冷态试验研究各种结构和运行参数的变化对液柱冲击塔脱硫效率的影响。研究结果表明，在试验范围内，液柱冲击塔的脱硫效率随着喷液密度、塔内风速、液气比、循环水池pH值、烟气入口温度的增大而提高，随着SO2入口质量浓度的增大而降低。减小液柱冲击塔阻挡层的通流截面或阻挡层与液柱喷嘴之间的距离可以有效改善脱硫浆液的雾化效果，从而显著提高脱硫效率。选择适当的结构和运行参数，液柱冲击塔的脱硫效率可以达到95%以上。     

Query indexing with containment-encoded intervals for efficient stream processing

Many continual range queries can be issued against data streams. To efficiently evaluate continual queries against a stream, a main memory-based query index with a small storage cost and a fast search time is needed, especially if the stream is rapid. In this paper, we study a CEI-based query index that meets both criteria for efficient processing of continual interval queries. This new query index is an indirect indexing approach. It centres around a set of predefined virtual containment-encoded intervals, or CEIs. The CEIs are used to first decompose query intervals and then perform efficient search operations. The CEIs are defined and labeled such that containment relationships among them are encoded in their IDs. The containment encoding makes decomposition and search operations efficient; from the encoding of the smallest CEI containing a data point, the encodings of other containing CEIs can be easily derived. Closed-form formulae for the bounds of the average index storage cost are derived. Simulations are conducted to evaluate the effectiveness of the CEI-based query index and to compare it with alternative approaches. The results show that the CEI-based query index significantly outperforms existing approaches in terms of both storage cost and search time. Kun-Lung Wu received the B.S. degree in electrical engineering from the National Taiwan University, Taipei, Taiwan, the M.S. and Ph.D. degrees in computer science from the University of Illinois at Urbana–Champaign. He is with the IBM Thomas J. Watson Research Center, currently a member of the Software Tools and Techniques Group. His current research interests include data streams, continual queries, mobile computing, Internet technologies and applications, database systems and distributed and parallel computing. He has published extensively and holds various patents in these areas. Dr. Wu is a Senior Member of the IEEE Computer Society and a member of the ACM. He was an Associate Editor for the IEEE Transactions on Knowledge and Data Engineering, 2000–2004. He was the general chair for the 3rd International Workshop on e-Commerce and Web-Based Information Systems (WECWIS 2001). He has served as an organising and program committee member on various conferences. He has received various IBM awards, including IBM Corporate Environmental Affair Excellence Award, Research Division Award and Invention Achievement Awards. He received a best paper award from IEEE EEE 2004. He is an IBM Master Inventor. Shyh-Kwei Chen received the B.S. degree in computer science and information engineering from National Taiwan University, Taipei, Taiwan, in 1983, the M.S. degree in computer science from the University of Minnesota, Minneapolis, in 1987, and the Ph.D. degree in computer science from University of Illinois at Urbana–Champaign, in 1994. Dr. Chen has been with the IBM Thomas J. Watson Research Center, Yorktown Heights, New York since October 1994, where he is currently a research staff member. His current research interests include XML, electronic commerce, business performance management, data engineering and compilers. He is a member of the ACM, the IEEE and the IEEE Computer Society. Philip S. Yu received the B.S. degree in electrical engineering from National Taiwan University, the M.S. and Ph.D. degrees in electrical engineering from Stanford University, and the M.B.A. degree from New York University. He is with the IBM Thomas J. Watson Research Center and is currently manager of the Software Tools and Techniques group. His research interests include data mining, Internet applications and technologies, database systems, multimedia systems, parallel and distributed processing and performance modelling. Dr. Yu has published more than 400 papers in refereed journals and conferences. He holds or has applied for more than 250 US patents. Dr. Yu is a Fellow of the ACM and a Fellow of the IEEE. He is an associate editor of ACM Transactions on Internet Technology. He is a member of the IEEE Data Engineering steering committee and is also on the steering committee of IEEE Conference on Data Mining. He was the Editor-in-Chief of IEEE Transactions on Knowledge and Data Engineering (2001–2004), an editor and advisory board member of IEEE Transactions on Knowledge and Data Engineering and also a guest coeditor of the special issue on mining of databases. He had also served as an associate editor of Knowledge and Information Systems. In addition to serving as program committee member on various conferences, he was the program cochair of the 11th International Conference on Data Engineering, the 6th Pacific Area Conference on Knowledge Discovery and Data Mining, and the 9th ACM SIGMOD Workshop on Research Issues in Data Mining and Knowledge Discovery, and the program chair of the 2nd International Workshop on Research Issues on Data Engineering: Transaction and Query Processing, the PAKDD Workshop on Knowledge Discovery from Advanced Databases and the 2nd International Workshop on Advanced Issues of E-Commerce and Web-based Information Systems. He served as the general chair of the 14th International Conference on Data Engineering and the general cochair of the 2nd IEEE International Conference on Data Mining. He has received several IBM honours, including two IBM Outstanding Innovation Awards, an Outstanding Technical Achievement Award, two Research Division Awards and the 81st Plateau of Invention Achievement Awards. He received an Outstanding Contributions Award from IEEE International Conference on Data Mining in 2003 and also an IEEE Region 1 Award for “promoting and perpetuating numerous new electrical engineering concepts” in 1999. Dr. Yu is an IBM Master Inventor and was recognised as one of the IBM's 10 top leading inventors in 1999.     

Tracking set-expression cardinalities over continuous update streams

There is growing interest in algorithms for processing and querying continuous data streams (i.e., data seen only once in a fixed order) with limited memory resources. In its most general form, a data stream is actually an update stream, i.e., comprising data-item deletions as well as insertions. Such massive update streams arise naturally in several application domains (e.g., monitoring of large IP network installations or processing of retail-chain transactions). Estimating the cardinality of set expressions defined over several (possibly distributed) update streams is perhaps one of the most fundamental query classes of interest; as an example, such a query may ask what is the number of distinct IP source addresses seen in passing packets from both router R ₁ and R ₂ but not router R ₃?. Earlier work only addressed very restricted forms of this problem, focusing solely on the special case of insert-only streams and specific operators (e.g., union). In this paper, we propose the first space-efficient algorithmic solution for estimating the cardinality of full-fledged set expressions over general update streams. Our estimation algorithms are probabilistic in nature and rely on a novel, hash-based synopsis data structure, termed 2-level hash sketch. We demonstrate how our 2-level hash sketch synopses can be used to provide low-error, high-confidence estimates for the cardinality of set expressions (including operators such as set union, intersection, and difference) over continuous update streams, using only space that is significantly sublinear in the sizes of the streaming input (multi-)sets. Furthermore, our estimators never require rescanning or resampling of past stream items, regardless of the number of deletions in the stream. We also present lower bounds for the problem, demonstrating that the space usage of our estimation algorithms is within small factors of the optimal. Finally, we propose an optimized, time-efficient stream synopsis (based on 2-level hash sketches) that provides similar, strong accuracy-space guarantees while requiring only guaranteed logarithmic maintenance time per update, thus making our methods applicable for truly rapid-rate data streams. Our results from an empirical study of our synopsis and estimation techniques verify the effectiveness of our approach.Received: 20 October 2003, Accepted: 16 April 2004, Published online: 14 September 2004Edited by: J. Gehrke and J. Hellerstein.Sumit Ganguly: sganguly@cse.iitk.ac.in Current affiliation: Department of Computer Science and Engineering, Indian Institute of Technology, Kanpur, India     

The design and implementation of a Jini/Java-based A/V stream control and management

Distributed multimedia applications make diverse demands on communication services and quality of service. These requirements must be met end-to-end in an efficient and integrated manner through the enabling middleware of end systems and communication networks. The middleware should allow an adaptive quality of service (QoS) to be specified and supported; it should also provide application programming interfaces with integrated group communication support that simplify the programming task of multimedia applications. This paper focuses on the latter aspect and presents a distributed solution known as Stream Manager. Stream Manager allows heterogeneous media devices to be connected by the same session initiation procedures of Stream Manager. Through an underlying network connection management service, it allows a new group stream to be supported in addition to the OMGs unicast streams and point-to-multipoint multicast streams. The basic operations of Stream Manager and its application interfaces will be described, and the design and implementation of a prototype in Jini/Java will be presented. The performance of the prototype was measured experimentally in terms of throughput, delay, and latencies of joining and leaving a stream. We then compared its performance with that of streams handled by using Java RMI, Java sockets, and CORBA A/V stream. The performance of our system was found to be superior to that of Java RMI and comparable to that of Java socket but slightly inferior to that of CORBA A/V stream due to the higher intrinsic Java processing overhead.     

Chemical,rheological and parotta making characteristics of flourmill streams

Chemical, rheological and parotta-making characteristics of wheat-milling flour streams were studied with the main aim of developing a speciality flour for parotta. Important quality characteristics such as ash, flour colour, dry gluten content, sedimentation value, free and bound lipid content, farinograph and alveograph characteristics of the mill streams were determined. The studies showed an increase in ash, gluten content and sedimentation value with increasing numbers of breaks in the flour streams. An increase in percentage whiteness and bound lipids was observed in the first five reduction passages (C1 to C5). The alveograph characteristics indicated that the average abscissa at rupture (L) length of the curve increased with increasing reduction streams from C1 to C5, and also that the curves were better balanced when compared to 1 BK to 5 BK. The parottas made from the first five break passages had a decreased spread ratio, dull brown colour, and fused layers and lower overall quality score (43–79). The initial reduction streams (C1 to C5) produced good quality parottas in terms of appearance, spread, layers and texture. The shear force values indicating the texture of the product ranged from 1,120 to 1,250 g. The speciality flour made by combining C1 to C5 streams had 0.45% ash, 9.63% dry gluten, 45 ml sodium dodecyl sulphate sedimentation value, 0.7% free lipid and 1.8% bound lipid and produced parottas having creamish white colour, excellent pliability, thin, transparent and distinct layers, soft texture and moderate chewiness. In the mouth the parottas broke down easily, without leaving any residue. The overall quality score of parotta was highest (93.5) for the speciality flour when compared to break and reduction streams (43–77 and 56–92 respectively).     

撞击流-微乳液法纳米羟基磷灰石的制备

在一定pH值条件下，采用水溶液（W）／环己烷（O）／OP-10（S）／正辛醇（A）四元反相微乳液体系并结合撞击流技术制备出粒径为20-50nm的针状纳米羟基磷灰石（HA）晶体。红外光谱（IR）、X射线衍射（XRD）和透射电镜（TEM）对羟基磷灰石颗粒进行表征和分析表明：颗粒分散均匀，成针状，沿C轴方向的微晶尺寸约30nm，沿a轴方向为20nm左右。并主要考察了表面活性剂、杂质、反应和静置时间等对HA的微观形貌的影响。     

Efficient tracking of moving objects using a relational database

Tracking uncooperative moving objects by means of radar is a complex task due to clutter and association problems in multi-target scenarios. An approach to solve this problem is probabilistic multiple hypothesis tracking (PMHT). This method combines classical track filtering with a likelihood ratio test for the estimation of the plot-to-track association. The basics of PMHT and similar algorithms have gained much attention recently. However, the efficient implementation of real world applications of this technique still represents a challenging task. Since a common requirement in this context is the reliable storage of track data in a database, an implementation of the tracker's calculation inside a database management system (DBMS) using SQL views is desirable. A naive implementation of PMHT using a commercial DBMS, however, usually leads to performance problems because of the high frequency of measurement updates. In this paper, we propose possible optimizations for solving these performance problems. Their usage leads to a dramatic run-time improvement in our sample case and makes the implementation of PMHT in a database context feasible.