A study of Q-learning considering negative rewards

In the reinforcement learning system, the agent obtains a positive reward, such as 1, when it achieves its goal. Positive rewards are propagated around the goal area, and the agent gradually succeeds in reaching its goal. If you want to avoid certain situations, such as dangerous places or poison, you might want to give a negative reward to the agent. However, in conventional Q-learning, negative rewards are not propagated in more than one state. In this article, we propose a new way to propagate negative rewards. This is a very simple and efficient technique for Q-learning. Finally, we show the results of computer simulations and the effectiveness of the proposed method.     

A dual K-Na selective Prussian blue nanotubes sensor

A strategy for dual sensing of Na⁺ and K⁺ ions using Prussian blue nanotubes via selective inter/deintercalation of K⁺ ion and competitive inhibition by Na⁺ ion, is reported. The analytical signal is derived from the cyclic voltammetry cathodic peak position E_pc of Prussian blue nanotubes. Na⁺ and K⁺ levels in a sample solution can be determined conveniently using one Prussian blue nanotubes sensor. In addition, this versatile method can be applied for the analysis of single type of either Na⁺ or K⁺ ions. The dual-ion sensor response towards Na⁺ and K⁺ can be described using a model based on the competitive inhibition effects of Na⁺ on K⁺ inter/deintercalation in Prussian blue nanotubes. Successful application of the Prussian blue nanotubes sensor for Na⁺ and K⁺ determination is demonstrated in artificial saliva.     

A dispatching method for automated lifting vehicles in automated port container terminals

In automated container terminals, containers are transported from the marshalling yard to a ship and vice versa by automated vehicles. The automated vehicle type studied in this paper is an automated lifting vehicle (ALV) that is capable of lifting a container from the ground by itself. This study discusses how to dispatch ALVs by utilizing information about pickup and delivery locations and time in future delivery tasks. A mixed-integer programming model is provided for assigning optimal delivery tasks to ALVs. A procedure for converting buffer constraints into time window constraints and a heuristic algorithm for overcoming the excessive computational time required for solving the mathematical model are suggested. Numerical experiments are reported to compare the objective values and computational times by a heuristic algorithm with those by an optimizing method and to analyze the effects of dual cycle operation, number of ALVs, and buffer capacity on the performance of ALVs.     

Efficient duration and hierarchical modeling for human activity recognition

A challenge in building pervasive and smart spaces is to learn and recognize human activities of daily living (ADLs). In this paper, we address this problem and argue that in dealing with ADLs, it is beneficial to exploit both their typical duration patterns and inherent hierarchical structures. We exploit efficient duration modeling using the novel Coxian distribution to form the Coxian hidden semi-Markov model (CxHSMM) and apply it to the problem of learning and recognizing ADLs with complex temporal dependencies. The Coxian duration model has several advantages over existing duration parameterization using multinomial or exponential family distributions, including its denseness in the space of nonnegative distributions, low number of parameters, computational efficiency and the existence of closed-form estimation solutions. Further we combine both hierarchical and duration extensions of the hidden Markov model (HMM) to form the novel switching hidden semi-Markov model (SHSMM), and empirically compare its performance with existing models. The model can learn what an occupant normally does during the day from unsegmented training data and then perform online activity classification, segmentation and abnormality detection. Experimental results show that Coxian modeling outperforms a range of baseline models for the task of activity segmentation. We also achieve a recognition accuracy competitive to the current state-of-the-art multinomial duration model, while gaining a significant reduction in computation. Furthermore, cross-validation model selection on the number of phases K in the Coxian indicates that only a small K is required to achieve the optimal performance. Finally, our models are further tested in a more challenging setting in which the tracking is often lost and the activities considerably overlap. With a small amount of labels supplied during training in a partially supervised learning mode, our models are again able to deliver reliable performance, again with a small number of phases, making our proposed framework an attractive choice for activity modeling.     

Information in the nonstationary case

Information estimates such as the direct method of Strong, Koberle, de Ruyter van Steveninck, and Bialek (1998) sidestep the difficult problem of estimating the joint distribution of response and stimulus by instead estimating the difference between the marginal and conditional entropies of the response. While this is an effective estimation strategy, it tempts the practitioner to ignore the role of the stimulus and the meaning of mutual information. We show here that as the number of trials increases indefinitely, the direct (or plug-in) estimate of marginal entropy converges (with probability 1) to the entropy of the time-averaged conditional distribution of the response, and the direct estimate of the conditional entropy converges to the time-averaged entropy of the conditional distribution of the response. Under joint stationarity and ergodicity of the response and stimulus, the difference of these quantities converges to the mutual information. When the stimulus is deterministic or nonstationary the direct estimate of information no longer estimates mutual information, which is no longer meaningful, but it remains a measure of variability of the response distribution across time.     

SVM for English semantic classification in parallel environment

Semantic analysis is very important and very helpful for many researches and many applications for a long time. SVM is a famous algorithm which is used in the researches and applications in many different fields. In this study, we propose a new model using a SVM algorithm with Hadoop Map (M)/Reduce (R) for English document-level emotional classification in the Cloudera parallel network environment. Cloudera is also a distributed system. Our English testing data set has 25,000 English documents, including 12,500 English positive reviews and 12,500 English negative reviews. Our English training data set has 90,000 English sentences, including 45,000 English positive sentences and 45,000 English negative sentences. Our new model is tested on the English testing data set and we achieve 63.7% accuracy of sentiment classification on this English testing data set.     

A decision tree using ID3 algorithm for English semantic analysis

Natural language processing has been studied for many years, and it has been applied to many researches and commercial applications. A new model is proposed in this paper, and is used in the English document-level emotional classification. In this survey, we proposed a new model by using an ID3 algorithm of a decision tree to classify semantics (positive, negative, and neutral) for the English documents. The semantic classification of our model is based on many rules which are generated by applying the ID3 algorithm to 115,000 English sentences of our English training data set. We test our new model on the English testing data set including 25,000 English documents, and achieve 63.6% accuracy of sentiment classification results.     

Examining the Effectiveness of Gamification in Human Computation

Within the human computation paradigm, gamification is increasingly gaining interest. This is because an enjoyable experience generated by game features can be a powerful approach to attract participants. Although potentially useful, little research has been conducted into understanding the effectiveness of gamification in human computation. In this experimental study, we operationalized effectiveness as perceived engagement and user acceptance and examined it by comparing the performance of a gamified human computation system against a non-gamified version. We also investigate the determinants of acceptance and how their effects differ between these two systems. Analysis of our data found that participants experienced more engagement and showed higher behavioral intentions toward the gamified system. Moreover, perceived output quality and perceived engagement were significant determinants of acceptance of the gamified system. In contrast, determinants for acceptance of the non-gamified system were perceived output quality and perceived usability.     

BSPNN: boosted subspace probabilistic neural network for email security

In the modern age of Internet connectivity, advanced information systems have accumulated huge volumes of data. Such fast growing, tremendous amount of data, collected and stored in large databases has far exceeded our human ability to comprehend without proper tools. There has been a great deal of research conducted to explore the potential applications of Machine Learning technologies in Security Informatics. This article studies the Network Security Detection problems in which predictive models are constructed to detect network security breaches such as spamming. Due to overwhelming volume of data, complexity and dynamics of computer networks and evolving cyber threats, current security systems suffer limited performance with low detection accuracy and high number of false alarms. To address such performance issues, a novel Machine Learning algorithm, namely Boosted Subspace Probabilistic Neural Network (BSPNN), has been proposed which combines a Radial Basis Function Neural Network with an innovative diversity-based ensemble learning framework. Extensive empirical analyses suggested that BSPNN achieved high detection accuracy with relatively small computational complexity compared with other conventional detection methods.