Supposed maximum information for comprehensible representations in SOM

In this paper, we propose a new information-theoretic method to simplify the computation of information and to unify several methods in one framework. The new method is called “supposed maximum information,” used to produce humanly comprehensible representations in competitive learning by taking into account the importance of input units. In the new learning method, by supposing the maximum information of input units, the actual information of input units is estimated. Then, the competitive network is trained with the estimated information in input units. The method is applied not to pure competitive learning, but to self-organizing maps, because it is easy to demonstrate visually how well the new method can produce more interpretable representations. We applied the method to three well-known sets of data, namely, the Kohonen animal data, the SPECT heart data and the voting data from the machine learning database. With these data, we succeeded in producing more explicit class boundaries on the U-matrices than did the conventional SOM. In addition, for all the data, quantization and topographic errors produced by our method were lower than those by the conventional SOM.     

Information enhancement for interpreting competitive learning

In this paper, we propose a new method called information enhancement to interpret internal representations of competitive learning. We consider competitive learning as a process of mutual information maximisation on input patterns. Then, we examine to what extent this mutual information can be increased or decreased by focusing upon or enhancing some elements in a network. If this enhancement for the elements increases information on input patterns, these elements possess more information on input patterns. Thus, we only have to carefully examine those elements in a network. We applied the method to an artificial problem, the Iris problem and an air pollution problem. In all problems, we succeeded in extracting important features in patterns. In addition, final maps were better than those obtained by the conventional self-organising map. We can say that this is the first step towards the full understanding of internal representations in competitive learning.     

Structural enhanced information and its application to improved visualization of self-organizing maps

In this paper, we propose structural enhanced information for detecting and visualizing main features in input patterns. We have so far proposed information enhancement for feature detection, where, if we want to focus upon components such as units and connection weights and interpret the functions of the components, we have only to enhance competitive units with the components. Though this information enhancement has given favorable results in feature detection, we further refine the information enhancement and propose structural enhanced information. In structural enhanced information, three types of enhanced information can be differentiated, that is, first-, second- and third-order enhanced information. The first-order information is related to the enhancement of competitive units themselves in a competitive network, and the second-order information is dependent upon the enhancement of competitive units with input patterns. Then, the third-order information is obtained by subtracting the effect of the first-order information from the second-order information. Thus, the third-order information more explicitly represents information on input patterns. With this structural enhanced information, we can estimate more detailed features in input patterns. For demonstrating explicitly and intuitively the improved performance of our method, the conventional SOM was used, and we transformed competitive unit outputs so as to improve visualization. The method was applied to the well-known Iris problem, an OECD countries classification problem and the Johns Hopkins University Ionosphere database. In all these problems, we succeeded in visualizing the detailed and important features of input patterns by using the third-order information.     

Repeated comprehensibility maximization in competitive learning

In this study, we propose a new type of information-theoretic method in which the comprehensibility of networks is progressively improved upon within a course of learning. The comprehensibility of networks is defined by using mutual information between competitive units and input patterns. When comprehensibility is maximized, the most simplified network configurations are expected to emerge. Comprehensibility is defined for competitive units, and the comprehensibility of the input units is measured by examining the comprehensibility of competitive units, with special attention being paid to the input units. The parameters to control the values of comprehensibility are then explicitly determined so as to maximize the comprehensibility of both the competitive units and the input units. For the sake of easy reproducibility, we applied the method to two problems from the well-known machine learning database, namely, the Senate problem and the cancer problem. In both experiments, any type of comprehensibility can be improved upon, and we observed that fidelity measures such as quantization errors could also be improved.     

Input information maximization for improving self-organizing maps

In this paper, we propose a new type of information-theoretic method called “input information maximization” to improve the performance of self-organizing maps. We consider outputs from input neurons by focusing on winning neurons. The outputs are based on the difference between input neurons and the corresponding winning neurons. Then, we compute the uncertainty of input neurons by normalizing the outputs. Input information is defined as a decrease in the uncertainty of input neurons from a maximum and observed value. When input information increases, fewer input neurons tend to be activated. In the maximum state, only one neuron is on, and all others are off. We applied the method to two data sets, namely, the Senate and voting attitude data sets. In both, experimental results confirmed that when input information increased, quantization and topographic errors decreased. In addition, clearer class structure could be extracted by increasing input information. In comparison to our previous methods to detect the importance of input neurons, the present method turned out to be good at producing faithful representations with much more simplified computational procedures.     

Performance enhancement of extreme learning machine for multi-category sparse data classification problems

This paper presents a performance enhancement scheme for the recently developed extreme learning machine (ELM) for multi-category sparse data classification problems. ELM is a single hidden layer neural network with good generalization capabilities and extremely fast learning capacity. In ELM, the input weights are randomly chosen and the output weights are analytically calculated. The generalization performance of the ELM algorithm for sparse data classification problem depends critically on three free parameters. They are, the number of hidden neurons, the input weights and the bias values which need to be optimally chosen. Selection of these parameters for the best performance of ELM involves a complex optimization problem.In this paper, we present a new, real-coded genetic algorithm approach called ‘RCGA-ELM’ to select the optimal number of hidden neurons, input weights and bias values which results in better performance. Two new genetic operators called ‘network based operator’ and ‘weight based operator’ are proposed to find a compact network with higher generalization performance. We also present an alternate and less computationally intensive approach called ‘sparse-ELM’. Sparse-ELM searches for the best parameters of ELM using K-fold validation. A multi-class human cancer classification problem using micro-array gene expression data (which is sparse), is used for evaluating the performance of the two schemes. Results indicate that the proposed RCGA-ELM and sparse-ELM significantly improve ELM performance for sparse multi-category classification problems.     

Improving information-theoretic competitive learning by accentuated information maximization

In this paper, we propose a new computational method for information-theoretic competitive learning. We have so far developed information-theoretic methods for competitive learning in which competitive processes can be simulated by maximizing mutual information between input patterns and competitive units. Though the methods have shown good performance, networks have had difficulty in increasing information content, and learning is very slow to attain reasonably high information. To overcome the shortcoming, we introduce the rth power of competitive unit activations used to accentuate actual competitive unit activations. Because of this accentuation, we call the new computational method “accentuated information maximization”. In this method, intermediate values are pushed toward extreme activation values, and we have a high possibility to maximize information content. We applied our method to a vowel–consonant classification problem in which connection weights obtained by our methods were similar to those obtained by standard competitive learning. The second experiment was to discover some features in a dipole problem. In this problem, we showed that as the parameter r increased, less clear representations could be obtained. For the third experiment of economic data analysis, much clearer representations were obtained by our method, compared with those obtained by the standard competitive learning method.     

Bio-inspired feature enhancement network for edge detection

As the basis of mid-level and high-level vision tasks, edge detection has great significance in the field of computer vision. Edge detection methods based on deep learning usually adopt the structure of the encoding-decoding network, among which the deep convolutional neural network is generally adopted in the encoding network, and the decoding network is designed by researchers. In the design of the encoding-decoding network, researchers pay more attention to the design of the decoding network and ignore the influence of the encoding network, which makes the existing edge detection methods have the problems of weak feature extraction ability and insufficient edge information extraction. To improve the existing methods, this work combines the information transmission mechanism of the retina/lateral geniculate nucleus with an edge detection network based on convolutional neural network and proposes a bionic feature enhancement network. It consists of a pre-enhanced network, an encoding network, and a decoding network. By simulating the information transfer mechanism of the retina/lateral geniculate nucleus, we designed the pre-enhanced network to enhance the ability of the encoding network to extract details and local features. Based on the hierarchical structure of the visual pathway and the integrated feature function of the inferior temporal (IT) cortex, we designed a novel feature fusion network as a decoding network. In a feature fusion network, a down-sampling enhancement module is introduced to boost the feature integration ability of the decoding network. Experimental results demonstrate that we achieve state-of-the-art performance on several available datasets.

     

A design and analysis of objective function-based unsupervised neural networks for fuzzy clustering

Fuzzy clustering has played an important role in solving many problems. In this paper, we design an unsupervised neural network model based on a fuzzy objective function, called OFUNN. The learning rule for the OFUNN model is a result of the formal derivation by the gradient descent method of a fuzzy objective function. The performance of the cluster analysis algorithm is often evaluated by counting the number of crisp clustering errors. However, the number of clustering errors alone is not a reliable and consistent measure for the performance of clustering, especially in the case of input data with fuzzy boundaries. We introduce two measures to evaluate the performance of the fuzzy clustering algorithm. The clustering results on three data sets, Iris data and two artificial data sets, are analyzed using the proposed measures. They show that OFUNN is very competitive in terms of speed and accuracy compared to the fuzzy c-means algorithm.     

Topographic map formation by maximizing unconditional entropy: aplausible strategy for “online” unsupervised competitivelearning and nonparametric density estimation

An unsupervised competitive learning rule, called the vectorial boundary adaptation rule (VBAR), is introduced for topographic map formation. Since VBAR is aimed at producing an equiprobable quantization of the input space, it yields a nonparametric model of the input probability density function. Furthermore, since equiprobable quantization is equivalent to unconditional entropy maximization, we argue that this is a plausible strategy for maximizing mutual information (Shannon information rate) in the case of "online" learning. We use mutual information as a tool for comparing the performance of our rule with Kohonen's self-organizing (feature) map algorithm.     

融合节点描述属性信息的网络表示学习算法

为融合节点描述信息提升网络表示学习质量，针对社会网络中节点描述属性信息存在的语义信息分散和不完备性问题，提出一种融合节点描述属性的网络表示（NPA-NRL）学习算法。首先，对属性信息进行独热编码，并引入随机扰动的数据集增强策略解决属性信息不完备问题；然后，将属性编码和结构编码拼接作为深度神经网络输入，实现两方面信息的相互补充制约；最后，设计了基于网络同质性的属性相似性度量函数和基于SkipGram模型的结构相似性度量函数，通过联合训练实现融合语义信息挖掘。在GPLUS、OKLAHOMA和UNC三个真实网络数据集上的实验结果表明，和经典的DeepWalk、TADW（Text-Associated DeepWalk）、UPP-SNE（User Profile Preserving Social Network Embedding）和SNE（Social Network Embedding）算法相比，NPA-NRL算法的链路预测AUC（Area Under Curve of ROC）值平均提升2.75%，节点分类F1值平均提升7.10%。     

Neuro-fuzzy synergism to the intelligent system for edge detection and enhancement

The paper presents a fuzzy neural network system for edge detection and enhancement. The system can both: (a) obtain edges and (b) enhance edges by recovering missing edges and eliminate false edges caused by noise. The research is comprised of three stages, namely, adaptive fuzzification which is employed to fuzzify the input patterns, edge detection by a three-layer feedforward fuzzy neural network, and edge enhancement by a modified Hopfield neural network. The typical sample patterns are first fuzzified. Then they are used to train the proposed fuzzy neural network. After that, the trained network is able to determine the edge elements with eight orientations. Pixels having high edge membership are traced for further processing. Based on constraint satisfaction and the competitive mechanism, interconnections among neurons are determined in the Hopfield neural network. A criterion is provided to find the final stable result that contains the enhanced edge measurement. The proposed neural networks are simulated on a SUN Sparc station. One hundred and twenty-three training samples are well chosen to cover all the edge and non-edge cases and the performance of the system will not be improved by adding more training samples. Test images are degraded by random noise up to 30% of the original images. Compared with standard edge detection operators and enhancement techniques, the proposed system based on the neuro-fuzzy synergism obtains very good results.     

融合注意力机制和上下文信息的微光图像增强

目的 微光图像存在低对比度、噪声伪影和颜色失真等退化问题,造成图像的视觉感受质量较差,同时也导致后续图像识别、分类和检测等任务的精度降低。针对以上问题,提出一种融合注意力机制和上下文信息的微光图像增强方法。方法 为提高运算精度,以U型结构网络为基础构建了一种端到端的微光图像增强网络框架,主要由注意力机制编/解码模块、跨尺度上下文模块和融合模块等组成。由混合注意力块(包括空间注意力和通道注意力)引导主干网络学习,其空间注意力模块用于计算空间位置的权重以学习不同区域的噪声特征,而通道注意力模块根据不同通道的颜色信息计算通道权重,以提升网络的颜色信息重建能力。此外,跨尺度上下文模块用于聚合各阶段网络中的深层和浅层特征,借助融合机制来提高网络的亮度和颜色增强效果。结果 本文方法与现有主流方法进行定量和定性对比实验,结果显示本文方法显著提升了微光图像亮度,并且较好保持了图像颜色一致性,原微光图像较暗区域的噪点显著去除,重建图像的纹理细节清晰。在峰值信噪比(peak signal-to-noise ratio,PSNR)、结构相似性(structural similarity,SSIM)和图像感知...     

Granular neural networks: A study of optimizing allocation of information granularity in input space

In this paper, we develop a granular input space for neural networks, especially for multilayer perceptrons (MLPs). Unlike conventional neural networks, a neural network with granular input is an augmented study on a basis of a well learned numeric neural network. We explore an efficient way of forming granular input variables so that the corresponding granular outputs of the neural network achieve the highest values of the criteria of specificity (and support). When we augment neural networks through distributing information granularities across input variables, the output of a network has different levels of sensitivity on different input variables. Capturing the relationship between input variables and output result becomes of a great help for mining knowledge from the data. And in this way, important features of the data can be easily found. As an essential design asset, information granules are considered in this construct. The quantification of information granules is viewed as levels of granularity which is given by the expert. The detailed optimization procedure of allocation of information granularity is realized by an improved partheno genetic algorithm (IPGA). The proposed algorithm is testified effective by some numeric studies completed for synthetic data and data coming from the machine learning and StatLib repositories. Moreover, the experimental studies offer a deep insight into the specificity of input features.     

A polynomial kernel neural network classifier based on random sampling and information gain

In this paper, we propose a polynomial kernel neural network classifier (PKNNC) based on the random sampling and information gain. Random sampling is used here to generate datasets for the construction of polynomial neurons located in the neural networks, while information gain is used to evaluate the importance of the input variables (viz. dataset features) of each neuron. Both random sampling and information gain stem from the concepts of well-known random forest models. Some traditional neural networks have certain limitations, such as slow convergence speed, easily falling to local optima and difficulty describing the polynomial relation between the input and output. In this regard, a general PKNNC is proposed, and it consists of three parts: the premise, conclusion, and aggregation. The method of designing the PKNNC is summarized as follows. In the premise section, random sampling and information gain are used to obtain multiple subdatasets that are passed to the aggregation part, and the conclusion part uses three types of polynomials. In the aggregation part, the least squares method (LSM) is used to estimate the parameters of polynomials. Furthermore, the particle swarm optimization (PSO) algorithm is exploited here to optimize the PKNNC. The overall optimization of the PKNNC combines structure optimization and parameter optimization. The PKNNC takes advantage of three types of polynomial kernel functions, random sampling techniques and information gain algorithms, which have a good ability to describe the higher-order nonlinear relationships between input and output variables and have high generalization and fast convergence capabilities. To evaluate the effectiveness of the PKNNC, numerical experiments are carried out on two types of data: machine learning data and face data. A comparative study illustrates that the proposed PKNNC leads to better performance than several conventional models.

     

Enhancing and Relaxing Competitive Units for Feature Discovery

In this paper, we propose a new information-theoretic method called enhancement and relaxation to discover main features in input patterns. We have so far shown that competitive learning is a process of mutual information maximization between input patterns and connection weights. However, because mutual information is an average over all input patterns and competitive units, it is not adequate for discovering detailed information on the roles of elements in a network. To extract information on the roles of elements in a networks, we enhance or relax competitive units through the elements. Mutual information should be changed by these processes. The change in information is called enhanced information. The enhanced information can be used to discover features in input patterns, because the information includes detailed information on elements in a network. We applied the method to the symmetry data, the well-known Iris problem and the OECD countries classification. In all cases, we succeeded in extracting the main features in input patterns.     

Partial mutual information based input variable selection for supervised learning approaches to voice activity detection

The paper presents a novel approach for voice activity detection. The main idea behind the presented approach is to use, next to the likelihood ratio of a statistical model-based voice activity detector, a set of informative distinct features in order to, via a supervised learning approach, enhance the detection performance. The statistical model-based voice activity detector, which is chosen based on the comparison to other similar detectors in an earlier work, models the spectral envelope of the signal and we derive the likelihood ratio thereof. Furthermore, the likelihood ratio together with 70 other various features was meticulously analyzed with an input variable selection algorithm based on partial mutual information. The resulting analysis produced a 13 element reduced input vector which when compared to the full input vector did not undermine the detector performance. The evaluation is performed on a speech corpus consisting of recordings made by six different speakers, which were corrupted with three different types of noises and noise levels. In the end, we tested three different supervised learning algorithms for the task, namely, support vector machine, Boost, and artificial neural networks. The experimental analysis was performed by 10-fold cross-validation due to which threshold averaged receiver operating characteristics curves were constructed. Also, the area under the curve score and Matthew's correlation coefficient were calculated for both the three supervised learning classifiers and the statistical model-based voice activity detector. The results showed that the classifier with the reduced input vector significantly outperformed the standalone detector based on the likelihood ratio, and that among the three classifiers, Boost showed the most consistent performance.     

A fuzzy ensemble of parallel polynomial neural networks with information granules formed by fuzzy clustering

In this paper, we introduce a new category of fuzzy models called a fuzzy ensemble of parallel polynomial neural network (FEP²N²), which consist of a series of polynomial neural networks weighted by activation levels of information granules formed with the use of fuzzy clustering. The two underlying design mechanisms of the proposed networks rely on information granules resulting from the use of fuzzy C-means clustering (FCM) and take advantage of polynomial neural networks (PNNs).The resulting model comes in the form of parallel polynomial neural networks. In the design procedure, in order to estimate the optimal values of the coefficients of polynomial neural networks we use a weighted least square estimation algorithm. We incorporate various types of structures as the consequent part of the fuzzy model when using the learning algorithm. Among the diverse structures being available, we consider polynomial neural networks, which exhibit highly nonlinear characteristics when being viewed as local learning models.We use FCM to form information granules and to overcome the high dimensionality problem. We adopt PNNs to find the optimal local models, which can describe the relationship between the input variables and output variable within some local region of the input space.We show that the generalization capabilities as well as the approximation abilities of the proposed model are improved as a result of using polynomial neural networks. The performance of the network is quantified through experimentation in which we use a number of benchmarks already exploited within the realm of fuzzy or neurofuzzy modeling.     

语义分割和HSV色彩空间引导的低光照图像增强

目的 低光照图像增强是图像处理中的基本任务之一。虽然已经提出了各种方法,但它们往往无法在视觉上产生吸引人的结果,这些图像存在细节不清晰、对比度不高和色彩失真等问题,同时也对后续目标检测、语义分割等任务有不利影响。针对上述问题,提出一种语义分割和HSV（hue,saturation and value）色彩空间引导的低光照图像增强方法。方法 首先提出一个迭代图像增强网络,逐步学习低光照图像与增强图像之间像素级的最佳映射,同时为了在增强过程中保留语义信息,引入一个无监督的语义分割网络并计算语义损失,该网络不需要昂贵的分割注释。为了进一步解决色彩失真问题,在训练时利用HSV色彩空间设计HSV损失;为了解决低光照图像增强中出现细节不清晰的问题,设计了空间一致性损失,使增强图像与对应的低光照图像尽可能细节一致。最终,本文的总损失函数由5个损失函数组成。结果 将本文方法与LIME（low-light image enhancement）、RetinexNet（deep retinex decomposition）、EnlightenGAN（deep light enhancement using generative adversarial networks）、Zero-DCE（zero-reference deep curve estimation）和SGZ（semantic-guided zero-shot learning）5种方法进行了比较。在峰值信噪比（peak signal-to noise ratio,PSNR）上,本文方法平均比Zero-DCE（zero-reference deep curve estimation）提高了0.32dB;在自然图像质量评价（natural image quality evaluation,NIQE）方面,本文方法比EnlightenGAN提高了6%。从主观上看,本文方法具有更好的视觉效果。结论 本文所提出的低光照图像增强方法能有效解决细节不清晰、色彩失真等问题,具有一定的应用价值。     

低光照图像增强算法综述

低光照图像增强旨在提高光照不足场景下捕获数据的视觉感知质量以获取更多信息,逐渐成为图像处理领域中的研究热点,在自动驾驶、安防等人工智能相关行业中具有十分广阔的应用前景。传统的低光照图像增强技术往往需要高深的数学技巧以及严格的数学推导,且导出的迭代过程普遍流程复杂,不利于实际应用。随着大规模数据集的相继诞生,基于深度学习的低光照图像增强已经成为当前的主流技术,然而此类技术受限于数据分布,存在性能不稳定、应用场景单一等问题。此外,在低光照环境下的高层视觉任务(如目标检测)对于低光照图像增强技术的发展带来了新的机遇与挑战。本文从3个方面系统地综述了低光照图像增强技术的研究现状。介绍了现有低光照图像数据集,详述了低光照图像增强技术的发展脉络,通过对比低光照图像增强质量与夜间人脸检测精度,进一步对现有低光照增强技术进行了全面评估与分析。基于对上述现状的探讨,结合实际应用,本文指出当前技术的局限性,并对其发展趋势进行预测。