Nearest neighbor estimate of conditional mutual information in feature selection |
| |
| Authors: | Alkiviadis Tsimpiris Ioannis Vlachos Dimitris Kugiumtzis |
| |
| Affiliation: | 1. Faculty of Engineering, Aristotle University of Thessaloniki, Greece;2. School Biological & Health Systems Engineering, Ira Fulton School of Engineering, Arizona State University, Tempe, Arizona, USA;1. Bioengineering and Optoelectronics (ByO) Group, Universidad Politécnica de Madrid, Ctra. Valencia, km. 7, Madrid 28031, Spain;2. Signal Processing and Recognition (SPR) Group, Universidad Nacional de Colombia, km. 7 vía al Magdalena, Manizales, Colombia;1. Department of Computer Science & Engineering, Tezpur University, Napaam, Tezpur 784028, Assam, India;2. Department of Computer Science, University of Colorado at Colorado Springs, CO 80933-7150, USA;1. College of Electronic Information and Control Engineering, Beijing University of Technology, Beijing 100124, China;2. Engineering Research Center of Digital Community, Ministry of Education, Beijing 100124, China;3. Beijing Key Laboratory of Computational Intelligence and Intelligent System, Beijing 100124, China;4. Hospital of Traditional Chinese Medicine, CPUMS, Beijing 100010, China;1. The University of New South Wales, Kensington Sydney 2052, Australia;2. The University of Melbourne, Melbourne, Australia |
| |
| Abstract: | Mutual information (MI) is used in feature selection to evaluate two key-properties of optimal features, the relevance of a feature to the class variable and the redundancy of similar features. Conditional mutual information (CMI), i.e., MI of the candidate feature to the class variable conditioning on the features already selected, is a natural extension of MI but not so far applied due to estimation complications for high dimensional distributions. We propose the nearest neighbor estimate of CMI, appropriate for high-dimensional variables, and build an iterative scheme for sequential feature selection with a termination criterion, called CMINN. We show that CMINN is equivalent to feature selection MI filters, such as mRMR and MaxiMin, in the presence of solely single feature effects, and more appropriate for combined feature effects. We compare CMINN to mRMR and MaxiMin on simulated datasets involving combined effects and confirm the superiority of CMINN in selecting the correct features (indicated also by the termination criterion) and giving best classification accuracy. The application to ten benchmark databases shows that CMINN obtains the same or higher classification accuracy compared to mRMR and MaxiMin at a smaller cardinality of the selected feature subset. |
| |
| Keywords: | |
| 本文献已被 ScienceDirect 等数据库收录！ |
|
