Mutual-information based feature selection considering feature sets rather than single-dimensional features.
The mutual information (MI) among two random variables X and Y, I(X;Y) can be computed from their joint and marginal probability density functions (pdf) fxy, fx and fy. The MI can be expanded to random vectors X and Y, however their pdf estimation becomes much harder.
Take the three types of feature selection methods:
- Forward selection
- Backward elimination
- Exhaustive search
Mutual information based selection methods in the literature traditionally follow the Forward Selection approach by using a variety of scores such as:
- MIM
- MRMR
- JMIM
- etc.
Which consist of different combinations of low-dim MI among the different candidate features and the target, keeping to the trivariate case MI(X,Y;Z) at most and avoiding the hindrance of estimating high-dimensional probability densities.
This repository provides implementations for:
- Mutual Information MI(X;Y)
- Forward selection methods in the literature
- Backward elimination from the methods in the literature
- Exhaustive selection based on the raw MI(X;Y)
- Vectorized versions of MRMR and DISR methods by replacing f(MI(Xi;Y)) -> MI(X;Y)
💥 See next section Shorts for a super brief tutorial on how to use the library.
💥 See subdirectory scripts for more detailed examples.
from vfs import *
from vfs.shorthands import df_iris, MRMR
df, features, targets = df_iris()
# Mutual information between two variables
mi = mi_frame(df)(['F1'], ['F2'])
print(mi)
# Mutual information between two groups of variables (vectors)
mi = mi_frame(df)(['F1','F2'], ['F3','F4', 'F5'])
print(mi)
# Select the two best features according to MRMR (forward), using shorthand
summary, __, __ = MRMR(df, ['F1', 'F2', 'F3', 'F4'], ['F5'], k=2)
print(summary)
# Select the best two features according to JMIM (backward), using default func
__, sel, disc = backward_eliminator(df, ['F1', 'F2', 'F3', 'F4'], ['F5'], k=2, loss=jmim, mi_fun=mi_frame(df))
print(sel)
print(disc)
# Select the best three features by testing all feat combinations
sel, score = exhaustive_searcher(df, ['F1', 'F2', 'F3', 'F4'], ['F5'], k=2, mi_fun=mi_frame(df))
print(sel)
# Select the best feature vector between two candidates
mifun = mi_frame(df)
aa = ['F1', 'F2']
bb = ['F3', 'F4']
best = aa if mifun(aa, ['F5']) > mifun(bb, ['F5']) else bb
-
Battiti, Roberto. 1994. “Using Mutual Information for Selecting Features in Supervised Neural Net Learning.” IEEE Transactions on Neural Networks 5 (4): 537–50.
-
Yang, H, and John Moody. 1999. “Feature Selection Based on Joint Mutual Information.” In Proceedings of International ICSC Symposium on Advances in Intelligent Data Analysis, 1999:22–25. Citeseer.
-
Fleuret, François. 2004. “Fast Binary Feature Selection with Conditional Mutual Information.” Journal of Machine Learning Research 5 (9).
-
Peng, Hanchuan, Fuhui Long, and Chris Ding. 2005. “Feature Selection Based on Mutual Information Criteria of Max-Dependency, Max-Relevance, and Min-Redundancy.” IEEE Transactions on Pattern Analysis and Machine Intelligence 27 (8): 1226–38.
-
Meyer, Patrick E, and Gianluca Bontempi. 2006. “On the Use of Variable Complementarity for Feature Selection in Cancer Classification.” In Workshops on Applications of Evolutionary Computation, 91–102. Springer.
-
Bennasar, Mohamed, Yulia Hicks, and Rossitza Setchi. 2015. “Feature Selection Using Joint Mutual Information Maximisation.” Expert Systems with Applications 42 (22): 8520–32.
-
Bommert, Andrea, Xudong Sun, Bernd Bischl, Jörg Rahnenführer, and Michel Lang. 2020. “Benchmark for Filter Methods for Feature Selection in High-Dimensional Classification Data.” Computational Statistics & Data Analysis 143: 106839.
-
Kursa, Miron B. 2021. “Praznik: High Performance Information-Based Feature Selection.” SoftwareX 16: 100819.