Comments (6)
@ndangtt I didn't encounter any issue when I tested it.
Could you send me a minimal working example, please?
from fanova.
Please find the example in the attachment. I called fanova through the PIMP package. You can find detailed information in test/Readme.txt :)
from fanova.
Hi Nguyen,
I encountered the same problem when incorporating the latest fANOVA changes into PIMP.
For that reason I opened #56. When I realized that the fANOVA issues were just caused by having the Parameter input being squeezed on a unit-hypercube I changed the PIMP fANOVA preprocessing code such that the data fANOVA sees are not on a unit-hypercube anymore.
The relevant changes to PIMP are only in the following function:
https://github.com/automl/ParameterImportance/blob/466918ac7ddb8631ba4d3a778d55155db4c7a36f/pimp/evaluator/fanova.py#L43-L69
Other than that @Krxsy changed how the grid is generated. It first is generated on [0, 1] and than for plotting purposes, all values are transformed back into the original parameter ranges. After that she closed #56.
If these latest changes still cause you trouble let me know and we'll see what is going wrong.
Best,
André
from fanova.
Hi André (and Christina),
Thank you! You're right. I missed the latest change in that function. I've also realized that since I didn't use instance features, the _preprocess function was not called. Now the plot is generated!
However, I am a little bit confused now. Why do you no longer give [0,1] data into fanova? If I understand it correctly, other tools in PIMP use the model trained on [0,1] data, so wouldn't it be inconsistent if fanova analysis use the non-normalized one?
I saw that in PIMP you now set config_on_hypercube=False, so the code when config_on_hypercube=True inside visualizer.py is actually not triggered. But suppose that config_on_hypercube=True, and you give [0,1] data to fanova, then in visualizer.py -> generate_marginal, I guess that the grid should be first generated in the range of [0,1] instead of [lower_bound,upper_bound], and after the mean and std are calculated, grid is then transformed back to the original range for plotting. Or perhaps I misunderstood it?
Kind regards,
Nguyen
from fanova.
No problem. Thanks for letting us know :-)
from fanova.
Hi @ndangtt,
internally fANOVA also uses the same ConfigSpace object as PIMP. That means, that in the end everything will still end up on [0, 1] when reading in the X and y matrices and ultimately computing the marginals.
So already putting everything into fANOVA on [0, 1] just meant that there didn't have to be a second transformation step. However this caused issues with how the fANOVA code handles the plotting.
The bug originally showed up, because the grid for prediction was not on [0, 1] as the retransformation step was executed too early. This caused the plots to just be empty, as the Prediction was outside of the plotted area.
The change for PIMP to not input the already transformed data was chosen such that the internal flow of how things are handled in fANOVA doesn't change. As PIMP uses the same ConfigSpace object that fANOVA uses, the transformation and retransformation will result in the correct values. It just now is more straight forward to plot the results in fANOVA.
Best,
André
from fanova.
Related Issues (20)
- Segmentation fault while running the example code with SWIG 4.0 HOT 4
- mislabeled plot being generated using plot_pairwise_marginal() HOT 2
- RuntimeError - ordinal or categorical features cannot be processed HOT 1
- Dealing with categorical string variables HOT 6
- Hyperparameter configuration space is unordered but is used as if is ordered by data columns HOT 1
- update pypi
- improve logging
- Support for Mac OSX? HOT 13
- no module resource when 'import pysmac' HOT 1
- support for hyperparameters of type Constant HOT 7
- fANOVA visualization fails with bools in CategoricalHyperparameter
- pyrfr library crashes when operating on Constant Hyperparameter
- Unit Tests
- Unassigned variable for marginal plot HOT 1
- Segmentation Fault when plotting pairwise plot of Numerical and Categorical hyperparameter HOT 3
- Exception: pickling interactive pairwise-marginal-plots
- create_all_plots error
- Constants not handled good (and "swap"-logic)
- Error with ConfigSpace if hyperparameters are not ordered alphabetically in X matrix HOT 4
- Inverted shape in heatmap matrix visualization HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fanova.