Comments (8)
Hi @bpkroth and @JosuaCarl,
I've been doing quite a bit of work to refactor ConfigSpace to get rid of Cython while maintaining the speed and reducing technical debt. Extending ConfigSpace has been less than easy and fixing issues neigh impossible.
I'll make sure to add these relations to the documentation. Is there anything else you'd like to see? Please feel free to raise an issue!
Please see this PR.
from configspace.
Thanks @eddiebergman that's very exciting to hear. I wonder if there's need/interest for more help in maintainership on some of this stuff going forward?
@bpkroth Amazing, there definitely would be, however I think this would be after the PR from #346 is done.
I feel like the library will be in much better condition for contribution and we would gladly accept help!
One other thought I'd had related to Forbidden Relations and/or Conditional Relations (which I think don't quite exist yet?): right now there's only basic comparison support. What about adding basic scalar operations to those comparison as well? For instance:
# constraint: buffer_size_mb < vm_size_gb * 1024
ForbiddenLessThanRelation(target_relation="buffer_size_mb", constraint_expr="vm_ram_gb * 1024")
# constraint: buffer_size_1 + buffer_size_2 < vm_size
ForbiddenGreaterThanRelation(target_relation="vm_size", constraint_expr="buffer_size_1 + buffer_size_2")
Happy to open a new Issue on this if you'd prefer.
Yes please do open a new issue, I will add this comment on that issue.
I'm pretty certain we'd like to avoid eval
expressions of relations. I think what would be much more accessible would be functional
forbiddens and conditionals that allow you to perform such operations in a python function.
# Not fleshed out but a rough idea
class MyForbidden(FunctionalForbidden):
def __init__(self, ...): # Not sure what is required in signature.
super.__init__(required_things)
# Set user specific things
# Min required
def is_valid(self, values: dict[str, Any]) -> None:
buffer_size = values.get("buffer_size_mb")
vm_ram_gb = values.get("vm_ram_gb")
if buffer size is None or vm_ram_gb is None:
return True # Forbiddens are valid if their hyperparameters are not active
return buffer_size < vm_size_gb * 1024
# Option to @override other methods so they don't rely on
# converting to the above, i.e. work on vectorized space.
def is_valid_vector(self, ...) -> None:
...
Why it wasn't possible before:
- Probably Cython
- Serialization
- Prevents all sorts of speed optimizations which are important for Bayesian Optimization optimizers (i.e. SMAC) who need to generate thousands of valid random samples and random neighborhood of parameters, quickly, in order to evaluate which configuration would be the next best one to try. This is the original constraint and reason for Cythonizing the code.
- As a side note, I've managed to make 2/3 of the benchmarked functions 3x faster and the other one is just as fast as the compiled Cython code at this point. I still would like to refactor the tests lastly, as they're pretty hard to read.
Why it's possible now:
- Not Cython
- We just accept that serialization will fail if functional conditionals/forbiddens are provided.
- We just accept that if there are user provided functional conditions/forbiddens, this can be a performance bottleneck for fast target functions being optimized.
from configspace.
Thanks, I was about to point out the same thing :) In the meantime, for all the people looking to restrict hyperparameters, based on the value of another hyperparameter: I would recommend reading the docstrings in https://github.com/automl/ConfigSpace/blob/main/ConfigSpace/forbidden.pyx to learn about the ForbiddenLessThanRelation
, ForbiddenEqualsRelation
and ForbiddenGreaterThanRelation
classes.
from configspace.
Thanks @eddiebergman that's very exciting to hear. I wonder if there's need/interest for more help in maintainership on some of this stuff going forward?
from configspace.
One other thought I'd had related to Forbidden Relations and/or Conditional Relations (which I think don't quite exist yet?): right now there's only basic comparison support. What about adding basic scalar operations to those comparison as well? For instance:
# constraint: buffer_size_mb < vm_size_gb * 1024
ForbiddenLessThanRelation(target_relation="buffer_size_mb", constraint_expr="vm_ram_gb * 1024")
# constraint: buffer_size_1 + buffer_size_2 < vm_size
ForbiddenGreaterThanRelation(target_relation="vm_size", constraint_expr="buffer_size_1 + buffer_size_2")
Happy to open a new Issue on this if you'd prefer.
from configspace.
The new documentation lists them in the overview reference: https://automl.github.io/ConfigSpace/latest/reference/forbiddens/
However the only real documentation is still API doc: https://automl.github.io/ConfigSpace/latest/api/ConfigSpace/forbidden/#ConfigSpace.forbidden.ForbiddenLessThanRelation
I hope this is explanatory enough but if not, please feel free to let me know what you'd like to see form documentation of it?
from configspace.
Thanks, at least now it can be found. One thing though, the old documentation can currently no longer be reached. Probably because of the new overall framework, but bold move on a Friday afternoon ;)
from configspace.
Makes my Monday's more exciting ;)
But yes, we can't host the old docs anymore unfortunately as it was a whole different doc setup. However it should be fully backwards compat except for some things mentioned on the homepage
from configspace.
Related Issues (20)
- [Bug] Ordinal Hyperparameter got Key Error when run `self.get_order` in `get_neighbors` HOT 1
- `HyperparameterNotFoundError` prohibits `.get` from functioning as expected HOT 1
- Inconsistent typing on quantization parameter for Float hyperparameters HOT 1
- Feature Request: basic scalar operations on Forbidden Relation constraints HOT 1
- Duplicate forbiddens in ConfigurationSpace HOT 1
- Importing a custom incomplete configuration space? HOT 4
- Read ForbiddenRelations clauses from json HOT 2
- [Feature Request] Support json read for Categorical with tuple entries. HOT 2
- TypeError: Expected float, got numpy.float64 (ConfigSpace)
- TypeError: Expected float, got numpy.float64 HOT 1
- `np.array` on `Configuration` returns names of HPs instead of values HOT 2
- Configure Github Sponsorship options HOT 2
- ConfigSpace is not compatible with numpy 2.0 HOT 4
- Bound for an Integer parameter depending on the value of another parameter? HOT 4
- Cannot create Integer parameter HOT 2
- Cannot create Constant of type bool
- Cannot create range-based hyperparameter with a range containing a single value
- Bug: ForbiddenEqualsRelation does not work HOT 1
- TransformerConstant is only supposed to handle np.number datatypes, but is used by Constant for union[int, float, str].
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from configspace.