Current situation
We do not catch errors during optimization.
Desired Situation
- KeyboardInterrupt should abort the optimization and return a result object similar to a normal optimization result, but at current parameter values and with "success" = False. Do we need any other clean-up?
- SystemExit should just be raised. All information about the current status is in the database. A question is if we should dump a pickle file with the current status if no logging is used.
- For other errors, the desired outcome depends on the situation:
- If the user code is correct, and the exception is due to extreme parameter values we want to continue and just return a very large criterion value.
- During debugging, we want to raise them.
Problem of flat spots in invalid regions
Gradient based optimizers will usually reach invalid parameter regions during an "actual step" and then not leave them during the subsequent gradient evaluation. If we implement the above naively, the gradient would be 0 and some optimizers will interpret this as convergence (without doing further checks).
Assuming that the start parameters are valid, a simple way of avoiding this flat spot would be to make the penalty increasing in the euclidean distance to the start parameters. Intuitively, we would tell the optimizer to take a step back to where it started from when hitting invalid parameters (which is what I would do if I were an optimizer).
We can put the start_values in the params DataFrame (if we do not already have them). Then it is really easy to implement.
Proposed Implementation
We would first run one evaluation of the criterion function without error catching to make sure that the error is not due to user code. The internal criterion function used during optimization is then wrapped into a try-except
block.
There is one new entry to general_options
:
{"criterion_exception_penalty": (constant, slope)}
The penalty is:
min(HUGE_FLOAT, constant + slope * np.linalg.norm(params - start_params))
@tobiasraabe This is a bit different than what we talked about today, but I think it is much better and even simpler to implement. Can you do this as part of the decorator PR?