tomerronen34 / treeboost_autograd Goto Github PK
View Code? Open in Web Editor NEWEasy Custom Losses for Tree Boosters using Pytorch
License: MIT License
treeboost_autograd's People
Contributors
Stargazers
Watchers
treeboost_autograd's Issues
CatboostRegressor and custom loss error
Hi there,
I am trying to use your tool to create a Spearman R custom loss to use for CatBoostRegressor.
However I get that this error regarding calculate_derivatives
To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
_catboost.pyx in _catboost._ObjectiveCalcDersRange()
/apps/python3/lib/python3.7/site-packages/treeboost_autograd/booster_objectives.py in calc_ders_range(self, preds, targets, weights)
40 ) -> List[Tuple[float, float]]:
---> 41 deriv1, deriv2 = self.calculate_derivatives(preds, targets, weights)
42 result = list(zip(deriv1, deriv2))
AttributeError: 'CatboostObjective' object has no attribute 'calculate_derivatives'
During handling of the above exception, another exception occurred:
RuntimeError Traceback (most recent call last)
_catboost.pyx in _catboost._ObjectiveCalcDersRange()
/apps/python3/lib/python3.7/site-packages/treeboost_autograd/booster_objectives.py in calc_ders_range(self, preds, targets, weights)
40 ) -> List[Tuple[float, float]]:
---> 41 deriv1, deriv2 = self.calculate_derivatives(preds, targets, weights)
42 result = list(zip(deriv1, deriv2))
/apps/python3/lib/python3.7/site-packages/treeboost_autograd/pytorch_objective.py in calculate_derivatives(self, preds, targets, weights)
24 objective = self.sign * self.loss_function(preds, targets)
---> 25 deriv1, deriv2 = self._calculate_derivatives(objective, preds)
26
/apps/python3/lib/python3.7/site-packages/treeboost_autograd/pytorch_objective.py in _calculate_derivatives(objective, preds)
34 def _calculate_derivatives(objective: Tensor, preds: Tensor) -> Tuple[np.ndarray, np.ndarray]:
---> 35 deriv1, = torch.autograd.grad(objective, preds, create_graph=True)
36
/apps/python3/lib/python3.7/site-packages/torch/autograd/__init__.py in grad(outputs, inputs, grad_outputs, retain_graph, create_graph, only_inputs, allow_unused)
227 outputs, grad_outputs_, retain_graph, create_graph,
--> 228 inputs, allow_unused, accumulate_grad=False)
229
RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.
During handling of the above exception, another exception occurred:
RuntimeError Traceback (most recent call last)
_catboost.pyx in _catboost._ObjectiveCalcDersRange()
/apps/python3/lib/python3.7/site-packages/treeboost_autograd/booster_objectives.py in calc_ders_range(self, preds, targets, weights)
40 ) -> List[Tuple[float, float]]:
---> 41 deriv1, deriv2 = self.calculate_derivatives(preds, targets, weights)
42 result = list(zip(deriv1, deriv2))
/apps/python3/lib/python3.7/site-packages/treeboost_autograd/pytorch_objective.py in calculate_derivatives(self, preds, targets, weights)
24 objective = self.sign * self.loss_function(preds, targets)
---> 25 deriv1, deriv2 = self._calculate_derivatives(objective, preds)
26
/apps/python3/lib/python3.7/site-packages/treeboost_autograd/pytorch_objective.py in _calculate_derivatives(objective, preds)
34 def _calculate_derivatives(objective: Tensor, preds: Tensor) -> Tuple[np.ndarray, np.ndarray]:
---> 35 deriv1, = torch.autograd.grad(objective, preds, create_graph=True)
36
/apps/python3/lib/python3.7/site-packages/torch/autograd/__init__.py in grad(outputs, inputs, grad_outputs, retain_graph, create_graph, only_inputs, allow_unused)
227 outputs, grad_outputs_, retain_graph, create_graph,
--> 228 inputs, allow_unused, accumulate_grad=False)
229
RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.
During handling of the above exception, another exception occurred:
RuntimeError Traceback (most recent call last)
_catboost.pyx in _catboost._ObjectiveCalcDersRange()
/apps/python3/lib/python3.7/site-packages/treeboost_autograd/booster_objectives.py in calc_ders_range(self, preds, targets, weights)
40 ) -> List[Tuple[float, float]]:
---> 41 deriv1, deriv2 = self.calculate_derivatives(preds, targets, weights)
42 result = list(zip(deriv1, deriv2))
/apps/python3/lib/python3.7/site-packages/treeboost_autograd/pytorch_objective.py in calculate_derivatives(self, preds, targets, weights)
24 objective = self.sign * self.loss_function(preds, targets)
---> 25 deriv1, deriv2 = self._calculate_derivatives(objective, preds)
26
/apps/python3/lib/python3.7/site-packages/treeboost_autograd/pytorch_objective.py in _calculate_derivatives(objective, preds)
34 def _calculate_derivatives(objective: Tensor, preds: Tensor) -> Tuple[np.ndarray, np.ndarray]:
---> 35 deriv1, = torch.autograd.grad(objective, preds, create_graph=True)
36
/apps/python3/lib/python3.7/site-packages/torch/autograd/__init__.py in grad(outputs, inputs, grad_outputs, retain_graph, create_graph, only_inputs, allow_unused)
227 outputs, grad_outputs_, retain_graph, create_graph,
--> 228 inputs, allow_unused, accumulate_grad=False)
229
RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.
During handling of the above exception, another exception occurred:
RuntimeError Traceback (most recent call last)
_catboost.pyx in _catboost._ObjectiveCalcDersRange()
/apps/python3/lib/python3.7/site-packages/treeboost_autograd/booster_objectives.py in calc_ders_range(self, preds, targets, weights)
40 ) -> List[Tuple[float, float]]:
---> 41 deriv1, deriv2 = self.calculate_derivatives(preds, targets, weights)
42 result = list(zip(deriv1, deriv2))
/apps/python3/lib/python3.7/site-packages/treeboost_autograd/pytorch_objective.py in calculate_derivatives(self, preds, targets, weights)
24 objective = self.sign * self.loss_function(preds, targets)
---> 25 deriv1, deriv2 = self._calculate_derivatives(objective, preds)
26
/apps/python3/lib/python3.7/site-packages/treeboost_autograd/pytorch_objective.py in _calculate_derivatives(objective, preds)
34 def _calculate_derivatives(objective: Tensor, preds: Tensor) -> Tuple[np.ndarray, np.ndarray]:
---> 35 deriv1, = torch.autograd.grad(objective, preds, create_graph=True)
36
/apps/python3/lib/python3.7/site-packages/torch/autograd/__init__.py in grad(outputs, inputs, grad_outputs, retain_graph, create_graph, only_inputs, allow_unused)
227 outputs, grad_outputs_, retain_graph, create_graph,
--> 228 inputs, allow_unused, accumulate_grad=False)
229
RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.
During handling of the above exception, another exception occurred:
RuntimeError Traceback (most recent call last)
_catboost.pyx in _catboost._ObjectiveCalcDersRange()
/apps/python3/lib/python3.7/site-packages/treeboost_autograd/booster_objectives.py in calc_ders_range(self, preds, targets, weights)
40 ) -> List[Tuple[float, float]]:
---> 41 deriv1, deriv2 = self.calculate_derivatives(preds, targets, weights)
42 result = list(zip(deriv1, deriv2))
/apps/python3/lib/python3.7/site-packages/treeboost_autograd/pytorch_objective.py in calculate_derivatives(self, preds, targets, weights)
24 objective = self.sign * self.loss_function(preds, targets)
---> 25 deriv1, deriv2 = self._calculate_derivatives(objective, preds)
26
/apps/python3/lib/python3.7/site-packages/treeboost_autograd/pytorch_objective.py in _calculate_derivatives(objective, preds)
34 def _calculate_derivatives(objective: Tensor, preds: Tensor) -> Tuple[np.ndarray, np.ndarray]:
---> 35 deriv1, = torch.autograd.grad(objective, preds, create_graph=True)
36
/apps/python3/lib/python3.7/site-packages/torch/autograd/__init__.py in grad(outputs, inputs, grad_outputs, retain_graph, create_graph, only_inputs, allow_unused)
227 outputs, grad_outputs_, retain_graph, create_graph,
--> 228 inputs, allow_unused, accumulate_grad=False)
229
RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.
During handling of the above exception, another exception occurred:
RuntimeError Traceback (most recent call last)
_catboost.pyx in _catboost._ObjectiveCalcDersRange()
/apps/python3/lib/python3.7/site-packages/treeboost_autograd/booster_objectives.py in calc_ders_range(self, preds, targets, weights)
40 ) -> List[Tuple[float, float]]:
---> 41 deriv1, deriv2 = self.calculate_derivatives(preds, targets, weights)
42 result = list(zip(deriv1, deriv2))
/apps/python3/lib/python3.7/site-packages/treeboost_autograd/pytorch_objective.py in calculate_derivatives(self, preds, targets, weights)
24 objective = self.sign * self.loss_function(preds, targets)
---> 25 deriv1, deriv2 = self._calculate_derivatives(objective, preds)
26
/apps/python3/lib/python3.7/site-packages/treeboost_autograd/pytorch_objective.py in _calculate_derivatives(objective, preds)
34 def _calculate_derivatives(objective: Tensor, preds: Tensor) -> Tuple[np.ndarray, np.ndarray]:
---> 35 deriv1, = torch.autograd.grad(objective, preds, create_graph=True)
36
/apps/python3/lib/python3.7/site-packages/torch/autograd/__init__.py in grad(outputs, inputs, grad_outputs, retain_graph, create_graph, only_inputs, allow_unused)
227 outputs, grad_outputs_, retain_graph, create_graph,
--> 228 inputs, allow_unused, accumulate_grad=False)
229
RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.
During handling of the above exception, another exception occurred:
RuntimeError Traceback (most recent call last)
_catboost.pyx in _catboost._ObjectiveCalcDersRange()
/apps/python3/lib/python3.7/site-packages/treeboost_autograd/booster_objectives.py in calc_ders_range(self, preds, targets, weights)
40 ) -> List[Tuple[float, float]]:
---> 41 deriv1, deriv2 = self.calculate_derivatives(preds, targets, weights)
42 result = list(zip(deriv1, deriv2))
/apps/python3/lib/python3.7/site-packages/treeboost_autograd/pytorch_objective.py in calculate_derivatives(self, preds, targets, weights)
24 objective = self.sign * self.loss_function(preds, targets)
---> 25 deriv1, deriv2 = self._calculate_derivatives(objective, preds)
26
/apps/python3/lib/python3.7/site-packages/treeboost_autograd/pytorch_objective.py in _calculate_derivatives(objective, preds)
34 def _calculate_derivatives(objective: Tensor, preds: Tensor) -> Tuple[np.ndarray, np.ndarray]:
---> 35 deriv1, = torch.autograd.grad(objective, preds, create_graph=True)
36
/apps/python3/lib/python3.7/site-packages/torch/autograd/__init__.py in grad(outputs, inputs, grad_outputs, retain_graph, create_graph, only_inputs, allow_unused)
227 outputs, grad_outputs_, retain_graph, create_graph,
--> 228 inputs, allow_unused, accumulate_grad=False)
229
RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.
During handling of the above exception, another exception occurred:
CatBoostError Traceback (most recent call last)
/tmp/ipykernel_22901/1472145058.py in <module>
----> 1 catb_model_muse = catbr_reg(data_m, catb_params, custom_objective, custom_eval_metric, plot=True)
/tmp/ipykernel_22901/98406515.py in catbr_reg(data, params, custom_objective, custom_eval_metric, plot)
9
10 xgbr_model = CatBoostRegressor(**params)
---> 11 xgbr_model.fit(X_train, y_train, eval_set=[(X_val, y_val)],plot=True)
12
13 test_score = xgbr_model.score(X_test, y_test)
/apps/python3/lib/python3.7/site-packages/catboost/core.py in fit(self, X, y, cat_features, sample_weight, baseline, use_best_model, eval_set, verbose, logging_level, plot, column_description, verbose_eval, metric_period, silent, early_stopping_rounds, save_snapshot, snapshot_file, snapshot_interval, init_model, callbacks, log_cout, log_cerr)
5591 use_best_model, eval_set, verbose, logging_level, plot, column_description,
5592 verbose_eval, metric_period, silent, early_stopping_rounds,
-> 5593 save_snapshot, snapshot_file, snapshot_interval, init_model, callbacks, log_cout, log_cerr)
5594
5595 def predict(self, data, prediction_type=None, ntree_start=0, ntree_end=0, thread_count=-1, verbose=None, task_type="CPU"):
/apps/python3/lib/python3.7/site-packages/catboost/core.py in _fit(self, X, y, cat_features, text_features, embedding_features, pairs, sample_weight, group_id, group_weight, subgroup_id, pairs_weight, baseline, use_best_model, eval_set, verbose, logging_level, plot, column_description, verbose_eval, metric_period, silent, early_stopping_rounds, save_snapshot, snapshot_file, snapshot_interval, init_model, callbacks, log_cout, log_cerr)
2281 params,
2282 allow_clear_pool,
-> 2283 train_params["init_model"]
2284 )
2285
/apps/python3/lib/python3.7/site-packages/catboost/core.py in _train(self, train_pool, test_pool, params, allow_clear_pool, init_model)
1703
1704 def _train(self, train_pool, test_pool, params, allow_clear_pool, init_model):
-> 1705 self._object._train(train_pool, test_pool, params, allow_clear_pool, init_model._object if init_model else None)
1706 self._set_trained_model_attributes()
1707
_catboost.pyx in _catboost._CatBoost._train()
_catboost.pyx in _catboost._CatBoost._train()
CatBoostError: catboost/python-package/catboost/helpers.cpp:42: Traceback (most recent call last):
File "_catboost.pyx", line 1399, in _catboost._ObjectiveCalcDersRange
File "/apps/python3/lib/python3.7/site-packages/treeboost_autograd/booster_objectives.py", line 41, in calc_ders_range
deriv1, deriv2 = self.calculate_derivatives(preds, targets, weights)
AttributeError: 'CatboostObjective' object has no attribute 'calculate_derivatives'
Code of loss function
import torch
from fast_soft_sort.pytorch_ops import soft_rank
def corrcoef(target, pred):
# np.corrcoef in torch from @mdo
# https://forum.numer.ai/t/custom-loss-functions-for-xgboost-using-pytorch/960
pred_n = pred - pred.mean()
target_n = target - target.mean()
pred_n = pred_n / pred_n.norm()
target_n = target_n / target_n.norm()
return (pred_n * target_n).sum()
def spearman(target, pred, regularization="l2", regularization_strength=1.0):
pred = soft_rank(pred, regularization=regularization, regularization_strength=regularization_strength)
return corrcoef(target, pred / pred.shape[-1])
def spearman_loss(ypred, ytrue):
lenypred = ypred.shape[0]
lenytrue = ytrue.shape[0]
ypred_th = torch.tensor(ypred.reshape(1, lenypred), requires_grad=True)
ytrue_th = torch.tensor(ytrue.reshape(1, lenytrue))
loss = spearman(ytrue_th, ypred_th, regularization_strength=3)
# print(f'Current loss:{loss}')
# calculate gradient and convert to numpy
loss_grads = torch.autograd.grad(loss, ypred_th)[0]
loss_grads = loss_grads.detach().numpy()
# return gradient and ones instead of Hessian diagonal
return loss_grads[0], np.ones(loss_grads.shape)[0]
custom_objective = CatboostObjective(loss_function=spearman_loss)
Loss doesn't decrease in the multiclass case
I've adapted the hinge loss from the example to be usable for multiclass data. In essence, I've transformed targets into a one-hot matrix and reshaped preds to (n_samples, n_classes) to match the one-hot matrix:
import numpy as np
import torch
from sklearn.datasets import load_breast_cancer, load_iris, load_digits
from sklearn.model_selection import train_test_split
from torch import Tensor
import torch.nn.functional as F
from lightgbm import LGBMClassifier
from booster_objectives import LightGbmObjective
def hinge_loss(preds: Tensor, targets: Tensor) -> Tensor:
n_classes = len(np.unique(targets))
n_samples = targets.size()[0]
preds = torch.reshape(preds, (n_samples, -1))
preds = torch.clamp(preds, min=1e-7, max=1.0)
targets = F.one_hot(targets.long(), n_classes).float()
targets = 2 * targets - 1
loss = torch.max(Tensor([0]), 1 - preds * targets) ** 2
loss = loss.sum()
return lossI've used the iris dataset instead and trained the model as usual:
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
custom_objective = LightGbmObjective(loss_function=hinge_loss)
model = LGBMClassifier(objective=custom_objective,
n_estimators=10, random_state=42)
model.fit(X_train, y_train, eval_metric="multi_error",
eval_set=(X_train, y_train), verbose=True)But the error doesn't decrease during training, instead looking like this:
[1] training's multi_error: 1 training's multi_logloss: 34.5388
[2] training's multi_error: 1 training's multi_logloss: 34.5388
[3] training's multi_error: 1 training's multi_logloss: 34.5388
[4] training's multi_error: 1 training's multi_logloss: 34.5388
[5] training's multi_error: 1 training's multi_logloss: 34.5388
[6] training's multi_error: 1 training's multi_logloss: 34.5388
[7] training's multi_error: 1 training's multi_logloss: 34.5388
[8] training's multi_error: 1 training's multi_logloss: 34.5388
[9] training's multi_error: 1 training's multi_logloss: 34.5388
[10] training's multi_error: 1 training's multi_logloss: 34.5388
The binary example works just fine. What may be the cause of this?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.