Comments (1)
I am also encountering similar issues when using a large dataset with CUDA. I have verified this behavior in at least 3 different machines. Every time I get similar logs before the Python script or notebook crashes.
In my case, I have a dataset with 11 million Rows and close to 1 GB. I am unsure if large bins are the reason because it crashes even on default settings. Here's my small setup
fixed_params = {
"objective": "binary",
"metric": "auc",
"boosting_type": "gbdt",
"data_sample_strategy": "bagging",
"num_iterations": 5000,
"device_type": "cuda",
"random_state": 6241,
"force_row_wise": True,
"bagging_seed": 113,
"early_stopping_rounds": 100,
"verbose": 2,
}
gbm = lightgbm.train(
**fixed_params,
train_pool,
valid_sets=[valid_pool],
valid_names=['valid'],
)
Here's the LGBM log before it crashes
Here are my Env Info
Driver Version: 535.104.05 CUDA Version: 12.2
lightgbm==4.4.0
but I have verified that this behavior is the same inv4.2.0
.- T4 GPU on colab with 15 gigs of GPU RAM.
from lightgbm.
Related Issues (20)
- [python-package] [gpu] GPU install on Windows fails HOT 2
- [python-package] Training data partially available when loading exported lgb.Dataset HOT 9
- [python-package] using lgb.Booster(model_file='model.txt') hangs when loading gbdt regression model HOT 8
- [python-package] How to pass additional attributes to custom eval or loss functions in regression task? HOT 8
- [python-package] ValueError when using scikit-learn API for multiclass binary classification HOT 2
- [RFC] Sync supported CUDA versions with a new support policy for CUDA Container Images HOT 1
- [R-package] CRAN warnings on gcc-14 HOT 1
- [GPU] GPU trainer takes more time than CPU's HOT 5
- [GPU] [performance] LightGBM Running Significantly Slower Than XGBoost on GPU for Classification Task HOT 6
- Extremely slow data loading on high dimensional dataset.
- [R-package], [GPU], Windows10, install error HOT 1
- `runtime error: member call on address 0x612000003ad8 which does not point to an object of type 'std::streambuf'` in macOS cpp_tests CI job HOT 2
- `OSError: [WinError 122] The data area passed to a system call is too small` in Windows bdist CI job HOT 4
- tests failed HOT 5
- [ci] [R-package] clang18 jobs failing: _mm_malloc compilation errors HOT 2
- [GPU][CUDA] Add support for linear tree with device=gpu, device=cuda
- Roadmap link does not work on the CONTRIBUTING.md file HOT 1
- Can't read old models saved with saveRDS.lgb.Booster on version 4.3.0 of lightgbm HOT 3
- After saving the datasets to disk, training with validation on the test set doesn't work (bins mapper issue) HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lightgbm.