[CUDA] illegal memory access when using CUDA and large max_bin and large dataset about lightgbm HOT 1 OPEN

I am also encountering similar issues when using a large dataset with CUDA. I have verified this behavior in at least 3 different machines. Every time I get similar logs before the Python script or notebook crashes.

In my case, I have a dataset with 11 million Rows and close to 1 GB. I am unsure if large bins are the reason because it crashes even on default settings. Here's my small setup

fixed_params = {
    "objective": "binary",
    "metric": "auc",
    "boosting_type": "gbdt",
    "data_sample_strategy": "bagging",
    "num_iterations": 5000,
    "device_type": "cuda",
    "random_state": 6241,
    "force_row_wise": True,
    "bagging_seed": 113,
    "early_stopping_rounds": 100,
    "verbose": 2,
}
  gbm = lightgbm.train(
      **fixed_params,
      train_pool,
      valid_sets=[valid_pool],
      valid_names=['valid'],
  )

Here's the LGBM log before it crashes

Here are my Env Info

1. Driver Version: 535.104.05 CUDA Version: 12.2
2. lightgbm==4.4.0 but I have verified that this behavior is the same in v4.2.0.
3. T4 GPU on colab with 15 gigs of GPU RAM.

from lightgbm.