CUDA error: invalid device ordinal when running graphgpt_stage1.sh about graphgpt HOT 1 CLOSED

Excuse me, when I run the first-stage instruction tuning on my machine with two A40 GPUs, the following error occurs:

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
Traceback (most recent call last):
  File "train_mem.py", line 13, in <module>
    train()
  File "/home/qinghua_mao/work/GraphGPT/graphgpt/train/train_graph.py", line 763, in train
    model_args, data_args, training_args = parser.parse_args_into_dataclasses()
  File "/home/qinghua_mao/lib/anaconda3/envs/graphgpt/lib/python3.8/site-packages/transformers/hf_argparser.py", line 338, in parse_args_into_dataclasses
    obj = dtype(**inputs)
  File "<string>", line 137, in __init__
  File "/home/qinghua_mao/lib/anaconda3/envs/graphgpt/lib/python3.8/site-packages/transformers/training_args.py", line 1551, in __post_init__
    and (self.device.type != "cuda")
  File "/home/qinghua_mao/lib/anaconda3/envs/graphgpt/lib/python3.8/site-packages/transformers/training_args.py", line 2027, in device
    return self._setup_devices
  File "/home/qinghua_mao/lib/anaconda3/envs/graphgpt/lib/python3.8/site-packages/transformers/utils/generic.py", line 63, in __get__
    cached = self.fget(obj)
  File "/home/qinghua_mao/lib/anaconda3/envs/graphgpt/lib/python3.8/site-packages/transformers/training_args.py", line 1963, in _setup_devices
    self.distributed_state = PartialState(
  File "/home/qinghua_mao/lib/anaconda3/envs/graphgpt/lib/python3.8/site-packages/accelerate/state.py", line 240, in __init__
    torch.cuda.set_device(self.device)
  File "/home/qinghua_mao/lib/anaconda3/envs/graphgpt/lib/python3.8/site-packages/torch/cuda/__init__.py", line 404, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: CUDA error: invalid device ordinal
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

You are using a model of type llama to instantiate a model of type GraphLlama. This is not supported for all configurations of models and can yield errors.
Traceback (most recent call last):
  File "train_mem.py", line 13, in <module>
    train()
  File "/home/qinghua_mao/work/GraphGPT/graphgpt/train/train_graph.py", line 763, in train
    model_args, data_args, training_args = parser.parse_args_into_dataclasses()
  File "/home/qinghua_mao/lib/anaconda3/envs/graphgpt/lib/python3.8/site-packages/transformers/hf_argparser.py", line 338, in parse_args_into_dataclasses
    obj = dtype(**inputs)
  File "<string>", line 137, in __init__
  File "/home/qinghua_mao/lib/anaconda3/envs/graphgpt/lib/python3.8/site-packages/transformers/training_args.py", line 1551, in __post_init__
    and (self.device.type != "cuda")
  File "/home/qinghua_mao/lib/anaconda3/envs/graphgpt/lib/python3.8/site-packages/transformers/training_args.py", line 2027, in device
    return self._setup_devices
  File "/home/qinghua_mao/lib/anaconda3/envs/graphgpt/lib/python3.8/site-packages/transformers/utils/generic.py", line 63, in __get__
    cached = self.fget(obj)
  File "/home/qinghua_mao/lib/anaconda3/envs/graphgpt/lib/python3.8/site-packages/transformers/training_args.py", line 1963, in _setup_devices
    self.distributed_state = PartialState(
  File "/home/qinghua_mao/lib/anaconda3/envs/graphgpt/lib/python3.8/site-packages/accelerate/state.py", line 240, in __init__
    torch.cuda.set_device(self.device)
  File "/home/qinghua_mao/lib/anaconda3/envs/graphgpt/lib/python3.8/site-packages/torch/cuda/__init__.py", line 404, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: CUDA error: invalid device ordinal
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Traceback (most recent call last):
  File "train_mem.py", line 13, in <module>
    train()
  File "/home/qinghua_mao/work/GraphGPT/graphgpt/train/train_graph.py", line 763, in train
    model_args, data_args, training_args = parser.parse_args_into_dataclasses()
  File "/home/qinghua_mao/lib/anaconda3/envs/graphgpt/lib/python3.8/site-packages/transformers/hf_argparser.py", line 338, in parse_args_into_dataclasses
    obj = dtype(**inputs)
  File "<string>", line 137, in __init__
  File "/home/qinghua_mao/lib/anaconda3/envs/graphgpt/lib/python3.8/site-packages/transformers/training_args.py", line 1551, in __post_init__
    and (self.device.type != "cuda")
  File "/home/qinghua_mao/lib/anaconda3/envs/graphgpt/lib/python3.8/site-packages/transformers/training_args.py", line 2027, in device
    return self._setup_devices
  File "/home/qinghua_mao/lib/anaconda3/envs/graphgpt/lib/python3.8/site-packages/transformers/utils/generic.py", line 63, in __get__
    cached = self.fget(obj)
  File "/home/qinghua_mao/lib/anaconda3/envs/graphgpt/lib/python3.8/site-packages/transformers/training_args.py", line 1963, in _setup_devices
    self.distributed_state = PartialState(
  File "/home/qinghua_mao/lib/anaconda3/envs/graphgpt/lib/python3.8/site-packages/accelerate/state.py", line 240, in __init__
    torch.cuda.set_device(self.device)
  File "/home/qinghua_mao/lib/anaconda3/envs/graphgpt/lib/python3.8/site-packages/torch/cuda/__init__.py", line 404, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: CUDA error: invalid device ordinal
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

You could refer to this issue to deal with it: lm-sys/FastChat#550.

from graphgpt.