Errors occurred when running training scripts in NeurIPS2019-Learn-to-Move-Challenge about parl HOT 9 CLOSED

I couldn't fix it anyway, using updated training scripts. Here is the detailed configuration of my system:
GPU: TITAN Xp * 4
Driver version: 410.93
CUDA version: 9.0.176
cudnn version: 7.5.1

python==3.6.1
paddlepaddle-gpu==1.5.1.post97
parl==1.2.1
osim-rl==3.0.11

I decided to try it on another machine......

from parl.

what is your system platform? (Windows/Linux/macOS)

from parl.

Ubuntu 16.04

from parl.

It looks like that paddle cannot restore the model normally, but we cannot reproduce the error in Ubuntu 16.04 with paddlepaddle-gpu==1.5.1.

Maybe you can try in another machine or raise an issue in Paddle repo.

We have updated our training scripts, you also can download the latest version and try again.

from parl.

Please try using the updated scripts and feel free to comment here if you have any problem:)

from parl.

Sorry, we fail to reproduce the problem at our machine. Our environment is almost the same as yours except the type of GPU cards.

from parl.

I tried it on another machine (almost everything is same except for a RTX 2080 GPU). A bunch of error message disappeared but it still popped up a LLVM error (shown below):

I1220 17:03:54.295152 6959 parallel_executor.cc:329] The number of CUDAPlace, which is used in ParallelExecutor, is 1. And the Program will be copied 1 copies
I1220 17:03:54.297013 6959 build_strategy.cc:340] SeqOnlyAllReduceOps:0, num_trainers:1
[12-20 17:03:54 MainThread @train.py:313] restore model from ./low_speed_model
LLVM ERROR: out of memory
Aborted (core dumped)

It seemed that there was something wrong when restoring model from ./low_speed_model.
I created a new folder named low_speed_model and then typed sh scripts/train_difficulty1.sh ./low_speed_model . I wonder if I did something mistakenly?

from parl.

As above, low_speed_model is our pre-trained model, you can download it from online storage service.

from parl.

Problem solved! Thanks.

from parl.