adityab / crossq Goto Github PK

Official code release for "CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity"

Home Page: http://aditya.bhatts.org/CrossQ

License: Other

Shell 17.91% Python 82.09%

batch-normalization reinforcement-learning

crossq's Introduction

[🌏 Webpage] [📕 Paper ] [💬 ICLR 2024 OpenReview (top 5% spotlight)]

Official code release for the ICLR 2024 paper 👇

CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity

Bhatt A.*, Palenicek D.*, Belousov B., Argus M., Amiranashvili A., Brox T., Peters J.

Setup

Execute the following commands to set up a conda environment to run experiments

conda create -n crossq python=3.11.5
conda activate crossq
conda install -c nvidia cuda-nvcc=12.3.52

pip install -e .
pip install "jax[cuda12_pip]==0.4.19" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html

Running Experiments

The main entry point for running experiments is train.py. You can configure experiments with the appropriate environment and agent flags. For more info run python train.py --help.

To train with WandB logging, run the following command to train a CrossQ agent on the Humanoid-v4 environment with seed 9, which will log the results to your WandB entity and project:

python train.py -algo crossq -env Humanoid-v4 -seed 9 -wandb_mode 'online' -wandb_entity my_team -wandb_project crossq

To train without WandB logging, run the following command, and in a different terminal run tensorboard --logdir logs to visualize training progress:

python train.py -algo crossq -env Humanoid-v4 -seed 9 -wandb_mode 'disabled'

To train on a cluster, we provide examples of slurm scripts in /slurm to run various experiments, baselines and ablations performed in the paper on a slurm cluster. These configurations are very cluster specific and probably need to be adjusted for your specific cluster. However, they should surve as a starting point.

Citing this Project and the Paper

To cite our paper and/or this repository in publications:

@inproceedings{
  bhatt2024crossq,
  title={CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity},
  author={Aditya Bhatt and Daniel Palenicek and Boris Belousov and Max Argus and Artemij Amiranashvili and Thomas Brox and Jan Peters},
  booktitle={The Twelfth International Conference on Learning Representations},
  year={2024},
  url={https://openreview.net/forum?id=PczQtTsTIX}
}

Acknowledgements

The implementation is built upon code from Stable Baselines JAX.

crossq's People

Contributors

Stargazers

Watchers

Forkers

portal-cornell jankowskichristopher dongtian95

crossq's Issues

Conflicts in environment.yml

Hello,
Could you please check whether installation of your requirements work correctly?
When I was trying to run the code, I got errors with pip resolver stating problems with torch version - most probably it was conflicting with JAX and CUDA. I was able to run your code with the requirements below, but I had to downgrade torch and I am worried that I might not reproduce your results due to this.

My requirements.txt:

absl-py==2.0.0
appdirs==1.4.4
brax==0.0.16
cachetools==5.3.1
certifi==2023.7.22
charset-normalizer==3.3.1
chex==0.1.84
click==8.1.7
cloudpickle==3.0.0
contextlib2==21.6.0
contourpy==1.1.1
cycler==0.12.1
dataclasses==0.6
decorator==5.1.1
distrax==0.1.4
dm-env==1.6
dm-tree==0.1.8
docker-pycreds==0.4.0
etils==1.5.1
Farama-Notifications==0.0.4
filelock==3.12.4
flax==0.7.4
fonttools==4.43.1
fsspec==2023.10.0
gast==0.5.4
gitdb==4.0.11
GitPython==3.1.40
glfw==2.6.2
google-auth==2.23.3
google-auth-oauthlib==1.0.0
grpcio==1.59.0
gym==0.26.2
gym-notices==0.0.8
gymnasium==0.29.1
idna==3.4
imageio==2.34.0
importlib-resources==6.1.0
jax==0.4.19
jaxlib==0.4.19+cuda12.cudnn89
Jinja2==3.1.2
kiwisolver==1.4.5
labmaze==1.0.6
lxml==5.1.0
Markdown==3.5
markdown-it-py==3.0.0
MarkupSafe==2.1.3
matplotlib==3.8.0
mdurl==0.1.2
ml-collections==0.1.1
ml-dtypes==0.3.1
mpmath==1.3.0
msgpack==1.0.7
mujoco==2.3.7
nest-asyncio==1.5.8
networkx==3.2
numpy==1.26.1
nvidia-cublas-cu11==11.10.3.66
nvidia-cublas-cu12==12.3.2.9
nvidia-cuda-cupti-cu12==12.3.52
nvidia-cuda-nvcc-cu12==12.3.52
nvidia-cuda-nvrtc-cu11==11.7.99
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu11==11.7.99
nvidia-cuda-runtime-cu12==12.3.52
nvidia-cudnn-cu11==8.5.0.96
nvidia-cudnn-cu12==8.9.2.26
nvidia-cufft-cu12==11.0.11.19
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.5.3.52
nvidia-cusparse-cu12==12.1.3.153
nvidia-nccl-cu12==2.18.3
nvidia-nvjitlink-cu12==12.3.52
nvidia-nvtx-cu12==12.1.105
oauthlib==3.2.2
opt-einsum==3.3.0
optax==0.1.7
orbax-checkpoint==0.4.1
packaging==23.2
pandas==2.1.1
pathtools==0.1.2
Pillow==10.1.0
protobuf==4.24.4
psutil==5.9.6
pyasn1==0.5.0
pyasn1-modules==0.3.0
Pygments==2.16.1
PyOpenGL==3.1.7
pyparsing==3.1.1
python-dateutil==2.8.2
pytinyrenderer==0.0.14
pytz==2023.3.post1
PyYAML==6.0.1
requests==2.31.0
requests-oauthlib==1.3.1
rich==13.5.2
rlax==0.1.6
rsa==4.9
scipy==1.11.3
sentry-sdk==1.32.0
setproctitle==1.3.3
six==1.16.0
smmap==5.0.1
stable-baselines3==2.1.0
sympy==1.12
tensorboard==2.14.0
tensorboard-data-server==0.7.2
tensorboardX==2.6.2.2
tensorflow-probability==0.21.0
tensorstore==0.1.46
tfp-nightly==0.20.0.dev20230524
toolz==0.12.0
torch==1.13.1
tqdm==4.66.1
trimesh==4.1.4
triton==2.1.0
typing_extensions==4.5.0
tzdata==2023.3
urllib3==2.0.7
wandb==0.15.10
Werkzeug==3.0.0
zipp==3.17.0

JAX allocating too much memory

Hello,
In your code you correctly specified:

os.environ['XLA_PYTHON_CLIENT_PREALLOCATE'] = 'false'

However currently this line of code is after import jax therefore is not working. I fixed this in #5, so if you would like, you can fix this by merging the pr. I provided more information in #5.

Cannot reproduce paper's results

Hello,
Could you please provide more detailed instructions in README to reproduce your results?
I ran your code (had to slightly change requirements due to conflicts, see: #2), but I cannot reproduce your paper's results. I am running HalfCheetah-v4 environment and the agent is training perfectly up to ~100k steps, at this moment the average rewards are around 8k, but suddenly later drop to almost 0 (even negative values). I even ran this on 4 seeds, but this behavior still persists.

The critic's loss raises to ~1e6, q values have large negative values ~ -5000, entropy coefficient starts to rise.
Could you please check with a fresh conda environment that everything in the code is correct and provide a more detailed step by step instructions how to run the agent (current README does not work)?
If you find it helpful I can share my wandb logs.

Improvement in default args

Hello,
I have a tiny suggestion to improve the code. In train.py, the current default algo is SAC, which is a little bit counter intuitive for the CrossQ repository. My suggestion is to change it to crossq in case user forgets to change it when running.

Suggested change:

parser.add_argument("--algo", type=str, required=False, default='crossq', choices=['crossq', 'sac', 'redq', 'droq', 'td3'], help="agent to run")

side note: currently in help there is a typo - "critic activation function" instead of help related to choosing the agent.

nan values in networks

Hello,
When running the code on deepmind/pendulum-swingup the training crashes as the action becomes nan. I attach stack trace below (I added some more logging to catch exactly which part of the agent produces nan action, the original error was later when interacting with the environment, but the cause is here). I believe that more envs share this problem as in my previous runs I also experienced this - happened mostly for dog tasks, but as I was using my custom wrapper instead of shimmy I thought that maybe it had been some problem with my wrapper. Now it happens with shimmy so it is not the case of the wrapper but probably some instabilities (maybe with BatchNorm?).

237 Traceback (most recent call last):
 238   File "/home/src/crossq/train.py", line 264, in <module>
 239     model.learn(total_timesteps=total_timesteps, progress_bar=True, callback=callback_list)
 240   File "/home/src/crossq/sbx/sac/sac.py", line 187, in learn
 241     return super().learn(
 242            ^^^^^^^^^^^^^^
 243   File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/off_policy_algorithm.py", line 312, in learn
 244     rollout = self.collect_rollouts(
 245               ^^^^^^^^^^^^^^^^^^^^^^
 246   File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/off_policy_algorithm.py", line 541, in collect_rollouts
 247     actions, buffer_actions = self._sample_action(learning_starts, action_noise, env.num_envs)
 248                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 249   File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/off_policy_algorithm.py", line 373, in _sample_action
 250     unscaled_action, _ = self.predict(self._last_obs, deterministic=False)
 251                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 252   File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/base_class.py", line 555, in predict
 253     return self.policy.predict(observation, state, episode_start, deterministic)
 254            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 255   File "/home/src/crossq/sbx/common/policies.py", line 64, in predict
 256     actions = self._predict(observation, deterministic=deterministic)
 257               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 258   File "/home/src/crossq/sbx/sac/policies.py", line 482, in _predict
 259     self.debug_log_action(observation, action, "_predict")
 260   File "/home/src/crossq/sbx/sac/policies.py", line 531, in debug_log_action
 261     raise ValueError("Action is None")
 262 ValueError: Action is None

When the error happens I added printing the state of the actor and the observation. nan values are mostly present in BatchRenorm:

Observations:  [[-0.98452299 -0.17525546 -0.13700339]]
Actor state:  ActorTrainState(step=Array(72082, dtype=int32, weak_type=True), apply_fn=<bound method Module.apply of Actor(
    # attributes
    net_arch = [256, 256]
    action_dim = 1
    batch_norm_momentum = 0.99
    log_std_min = -20
    log_std_max = 2
    use_batch_norm = True
    bn_mode = 'brn_actor'
)>, params={'BatchRenorm_0': {'bias': Array([nan, nan, nan], dtype=float32), 'scale': Array([nan, nan, nan], dtype=float32)}, 'BatchRenorm_1': {'bias': Array([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan], dtype=float32), 'scale': Array([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan], dtype=float32)}, 'BatchRenorm_2': {'bias': Array([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan], dtype=float32), 'scale': Array([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan], dtype=float32)}, 'Dense_0': {'bias': Array([        nan,         nan, -0.72661287,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan, -0.7797777 ,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan, -0.81983244,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
       -0.65440995,         nan, -0.5785902 ,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan, -0.3541636 ,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan,         nan,         nan,         nan,         nan,
               nan], dtype=float32), 'kernel': Array([[        nan,         nan, -0.14071447,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,  0.22611286,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan, -0.01891099,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
         0.11054939,         nan,  0.04355304,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan, -0.9748605 ,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan],
       [        nan,         nan,  0.07693207,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,  0.05971847,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan, -0.07279737,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
         0.06947228,         nan,  0.04604982,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan, -0.05010498,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan],
       [        nan,         nan, -0.03049578,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan, -0.00323228,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,  0.0017229 ,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
        -0.02165468,         nan, -0.02158494,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,  0.00452778,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan,         nan,         nan,         nan,         nan,
                nan]], dtype=float32)}, 'Dense_1': {'bias': Array([-3.55079502e-01, -2.61681288e-01, -3.62839073e-01, -1.15681604e-01,
       -5.08833826e-01, -1.33646116e-01,             nan, -1.64892986e-01,
       -9.87671614e-02, -3.30210567e-01,  6.11294284e-02, -2.26123795e-01,
       -2.73534119e-01, -3.34397793e-01,             nan, -8.75263959e-02,
       -1.58562064e-01, -3.51377517e-01, -1.74645379e-01, -8.94286670e-03,
       -1.91893145e-01, -1.28213629e-01,  2.03128159e-02, -2.56696284e-01,
       -1.50657192e-01, -3.45063061e-01, -2.13366076e-01, -1.69571996e-01,
       -3.34517241e-01, -3.00842196e-01, -1.06576160e-01, -1.35408074e-01,
       -6.20634668e-02, -5.48866615e-02, -2.52332807e-01, -1.78462148e-01,
       -2.34845892e-01, -1.56766266e-01, -4.78359222e-01, -1.16198920e-01,
       -1.25731722e-01, -2.61006474e-01,             nan, -6.05887733e-02,
       -2.15052500e-01,             nan, -1.48657292e-01, -3.27274710e-01,
        1.07243955e-01, -1.11210242e-01, -3.31136845e-02, -5.49518578e-02,
       -2.12549612e-01, -2.13353574e-01, -1.78537995e-01, -2.18994096e-02,
       -7.21647069e-02, -1.74253643e-01, -3.13391834e-01,  2.16715410e-02,
       -1.14866629e-01, -4.00419235e-01, -2.60464311e-01, -3.07593644e-01,
                   nan, -2.45736688e-01, -1.73763752e-01, -6.66186884e-02,
        1.08856119e-01,             nan, -2.16983825e-01,  2.44164586e-01,
                   nan,             nan, -3.26141238e-01, -8.73360708e-02,
       -3.75555217e-01,             nan, -3.91870797e-01, -2.39072606e-01,
       -1.24068327e-01, -4.32559103e-01,  2.30513979e-02,             nan,
       -2.23912150e-01, -2.17534795e-01, -1.92928210e-01, -1.64950922e-01,
                   nan, -3.08977306e-01, -3.68163049e-01,  2.81006261e-03,
       -5.07392526e-01, -1.65657967e-01, -3.15613002e-01, -1.74545765e-01,
       -2.78588176e-01, -4.34532404e-01, -2.61619866e-01, -1.43855408e-01,
                   nan,             nan, -3.72981817e-01, -1.94371045e-01,
        1.83636006e-02, -4.24602851e-02, -8.58307257e-02, -2.71321237e-01,
       -1.97004348e-01, -4.95876729e-01, -4.74496722e-01,             nan,
       -3.33254598e-02, -4.79034781e-01, -2.55109280e-01, -1.87851325e-01,
       -3.39175999e-01, -4.80552763e-01, -4.50025231e-01, -1.03966720e-01,
       -7.74463296e-01, -5.16545363e-02,  1.01213539e-02,             nan,
       -1.24744892e-01, -2.21584707e-01, -2.19108924e-01, -4.01318192e-01,
       -2.04100892e-01, -2.66580433e-01, -7.59028137e-01,             nan,
       -2.50042826e-01, -4.02819782e-01, -2.02461675e-01, -3.39741558e-01,
       -5.28345779e-02, -8.42932388e-02, -1.62568614e-01,  1.58098206e-01,
       -1.10761724e-01,  2.35181837e-03, -3.26542675e-01,             nan,
        5.05282357e-03, -1.25751108e-01,             nan, -3.03706586e-01,
                   nan, -2.41995305e-01, -2.53088415e-01, -2.43461326e-01,
       -2.04102136e-02, -1.84795737e-01, -2.18806162e-01,             nan,
       -2.36812025e-01, -1.80641860e-01, -3.41657400e-01, -3.14457595e-01,
       -2.63056546e-01, -3.97427410e-01, -2.54380584e-01,             nan,
                   nan, -1.74986079e-01, -2.74913579e-01, -1.29359856e-01,
       -3.59678119e-02,  3.35261613e-01, -8.78777653e-02, -5.04442751e-01,
       -4.00152236e-01, -2.42632881e-01,             nan, -4.05929804e-01,
       -2.45563030e-01, -1.88916773e-01, -2.40435839e-01, -1.00784957e-01,
                   nan,             nan, -3.35613132e-01, -1.17802337e-01,
                   nan,             nan, -2.27645561e-01,             nan,
       -3.03044826e-01,  5.99465857e-04, -1.56689212e-01, -1.43252518e-02,
       -1.03414640e-01, -3.61972488e-02, -2.86053956e-01,  1.54133691e-02,
       -2.91877747e-01, -4.44084078e-01, -1.14257067e-01, -3.59545112e-01,
       -1.86518461e-01, -4.90693688e-01, -1.78244472e-01, -4.35604304e-01,
       -1.25659660e-01, -1.01315916e-01, -1.45916626e-01, -2.43625432e-01,
       -1.30847663e-01,             nan, -1.70976147e-01, -1.98871285e-01,
                   nan,  3.85484435e-02, -3.26892465e-01, -2.91502178e-01,
                   nan, -1.78116813e-01, -1.97384760e-01, -2.32053742e-01,
       -2.82236040e-01, -1.08087726e-01, -4.09883052e-01, -5.29915988e-01,
       -3.24332803e-01, -1.00257874e-01,             nan,             nan,
                   nan, -2.54310161e-01,             nan, -5.29110789e-01,
       -2.61053085e-01, -5.08699298e-01, -2.21153900e-01, -5.59086382e-01,
       -2.46261105e-01,             nan,             nan, -3.05840541e-02,
       -2.34860018e-01, -3.22149009e-01, -3.99790168e-01, -2.31906787e-01,
       -3.94329689e-02,  8.35715458e-02, -2.45865479e-01, -4.13744181e-01,
                   nan, -4.18445647e-01,             nan, -3.28062594e-01,
                   nan, -4.72936690e-01, -1.81261748e-01,  1.46970540e-01],      dtype=float32), 'kernel': Array([[ 4.8442278e-02, -2.5370871e-04,  1.9979328e-01, ...,
         1.5068804e-01,  3.6767788e-02,  1.3192339e-01],
       [-6.7832903e-03, -5.0304595e-02, -2.1431591e-01, ...,
        -1.1181718e-01,  2.0614813e-01,  1.2734850e-01],
       [ 1.3255176e-01,  5.4206248e-02,  1.9638033e-01, ...,
        -5.9157098e-03, -1.5652535e-02, -5.7662982e-03],
       ...,
       [ 2.7093706e-03, -4.8780489e-01, -1.7505699e-01, ...,
         7.1161285e-02,  2.8860131e-02,  5.7024822e-02],
       [ 8.8524841e-02, -6.1251257e-02, -2.8650817e-02, ...,
        -9.4492398e-02,  2.4803801e-01,  7.7640779e-02],
       [-1.5573186e-01, -1.6367893e-01, -1.5592015e-01, ...,
        -1.1927266e-01, -2.0962511e-01, -9.1291368e-02]], dtype=float32)}, [skipped many lines]
        
        Action:  [[nan]]

The log is not complete as it has more than 100KB in size, so I attach just the beginning.

Request for Access to Raw Data.

My name is seungju kim, and I am a reinforcement learning researcher.
I recently read your paper. I found your work to be incredibly insightful and highly relevant to my own research.

To further my research, I am particularly interested in the raw data used in your study, specifically the data related to Evaluation Result of Mujoco GYM. Your evaluation protocal seems to be very reasonable and i would follow your evaluation protocal in my paper. As you know, It takes too long time to reproduce your results.

I understand that sharing raw data can sometimes involve additional effort, and I greatly appreciate your consideration of my request. If there are any specific conditions or agreements required for accessing the data, please let me know, and I will be happy to comply.

Some tasks from deepmind/* not working

Hello,
I am trying to benchmark your code on more tasks from deepmind/* but they are not working. There seems to be a bug in the prepare_obs function in sbx/common/policies.py. I attach stack trace below:

Task deepmind/quadruped-run

Traceback (most recent call last):
  File "/home/src/crossq/train.py", line 264, in <module>
    model.learn(total_timesteps=total_timesteps, progress_bar=True, callback=callback_list)
  File "/home/src/crossq/sbx/sac/sac.py", line 187, in learn
    return super().learn(
           ^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/off_policy_algorithm.py", line 312, in learn
    rollout = self.collect_rollouts(
              ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/off_policy_algorithm.py", line 552, in collect_rollouts
    if callback.on_step() is False:
       ^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/callbacks.py", line 104, in on_step
    return self._on_step()
           ^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/callbacks.py", line 208, in _on_step
    continue_training = callback.on_step() and continue_training
                        ^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/callbacks.py", line 104, in on_step
    return self._on_step()
           ^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/callbacks.py", line 208, in _on_step
    continue_training = callback.on_step() and continue_training
                        ^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/callbacks.py", line 104, in on_step
    return self._on_step()
           ^^^^^^^^^^^^^^^
  File "/home/src/crossq/sbx/sac/actor_critic_evaluation_callback.py", line 355, in _on_step
    episode_rewards, episode_lengths = evaluate_policy(
                                       ^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/evaluation.py", line 88, in evaluate_policy
    actions, states = model.predict(
                      ^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/base_class.py", line 555, in predict
    return self.policy.predict(observation, state, episode_start, deterministic)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/src/crossq/sbx/common/policies.py", line 62, in predict
    observation, vectorized_env = self.prepare_obs(observation)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/src/crossq/sbx/common/policies.py", line 95, in prepare_obs
    observation = np.concatenate(
                  ^^^^^^^^^^^^^^^
ValueError: all the input arrays must have same number of dimensions, but the array at index 0 has 2 dimension(s) and the array at index 3 has 1 dimension(s)

Task deepmind/humanoid-walk

Traceback (most recent call last):
  File "/home/src/crossq/train.py", line 264, in <module>
    model.learn(total_timesteps=total_timesteps, progress_bar=True, callback=callback_list)
  File "/home/src/crossq/sbx/sac/sac.py", line 187, in learn
    return super().learn(
           ^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/off_policy_algorithm.py", line 312, in learn
    rollout = self.collect_rollouts(
              ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/off_policy_algorithm.py", line 552, in collect_rollouts
    if callback.on_step() is False:
       ^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/callbacks.py", line 104, in on_step
    return self._on_step()
           ^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/callbacks.py", line 208, in _on_step
    continue_training = callback.on_step() and continue_training
                        ^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/callbacks.py", line 104, in on_step
    return self._on_step()
           ^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/callbacks.py", line 208, in _on_step
    continue_training = callback.on_step() and continue_training
                        ^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/callbacks.py", line 104, in on_step
    return self._on_step()
           ^^^^^^^^^^^^^^^
  File "/home/src/crossq/sbx/sac/actor_critic_evaluation_callback.py", line 355, in _on_step
    episode_rewards, episode_lengths = evaluate_policy(
                                       ^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/evaluation.py", line 88, in evaluate_policy
    actions, states = model.predict(
                      ^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/base_class.py", line 555, in predict
    return self.policy.predict(observation, state, episode_start, deterministic)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/src/crossq/sbx/common/policies.py", line 62, in predict
    observation, vectorized_env = self.prepare_obs(observation)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/src/crossq/sbx/common/policies.py", line 95, in prepare_obs
    observation = np.concatenate(
                  ^^^^^^^^^^^^^^^
ValueError: all the input arrays must have same number of dimensions, but the array at index 0 has 2 dimension(s) and the array at index 2 has 1 dimension(s)

Task deepmind/humanoid-run

Traceback (most recent call last):
  File "/home/src/crossq/train.py", line 264, in <module>
    model.learn(total_timesteps=total_timesteps, progress_bar=True, callback=callback_list)
  File "/home/src/crossq/sbx/sac/sac.py", line 187, in learn
    return super().learn(
           ^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/off_policy_algorithm.py", line 312, in learn
    rollout = self.collect_rollouts(
              ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/off_policy_algorithm.py", line 552, in collect_rollouts
    if callback.on_step() is False:
       ^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/callbacks.py", line 104, in on_step
    return self._on_step()
           ^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/callbacks.py", line 208, in _on_step
    continue_training = callback.on_step() and continue_training
                        ^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/callbacks.py", line 104, in on_step
    return self._on_step()
           ^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/callbacks.py", line 208, in _on_step
    continue_training = callback.on_step() and continue_training
                        ^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/callbacks.py", line 104, in on_step
    return self._on_step()
           ^^^^^^^^^^^^^^^
  File "/home/src/crossq/sbx/sac/actor_critic_evaluation_callback.py", line 355, in _on_step
    episode_rewards, episode_lengths = evaluate_policy(
                                       ^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/evaluation.py", line 88, in evaluate_policy
    actions, states = model.predict(
                      ^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/base_class.py", line 555, in predict
    return self.policy.predict(observation, state, episode_start, deterministic)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/src/crossq/sbx/common/policies.py", line 62, in predict
    observation, vectorized_env = self.prepare_obs(observation)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/src/crossq/sbx/common/policies.py", line 95, in prepare_obs
    observation = np.concatenate(
                  ^^^^^^^^^^^^^^^
ValueError: all the input arrays must have same number of dimensions, but the array at index 0 has 2 dimension(s) and the array at index 2 has 1 dimension(s)

Task deepmind/walker-run

Traceback (most recent call last):
  File "/home/src/crossq/train.py", line 264, in <module>
    model.learn(total_timesteps=total_timesteps, progress_bar=True, callback=callback_list)
  File "/home/src/crossq/sbx/sac/sac.py", line 187, in learn
    return super().learn(
           ^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/off_policy_algorithm.py", line 312, in learn
    rollout = self.collect_rollouts(
              ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/off_policy_algorithm.py", line 552, in collect_rollouts
    if callback.on_step() is False:
       ^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/callbacks.py", line 104, in on_step
    return self._on_step()
           ^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/callbacks.py", line 208, in _on_step
    continue_training = callback.on_step() and continue_training
                        ^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/callbacks.py", line 104, in on_step
    return self._on_step()
           ^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/callbacks.py", line 208, in _on_step
    continue_training = callback.on_step() and continue_training
                        ^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/callbacks.py", line 104, in on_step
    return self._on_step()
           ^^^^^^^^^^^^^^^
  File "/home/src/crossq/sbx/sac/actor_critic_evaluation_callback.py", line 355, in _on_step
    episode_rewards, episode_lengths = evaluate_policy(
                                       ^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/evaluation.py", line 88, in evaluate_policy
    actions, states = model.predict(
                      ^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/crossq/lib/python3.11/site-packages/stable_baselines3/common/base_class.py", line 555, in predict
    return self.policy.predict(observation, state, episode_start, deterministic)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/src/crossq/sbx/common/policies.py", line 62, in predict
    observation, vectorized_env = self.prepare_obs(observation)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/src/crossq/sbx/common/policies.py", line 95, in prepare_obs
    observation = np.concatenate(
                  ^^^^^^^^^^^^^^^
numpy.exceptions.AxisError: axis 1 is out of bounds for array of dimension 1

I believe that this is due to different dict formats of observation returned by shimmy. I once had similar problems in another project and fixed them by using the function from TD-MPC2 GitHub repository:

def _obs_to_array(self, obs):
    return np.concatenate([v.flatten() for v in obs.values()])

Maybe you can try this as well and will work better.

JAX version and installation instructions

Hello,
I am trying to run your code, but there seems to be an issue with installation. The environment file specifies that JAX version is 0.4.19, however last line in README:

python -m pip install --upgrade "jax[cuda12_pip]" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html

upgrades JAX version to 0.4.25 and causes many bugs in code e.g. module 'jax.random' has no attribute 'KeyArray' (but this is the tip of the iceberg, because deleting KeyArray from typing unveils more errors).

Do I correctly assume that this line should not be present in README and the code should work with JAX 0.4.19?
I was able to run the code with JAX 0.4.19.
Could you also check the installation instruction in README that without this last line it works correctly? I probably faced more issues and had to install some packages manually as these lines did not install it (sorry cannot recall now what was exactly the issue, as I cannot reproduce it now due to many steps taken when trying to run your code).