maxpumperla / learning_ray Goto Github PK
View Code? Open in Web Editor NEWNotebooks for the O'Reilly book "Learning Ray"
Home Page: https://maxpumperla.com/learning_ray
License: MIT License
Notebooks for the O'Reilly book "Learning Ray"
Home Page: https://maxpumperla.com/learning_ray
License: MIT License
In ch_03 https://github.com/maxpumperla/learning_ray/blob/main/notebooks/ch_03_core_app.ipynb train_policy_parallel function,
def train_policy_parallel(env, num_episodes=1000, num_simulations=4):
"""Parallel policy training function."""
policy = Policy(env)
simulations = [SimulationActor.remote() for _ in range(num_simulations)]
policy_ref = ray.put(policy)
for _ in range(num_episodes):
experiences = [sim.rollout.remote(policy_ref) for sim in simulations]
while len(experiences) > 0:
finished, experiences = ray.wait(experiences)
for xp in ray.get(finished):
update_policy(policy, xp)
return policy
If i'm not mistaken, it appears that each episode use the initially initialized policy rather than the updated one
ModuleNotFoundError: No module named 'pygame'
details
rllib train -f pendulum.yml
2022-10-18 17:10:38,085 INFO worker.py:1333 -- Connecting to existing Ray cluster at address: 172.24.16.114:6379...
2022-10-18 17:10:38,091 INFO worker.py:1509 -- Connected to Ray cluster. View the dashboard at 127.0.0.1:8265
/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/util/placement_group.py:78: DeprecationWarning: placement_group parameter is deprecated. Use scheduling_strategy=PlacementGroupSchedulingStrategy(...) instead, see the usage at https://docs.ray.io/en/releases-2.0.0/ray-core/package-ref.html#ray-remote.
return bundle_reservation_check.options(
/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/_private/ray_option_utils.py:266: DeprecationWarning: Setting 'object_store_memory' for actors is deprecated since it doesn't actually reserve the required object store memory. Use object spilling that's enabled by default (https://docs.ray.io/en/releases-2.0.0/ray-core/objects/object-spilling.html) instead to bypass the object store memory size limitation.
warnings.warn(
/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/actor.py:637: DeprecationWarning: placement_group parameter is deprecated. Use scheduling_strategy=PlacementGroupSchedulingStrategy(...) instead, see the usage at https://docs.ray.io/en/releases-2.0.0/ray-core/package-ref.html#ray-remote.
return actor_cls._remote(args=args, kwargs=kwargs, **updated_options)
/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/actor.py:637: DeprecationWarning: placement_group_bundle_index parameter is deprecated. Use scheduling_strategy=PlacementGroupSchedulingStrategy(...) instead, see the usage at https://docs.ray.io/en/releases-2.0.0/ray-core/package-ref.html#ray-remote.
return actor_cls._remote(args=args, kwargs=kwargs, **updated_options)
/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/actor.py:637: DeprecationWarning: placement_group_capture_child_tasks parameter is deprecated. Use scheduling_strategy=PlacementGroupSchedulingStrategy(...) instead, see the usage at https://docs.ray.io/en/releases-2.0.0/ray-core/package-ref.html#ray-remote.
return actor_cls._remote(args=args, kwargs=kwargs, **updated_options)
(PPO pid=52305) 2022-10-18 17:10:42,722 INFO algorithm.py:1871 -- Your framework setting is 'tf', meaning you are using static-graph mode. Set framework='tf2' to enable eager execution with tf2.x. You may also then want to set eager_tracing=True in order to reach similar execution speed as with static-graph mode.
(PPO pid=52305) 2022-10-18 17:10:42,722 INFO ppo.py:378 -- In multi-agent mode, policies will be optimized sequentray::PPO.__init__() (pid=52305, ip=172.24.16.114, repr=PPO)
File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/rllib/algorithms/algorithm.py", line 308, in __init__
super().__init__(config=config, logger_creator=logger_creator, **kwargs)
File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/tune/trainable/trainable.py", line 157, in __init__
self.setup(copy.deepcopy(self.config))
File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/rllib/algorithms/algorithm.py", line 443, in setup
raise e.args[0].args[2]
ModuleNotFoundError: No module named 'pygame'
(PPO pid=52305) cls = load(self.entry_point)
(PPO pid=52305) File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/gym/envs/registration.py", line 55, in load
(PPO pid=52305) mod = importlib.import_module(mod_name)
(PPO pid=52305) File "/root/miniconda3/lib/python3.9/importlib/__init__.py", line 127, in import_module
(PPO pid=52305) return _bootstrap._gcd_import(name[level:], package, level)
(PPO pid=52305) File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
(PPO pid=52305) File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
(PPO pid=52305) File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
(PPO pid=52305) File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
(PPO pid=52305) File "<frozen importlib._bootstrap_external>", line 850, in exec_module
(PPO pid=52305) File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
(PPO pid=52305) File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/gym/envs/classic_control/__init__.py", line 1, in <module>
(PPO pid=52305) from gym.envs.classic_control.cartpole import CartPoleEnv
(PPO pid=52305) File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/gym/envs/classic_control/cartpole.py", line 10, in <module>
(PPO pid=52305) import pygame
(PPO pid=52305) ModuleNotFoundError: No module named 'pygame'
(PPO pid=52305)
(PPO pid=52305) During handling of the above exception, another exception occurred:
(PPO pid=52305)
(PPO pid=52305) ray::PPO.__init__() (pid=52305, ip=172.24.16.114, repr=PPO)
(PPO pid=52305) File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/rllib/algorithms/algorithm.py", line 308, in __init__
(PPO pid=52305) super().__init__(config=config, logger_creator=logger_creator, **kwargs)
(PPO pid=52305) File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/tune/trainable/trainable.py", line 157, in __init__
(PPO pid=52305) self.setup(copy.deepcopy(self.config))
(PPO pid=52305) File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/rllib/algorithms/algorithm.py", line 443, in setup
(PPO pid=52305) raise e.args[0].args[2]
(PPO pid=52305) ModuleNotFoundError: No module named 'pygame'
(RolloutWorker pid=52448) 2022-10-18 17:10:46,627 ERROR worker.py:756 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=52448, ip=172.24.16.114, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x7edf6bb79250>)
(RolloutWorker pid=52448) File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/rllib/evaluation/rollout_worker.py", line 490, in __init__
(RolloutWorker pid=52448) self.env = env_creator(copy.deepcopy(self.env_context))
(RolloutWorker pid=52448) File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/rllib/env/utils.py", line 50, in _gym_env_creator
(RolloutWorker pid=52448) return gym.make(env_descriptor, **env_context)
(RolloutWorker pid=52448) File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/gym/envs/registration.py", line 676, in make
== Status ==
Current time: 2022-10-18 17:10:46 (running for 00:00:08.53)
Memory usage on this node: 275.2/502.1 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/64 CPUs, 0/0 GPUs, 0.0/254.8 GiB heap, 0.0/113.19 GiB objects
Result logdir: /root/ray_results/pendulumppo
Number of trials: 1/1 (1 ERROR)
Number of errored trials: 1
+-----------------------------+--------------+-------------------------------------------------------------------------------------------+
| Trial name | # failures | error file |
|-----------------------------+--------------+-------------------------------------------------------------------------------------------|
| PPO_Pendulum-v1_bb4a6_00000 | 1 | /root/ray_results/pendulumppo/PPO_Pendulum-v1_bb4a6_00000_0_2022-10-18_17-10-38/error.txt |
+-----------------------------+--------------+-------------------------------------------------------------------------------------------+
(RolloutWorker pid=52448) return registry.make(id, **kwargs)
(RolloutWorker pid=52448) File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/gym/envs/registration.py", line 520, in make
(RolloutWorker pid=52448) return spec.make(**kwargs)
(RolloutWorker pid=52448) File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/gym/envs/registration.py", line 139, in make
(RolloutWorker pid=52448) cls = load(self.entry_point)
(RolloutWorker pid=52448) File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/gym/envs/registration.py", line 55, in load
(RolloutWorker pid=52448) mod = importlib.import_module(mod_name)
(RolloutWorker pid=52448) File "/root/miniconda3/lib/python3.9/importlib/__init__.py", line 127, in import_module
(RolloutWorker pid=52448) return _bootstrap._gcd_import(name[level:], package, level)
(RolloutWorker pid=52448) File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
(RolloutWorker pid=52448) File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
(RolloutWorker pid=52448) File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
(RolloutWorker pid=52448) File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
(RolloutWorker pid=52448) File "<frozen importlib._bootstrap_external>", line 850, in exec_module
(RolloutWorker pid=52448) File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
(RolloutWorker pid=52448) File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/gym/envs/classic_control/__init__.py", line 1, in <module>
(RolloutWorker pid=52448) from gym.envs.classic_control.cartpole import CartPoleEnv
(RolloutWorker pid=52448) File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/gym/envs/classic_control/cartpole.py", line 10, in <module>
(RolloutWorker pid=52448) import pygame
(RolloutWorker pid=52448) ModuleNotFoundError: No module named 'pygame'
(RolloutWorker pid=52447) 2022-10-18 17:10:46,644 ERROR worker.py:756 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=52447, ip=172.24.16.114, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x7fcf394a5280>)
(RolloutWorker pid=52447) File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/rllib/evaluation/rollout_worker.py", line 490, in __init__
(RolloutWorker pid=52447) self.env = env_creator(copy.deepcopy(self.env_context))
(RolloutWorker pid=52447) File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/rllib/env/utils.py", line 50, in _gym_env_creator
(RolloutWorker pid=52447) return gym.make(env_descriptor, **env_context)
(RolloutWorker pid=52447) File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/gym/envs/registration.py", line 676, in make
(RolloutWorker pid=52447) return registry.make(id, **kwargs)
(RolloutWorker pid=52447) File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/gym/envs/registration.py", line 520, in make
(RolloutWorker pid=52447) return spec.make(**kwargs)
(RolloutWorker pid=52447) File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/gym/envs/registration.py", line 139, in make
(RolloutWorker pid=52447) cls = load(self.entry_point)
(RolloutWorker pid=52447) File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/gym/envs/registration.py", line 55, in load
(RolloutWorker pid=52447) mod = importlib.import_module(mod_name)
(RolloutWorker pid=52447) File "/root/miniconda3/lib/python3.9/importlib/__init__.py", line 127, in import_module
(RolloutWorker pid=52447) return _bootstrap._gcd_import(name[level:], package, level)
(RolloutWorker pid=52447) File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
(RolloutWorker pid=52447) File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
(RolloutWorker pid=52447) File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
(RolloutWorker pid=52447) File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
(RolloutWorker pid=52447) File "<frozen importlib._bootstrap_external>", line 850, in exec_module
(RolloutWorker pid=52447) File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
(RolloutWorker pid=52447) File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/gym/envs/classic_control/__init__.py", line 1, in <module>
(RolloutWorker pid=52447) from gym.envs.classic_control.cartpole import CartPoleEnv
(RolloutWorker pid=52447) File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/gym/envs/classic_control/cartpole.py", line 10, in <module>
(RolloutWorker pid=52447) import pygame
(RolloutWorker pid=52447) ModuleNotFoundError: No module named 'pygame'
== Status ==
Current time: 2022-10-18 17:10:46 (running for 00:00:08.53)
Memory usage on this node: 275.2/502.1 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/64 CPUs, 0/0 GPUs, 0.0/254.8 GiB heap, 0.0/113.19 GiB objects
Result logdir: /root/ray_results/pendulumppo
Number of trials: 1/1 (1 ERROR)
Number of errored trials: 1
+-----------------------------+--------------+-------------------------------------------------------------------------------------------+
| Trial name | # failures | error file |
|-----------------------------+--------------+-------------------------------------------------------------------------------------------|
| PPO_Pendulum-v1_bb4a6_00000 | 1 | /root/ray_results/pendulumppo/PPO_Pendulum-v1_bb4a6_00000_0_2022-10-18_17-10-38/error.txt |
+-----------------------------+--------------+-------------------------------------------------------------------------------------------+
2022-10-18 17:10:46,656 ERROR ray_trial_executor.py:103 -- An exception occurred when trying to stop the Ray actor:Traceback (most recent call last):
File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/tune/execution/ray_trial_executor.py", line 94, in _post_stop_cleanup
ray.get(future, timeout=0)
File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
return func(*args, **kwargs)
File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/_private/worker.py", line 2277, in get
raise value
ray.exceptions.RayActorError: The actor died because of an error raised in its creation task, ray::PPO.__init__() (pid=52305, ip=172.24.16.114, repr=PPO)
File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/rllib/evaluation/worker_set.py", line 125, in __init__
self.add_workers(
File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/rllib/evaluation/worker_set.py", line 391, in foreach_worker
remote_results = ray.get([w.apply.remote(func) for w in self.remote_workers()])
ray.exceptions.RayActorError: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=52448, ip=172.24.16.114, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x7edf6bb79250>)
File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/rllib/evaluation/rollout_worker.py", line 490, in __init__
self.env = env_creator(copy.deepcopy(self.env_context))
File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/rllib/env/utils.py", line 50, in _gym_env_creator
return gym.make(env_descriptor, **env_context)
File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/gym/envs/registration.py", line 676, in make
return registry.make(id, **kwargs)
File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/gym/envs/registration.py", line 520, in make
return spec.make(**kwargs)
File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/gym/envs/registration.py", line 139, in make
cls = load(self.entry_point)
File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/gym/envs/registration.py", line 55, in load
mod = importlib.import_module(mod_name)
File "/root/miniconda3/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/gym/envs/classic_control/__init__.py", line 1, in <module>
from gym.envs.classic_control.cartpole import CartPoleEnv
File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/gym/envs/classic_control/cartpole.py", line 10, in <module>
import pygame
ModuleNotFoundError: No module named 'pygame'
During handling of the above exception, another exception occurred:
ray::PPO.__init__() (pid=52305, ip=172.24.16.114, repr=PPO)
File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/rllib/algorithms/algorithm.py", line 308, in __init__
super().__init__(config=config, logger_creator=logger_creator, **kwargs)
File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/tune/trainable/trainable.py", line 157, in __init__
self.setup(copy.deepcopy(self.config))
File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/rllib/algorithms/algorithm.py", line 443, in setup
raise e.args[0].args[2]
ModuleNotFoundError: No module named 'pygame'
Traceback (most recent call last):
File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/bin/rllib", line 8, in <module>
sys.exit(cli())
File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/rllib/scripts.py", line 41, in cli
train.run(options, train_parser)
File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/rllib/train.py", line 269, in run
run_experiments(
File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/tune/tune.py", line 848, in run_experiments
return run(
File "/root/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/tune/tune.py", line 752, in run
raise TuneError("Trials did not complete", incomplete_trials)
ray.tune.error.TuneError: ('Trials did not complete', [PPO_Pendulum-v1_bb4a6_00000])
As I start to learning ray, when I start to Chapter 4 that it's seem I can't run the gym as it's not supported.
Hello @maxpumperla ,
Thank you for the amazing book, it's been a real pleasure working through it so far!
I am, however, having a problem with evaluating the maze game using RLlib in chapter 4. When I get to the command
rllib evaluate ~/ray_results/maze_env/ --algo DQN --env maze_gym_env.Environment --steps 100
I get the following error:
ValueError: Env must be of one of the following supported types: BaseEnv, gym.Env, MultiAgentEnv, VectorEnv,
RemoteBaseEnv, ExternalMultiAgentEnv, ExternalEnv, but instead is of type class <'maze_gym_env.Environment'>.
The above error has been found in your environment! We've added a module for checking your custom environments. It may
cause your experiment to fail if your environment is not set up correctly. You can disable this behavior by setting
'disable_env_checking=True' in your environment config dictionary. You can run the environment checking module
standalone by calling ray.rllib.utils.check_env([env]).
I'm don't understand why I am getting this error as I followed your instructions to the letter. It's also not clear where the config dictionary is supposed to be stored.
I was wondering if you could help me resolve this issue?
Thank you!
Who am I?
I read early release of Learning Ray. I love your book, and I am looking forward to sharing it with others.
May I contribute chinese transilation to this awsome book?
NameError: name 'Scaler' is not defined
NameError: name 'Repartitioner' is not defined
NameError: name 'Chain' is not defined
Hi, thank you for making this book available before publishing. It is a treasure for learning ray. I have a doubt about a function in Chapter 3 at section "building a distributed Ray Training" as copied below. This function synchronously updates the policy, which is in essence a Q table, with 4 tasks each of which has an experience list. I wonder if two tasks are updating the same Q value, would this cause a conflict?
def update_policy_task(policy_ref, experiences_list):
"""Remote Ray task for updating a policy with experiences in parallel."""
[update_policy(policy_ref, ray.get(xp)) for xp in experiences_list]
return policy_ref
If the policy is a neural network, do we need to lock the weight for each update? Because the fact that each task updates the weights independently might lead to the problem that one task updates the weights on the ones that have been updated by another task, not the weights the gradients were calculated on. This should have already been dealt with in the Ray Train, but I do not know how it is handled. This problem has been haunting in my mind for a long while. Could you please share your thoughts about it? Thank you very much. I am looking forward to your reply.
In [267]: from ray.rllib.models.preprocessors import get_preprocessor
...: env = GymEnvironment()
...: obs_space = env.observation_space
...: preprocessor = get_preprocessor(obs_space)(obs_space)
...:
...: observations = env.reset()
...: transformed = preprocessor.transform(observations).reshape(1, -1)
...:
...: model_output, _ = model.from_batch({"obs": transformed})
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Input In [267], in <cell line: 9>()
6 observations = env.reset()
7 transformed = preprocessor.transform(observations).reshape(1, -1)
----> 9 model_output, _ = model.from_batch({"obs": transformed})
File ~/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/rllib/utils/deprecation.py:122, in Deprecated.<locals>._inner.<locals>._ctor(*args, **kwargs)
115 deprecation_warning(
116 old=old or obj.__name__,
117 new=new,
118 help=help,
119 error=error,
120 )
121 # Call the deprecated method/function.
--> 122 return obj(*args, **kwargs)
File ~/.cache/pypoetry/virtualenvs/ray-KeD1w6JO-py3.9/lib/python3.9/site-packages/ray/rllib/models/modelv2.py:357, in ModelV2.from_batch(self, train_batch, is_training)
350 """Convenience function that calls this model with a tensor batch.
351
352 All this does is unpack the tensor batch to call this model with the
353 right input dict, state, and seq len arguments.
354 """
356 input_dict = train_batch.copy()
--> 357 input_dict.set_training(is_training)
358 states = []
359 i = 0
AttributeError: 'dict' object has no attribute 'set_training'
ValueError: Cannot evaluate w/o an evaluation worker set in the Trainer or w/o an env on the local worker!
Try one of the following:
1) Set `evaluation_interval` >= 0 to force creating a separate evaluation worker set.
2) Set `create_env_on_driver=True` to force the local (non-eval) worker to have an environment to evaluate on.
I'm going through this book and running the code as I go. I am running into the following issue:
ALL DONE Exception ignored in: <function RolloutWorker.__del__ at 0x7fb2260368c0> Traceback (most recent call last): File "/home/rustykettle/.local/lib/python3.10/site-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span TypeError: 'NoneType' object is not callable
The ALL DONE is my print:
ray.init()
gymEnv = GymEnvironment()
trainer = DQNTrainer(env=GymEnvironment, config={"num_workers": 4}) # <1>
config = trainer.get_config() # <2>
#print(pretty_print(config))
for i in range(10):
result = trainer.train() # <3>
ray.shutdown()
print(pretty_print(result)) # <4>
print("ALL DONE")
It is to show that the exception is something to do with Ray, but it also might have something to do with my configuration. I followed the book very closely, but I'm not discounting a user-error on my part.
output.log
I'm also getting depreciation warnings that don't appear in the log. I'm imagining that this happens because it is happening on a different thread.
I'm reading the book here https://learning.oreilly.com/library/view/learning-ray/9781098117214/ch01.html#section_data_processing and I was trying to run rllib with the provided pendulum.yml
and got an error containing this line:
gym.error.Error: Attempted to look up malformed environment ID: b'Pendulumv0'. (Currently all IDs must be of the form ^(?:[\w:-]+\/)?([\w:.-]+)-v(\d+)$.)
The text mentions Pendulum-v0
, but the yaml example says it's Pendulumv0
, judging by the error Pendulum-v0
is correct.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.