fuxiailab / rl4rs Goto Github PK

Thanks a lot for making this code open source. Is there a way to make this package installable via pip instead of using conda and export command? It would be great to have a release of this repo on PYPI for ease of use.

I get the following error while trying to install using the given instructions.

ResolvePackageNotFound:
  - libxml2==2.9.12=h03d6c58_0 -> libgcc-ng[version='>=9.3.0'] -> __glibc[version='>=2.17']
  - libstdcxx-ng==9.3.0=hd4cf53a_17 -> __glibc[version='>=2.17']
  - libgcc-ng==9.3.0=h5101ec6_17 -> __glibc[version='>=2.17']
  - libgfortran-ng==7.5.0=ha8ba4b0_17 -> __glibc[version='>=2.17']

The system specifications are

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.5 LTS"
NAME="Ubuntu"
VERSION="16.04.5 LTS (Xenial Xerus)"

I reproduced run_batch_rl according to the guidelines but the errors are as follows.

`WARNING:tensorflow:From /root/miniconda3/envs/rl4rs/lib/python3.6/site-packages/tensorflow_core/python/ops/rnn_cell_impl.py:575: calling Zeros.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /root/miniconda3/envs/rl4rs/lib/python3.6/site-packages/deepctr/contrib/rnn.py:257: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /mnt/rl4rs_pro/RL4RS/RL4RS/script/rl4rs/nets/dien.py:43: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.

WARNING:tensorflow:From /mnt/rl4rs_pro/RL4RS/RL4RS/script/rl4rs/nets/dien.py:43: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.

WARNING:tensorflow:From /mnt/rl4rs_pro/RL4RS/RL4RS/script/rl4rs/env/base.py:124: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

WARNING:tensorflow:From /mnt/rl4rs_pro/RL4RS/RL4RS/script/rl4rs/env/base.py:125: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From /mnt/rl4rs_pro/RL4RS/RL4RS/script/rl4rs/env/base.py:129: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

/mnt/rl4rs_pro/RL4RS/RL4RS/script/rl4rs/env/slate.py:279: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
complete_states = np.array(samples.get_complete_states())
run_batch_rl.sh: line 82: 180 Killed python -u batchrl_train.py $algo 'dataset_generate' "{'env':'SlateRecEnv-v0','iteminfo_file':'${rl4rs_dataset_dir}/item_info.csv','sample_file':'${rl4rs_dataset_dir}/rl4rs_dataset_a_shuf.csv','model_file':'${rl4rs_output_dir}/simulator_a_dien/model','trial_name':'a_all'}"`

First it seems to be some warnings with the TensorFlow version, my own version is 1.15.0, I checked the environment file that what it need is also 1.15.0. I tried other versions such as 1.14.0 and 2.0.0 but still failed. However actually they are just warnings but not errors, so I don't know if I do have to use another version. And another problem is that finally it reported killed and aborted.

The "Dataset Download(for reproduction):" connection is invalid. Can you provide a valid download link? Thank you very much.

Thanks for this repo! I find the section 4.2 of the paper says that we can know more about data details in Appendix C, and the section 5.1 says that more details about the environment simulation model are shown in Appendix D. However, I can't find any appendix in the paper from this url shown in the repo. Maybe forget to add appendix to the paper? Or where can I find all the appendix? ~~ Thanks again!

Hi!

First and foremost, thanks for your contribution.

I'm using this dataset in my research; however, I'm having troubles to use the dataset after reading the SIGIR paper "RL4RS: A Real-World Dataset for Reinforcement Learning based Recommender System" . I'm hoping you could answer the following questions:

1. Could you please explain me what is the meaning of the a_ and b_ prefixes in the data files? e.g., rl4rs_dataset_a_rl vs rl4rs_dataset_b_rl.
2. Could you please explain me what is the meaning of the _rl and _sl suffixes in the data files? e.g., rl4rs_dataset_a_rl vs rl4rs_dataset_a_sl.
3. Do users have a unique numerical identifiers? I tried doing a .unique() operation on the user_protrait column. However, I got way more unique strings than what is reported in Table 2.
4. Inside the item_feature column, how can I identify the item numerical identifier? The paper says that the ID is inside this column but does not specify its position inside the array.
5. If I want to perform an offline evaluation using a traditional user-rating matrix, can I join those datasets into a single matrix? or, instead, should I keep four different matrices (one for each data file)?
6. Could you please provide or highlight the code that computes the statistics of the dataset?
7. I'm trying to replicate Table 2 at the moment, however, I do not know how to map Slate-SL, Slate-RL, SeqSlate-SL, SeqSlate-RL to the data files.
8. Similar to 7, how can I create the Slate and SeqSlate datasets shown on the same Table?

Thanks in advance!

The original code of Exact-k is written based on python 2, so there are several incompatility issues in code.
for example, in layers.py:

outputs_l = [nest.pack_sequence_as(inputs_l[0], output_ta_flat)
                             for output_ta_flat in zip(*outputs_ta_flat)]

zip(*outputs_ta_flat) returns map object in Python 3.6, even we try to list it like zip(*outputs_ta_flat), the error still exists:

TypeError: 'map' object is not subscriptable
in your source code you tried the same way, i don't think it's a feasible way to reproduce Exact-k MODEL

I'm sorry that there is a program error report and I would like to ask you for advice. When running bash run_modelfree_rl.sh DQN, a connection error occurs. The error message is as follows:

2022-11-15 08:19:12,029 INFO replay_buffer.py:46 -- Estimated max memory usage for replay buffer is 0.4361 GB (100000.0 batches of size 1, 4361 bytes each), available system memory is 201.44095232 GB
2022-11-15 08:19:14,843 INFO tf_policy.py:712 -- Optimizing variable <tf.Variable 'default_policy/fc_1/kernel:0' shape=(256, 64) dtype=float32>
2022-11-15 08:19:14,843 INFO tf_policy.py:712 -- Optimizing variable <tf.Variable 'default_policy/fc_1/bias:0' shape=(64,) dtype=float32>
2022-11-15 08:19:14,843 INFO tf_policy.py:712 -- Optimizing variable <tf.Variable 'default_policy/fc_out/kernel:0' shape=(64, 284) dtype=float32>
2022-11-15 08:19:14,843 INFO tf_policy.py:712 -- Optimizing variable <tf.Variable 'default_policy/fc_out/bias:0' shape=(284,) dtype=float32>
2022-11-15 08:19:14,846 INFO multi_gpu_impl.py:143 -- Training on concatenated sample batches:

{ 'inputs': [ np.ndarray((576, 540), dtype=float32, min=-1.0, max=37.179, mean=-0.169),
np.ndarray((576, 540), dtype=float32, min=-1.0, max=38.907, mean=-0.207),
np.ndarray((576,), dtype=int64, min=1.0, max=283.0, mean=103.844),
np.ndarray((576,), dtype=float32, min=0.0, max=162.121, mean=7.551),
np.ndarray((576,), dtype=bool, min=0.0, max=1.0, mean=0.135),
np.ndarray((576,), dtype=float64, min=1.0, max=1.0, mean=1.0)],
'placeholders': [ <tf.Tensor 'default_policy/obs:0' shape=(?, 540) dtype=float32>,
<tf.Tensor 'default_policy/new_obs:0' shape=(?, 540) dtype=float32>,
<tf.Tensor 'default_policy/action:0' shape=(?,) dtype=int64>,
<tf.Tensor 'default_policy/rewards:0' shape=(?,) dtype=float32>,
<tf.Tensor 'default_policy/dones:0' shape=(?,) dtype=float32>,
<tf.Tensor 'default_policy/weights:0' shape=(?,) dtype=float32>],
'state_inputs': []}

2022-11-15 08:19:14,846 INFO multi_gpu_impl.py:188 -- Divided 576 rollout sequences, each of length 1, among 1 devices.
Traceback (most recent call last):
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/response.py", line 438, in _error_catcher
yield
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/response.py", line 519, in read
data = self._fp.read(amt) if not fp_closed else b""
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/http/client.py", line 463, in read
n = self.readinto(b)
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/http/client.py", line 507, in readinto
n = self.fp.readinto(b)
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/wlxy/.local/lib/python3.6/site-packages/requests/models.py", line 760, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/response.py", line 576, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/response.py", line 541, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/contextlib.py", line 99, in exit
self.gen.throw(type, value, traceback)
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/response.py", line 455, in _error_catcher
raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "modelfree_train.py", line 429, in
result = trainer.train()
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 643, in train
raise e
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 629, in train
result = Trainable.train(self)
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/tune/trainable.py", line 237, in train
result = self.step()
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/rllib/agents/trainer_template.py", line 170, in step
res = next(self.train_exec_impl)
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/util/iter.py", line 756, in next
return next(self.built_iterator)
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/util/iter.py", line 783, in apply_foreach
for item in it:
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/util/iter.py", line 843, in apply_filter
for item in it:
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/util/iter.py", line 843, in apply_filter
for item in it:
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/util/iter.py", line 783, in apply_foreach
for item in it:
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/util/iter.py", line 843, in apply_filter
for item in it:
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/util/iter.py", line 1075, in build_union
item = next(it)
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/util/iter.py", line 756, in next
return next(self.built_iterator)
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/util/iter.py", line 783, in apply_foreach
for item in it:
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/util/iter.py", line 783, in apply_foreach
for item in it:
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/util/iter.py", line 783, in apply_foreach
for item in it:
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/rllib/execution/rollout_ops.py", line 75, in sampler
yield workers.local_worker().sample()
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/rllib/evaluation/rollout_worker.py", line 739, in sample
batches = [self.input_reader.next()]
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 101, in next
batches = [self.get_data()]
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 231, in get_data
item = next(self.rollout_provider)
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 615, in _env_runner
sample_collector=sample_collector,
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 934, in _process_observations
env_id)
File "/home/wlxy/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/rllib/env/base_env.py", line 368, in try_reset
return {_DUMMY_AGENT_ID: self.vector_env.reset_at(env_id)}
File "/home/wlxy/userfolder/RL4RS/rl4rs/utils/rllib_vector_env.py", line 44, in reset_at
self.reset_cache = self.env.reset()
File "/home/wlxy/userfolder/RL4RS/rl4rs/server/httpEnv.py", line 43, in reset
observation = self.client.env_reset(self.instance_id)
File "/home/wlxy/userfolder/RL4RS/rl4rs/server/gymHttpClient.py", line 67, in env_reset
resp = self._post_request(route, None)
File "/home/wlxy/userfolder/RL4RS/rl4rs/server/gymHttpClient.py", line 43, in _post_request
data=json.dumps(data))
File "/home/wlxy/.local/lib/python3.6/site-packages/requests/sessions.py", line 577, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/home/wlxy/.local/lib/python3.6/site-packages/requests/sessions.py", line 529, in request
resp = self.send(prep, **send_kwargs)
File "/home/wlxy/.local/lib/python3.6/site-packages/requests/sessions.py", line 687, in send
r.content
File "/home/wlxy/.local/lib/python3.6/site-packages/requests/models.py", line 838, in content
self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
File "/home/wlxy/.local/lib/python3.6/site-packages/requests/models.py", line 763, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

I would like to ask for your help, thank you very much.

Hi, I run your code and get

Traceback (most recent call last):
File "data_preprocess.py", line 179, in
dataset2tfrecord(config, file, out_file, is_slate=False)
File "data_preprocess.py", line 149, in dataset2tfrecord
samples = feature_construct(tmp, is_slate)
File "data_preprocess.py", line 97, in feature_construct
assert sequence_id - 1 == i
AssertionError

When I run bash run_split.sh, How can I solve it and Why I get it?
Thanks

Dear RL4RS Team,

When I run the nohup python -u rl4rs/server/gymHttpServer.py & bash run_modelfree_rl.sh DQN, it always appears the ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=5000), the same as I run the tutorial.ipynb's last cell. I don't know the reason. Is that related to my local network? I can connect to the Mainland Internet. I also tried using a cloud server, which shows the same error. I wonder to know how to solve it. I'm sharing the complete error information here. Many thanks!!!

/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/autoscaler/_private/cli_logger.py:61: FutureWarning: Not all Ray CLI dependencies were found. In Ray 1.4+, the Ray CLI, autoscaler, and dashboard will only be usable via pip install 'ray[default]'. Please update your install command.
"update your install command.", FutureWarning)
2024-03-11 15:06:36,349 INFO services.py:1247 -- View the Ray dashboard at http://127.0.0.1:8265
2024-03-11 15:06:37,418 INFO trainer.py:706 -- Tip: set framework=tfe or the --eager flag to enable TensorFlow eager execution

{'epoch': 5, 'maxlen': 64, 'batch_size': 64, 'action_size': 284, 'class_num': 2, 'dense_feature_num': 432, 'category_feature_num': 21, 'category_hash_size': 100000, 'seq_num': 2, 'emb_size': 128, 'is_eval': False, 'hidden_units': 128, 'max_steps': 9, 'action_emb_size': 32, 'sample_file': 'simulator/rl4rs_dataset_a_shuf.csv', 'model_file': 'simulator/finetuned/simulator_a_dien/model', 'iteminfo_file': 'raw_data/item_info.csv', 'remote_base': 'http://127.0.0.1:5000', 'trial_name': 'all', 'support_rllib_mask': True, 'env': 'SlateRecEnv-v0'}
rllib_config {'env': 'rllibEnv-v0', 'gamma': 1, 'explore': True, 'exploration_config': {'type': 'SoftQ'}, 'num_gpus': 1, 'num_workers': 2, 'framework': 'tf', 'rollout_fragment_length': 9, 'batch_mode': 'complete_episodes', 'train_batch_size': 576, 'evaluation_interval': 1, 'evaluation_num_episodes': 8192, 'evaluation_config': {'explore': False}, 'log_level': 'INFO', 'use_critic': True, 'use_gae': True, 'lambda': 1.0, 'kl_coeff': 0.2, 'sgd_minibatch_size': 256, 'shuffle_sequences': True, 'num_sgd_iter': 1, 'lr': 0.0001, 'vf_loss_coeff': 0.5, 'clip_param': 0.3, 'vf_clip_param': 500.0, 'kl_target': 0.01}

(pid=41960) 2024-03-11 15:06:38,687 ERROR worker.py:421 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::RolloutWorker.init() (pid=41960, ip=192.168.1.4)
(pid=41960) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/util/connection.py", line 96, in create_connection
(pid=41960) raise err
(pid=41960) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/util/connection.py", line 86, in create_connection
(pid=41960) sock.connect(sa)
(pid=41960) ConnectionRefusedError: [Errno 111] Connection refused
(pid=41960)
(pid=41960) During handling of the above exception, another exception occurred:
(pid=41960)
(pid=41960) ray::RolloutWorker.init() (pid=41960, ip=192.168.1.4)
(pid=41960) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
(pid=41960) chunked=chunked,
(pid=41960) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/connectionpool.py", line 394, in _make_request
(pid=41960) conn.request(method, url, **httplib_request_kw)
(pid=41960) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/connection.py", line 234, in request
(pid=41960) super(HTTPConnection, self).request(method, url, body=body, headers=headers)
(pid=41960) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/http/client.py", line 1287, in request
(pid=41960) self._send_request(method, url, body, headers, encode_chunked)
(pid=41960) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/http/client.py", line 1333, in _send_request
(pid=41960) self.endheaders(body, encode_chunked=encode_chunked)
(pid=41960) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/http/client.py", line 1282, in endheaders
(pid=41960) self._send_output(message_body, encode_chunked=encode_chunked)
(pid=41960) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/http/client.py", line 1042, in _send_output
(pid=41960) self.send(msg)
(pid=41960) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/http/client.py", line 980, in send
(pid=41960) self.connect()
(pid=41960) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/connection.py", line 200, in connect
(pid=41960) conn = self._new_conn()
(pid=41960) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/connection.py", line 182, in _new_conn
(pid=41960) self, "Failed to establish a new connection: %s" % e
(pid=41960) urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f21554e6518>: Failed to establish a new connection: [Errno 111] Connection refused
(pid=41960)
(pid=41960) During handling of the above exception, another exception occurred:
(pid=41960)
(pid=41960) ray::RolloutWorker.init() (pid=41960, ip=192.168.1.4)
(pid=41960) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/requests/adapters.py", line 450, in send
(pid=41960) timeout=timeout
(pid=41960) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
(pid=41960) method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
(pid=41960) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/util/retry.py", line 574, in increment
(pid=41960) raise MaxRetryError(_pool, url, error or ResponseError(cause))
(pid=41960) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /v1/envs/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f21554e6518>: Failed to establish a new connection: [Errno 111] Connection refused',))
(pid=41960)
(pid=41960) During handling of the above exception, another exception occurred:
(pid=41960)
(pid=41960) ray::RolloutWorker.init() (pid=41960, ip=192.168.1.4)
(pid=41960) File "python/ray/_raylet.pyx", line 523, in ray._raylet.execute_task
(pid=41960) File "python/ray/_raylet.pyx", line 530, in ray._raylet.execute_task
(pid=41960) File "python/ray/_raylet.pyx", line 534, in ray._raylet.execute_task
(pid=41960) File "python/ray/_raylet.pyx", line 484, in ray._raylet.execute_task.function_executor
(pid=41960) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/_private/function_manager.py", line 563, in actor_method_executor
(pid=41960) return method(__ray_actor, *args, **kwargs)
(pid=41960) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/rllib/evaluation/rollout_worker.py", line 392, in init
(pid=41960) self.env = env_creator(env_context)
(pid=41960) File "", line 43, in
(pid=41960) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/gym/envs/registration.py", line 184, in make
(pid=41960) return registry.make(id, **kwargs)
(pid=41960) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/gym/envs/registration.py", line 106, in make
(pid=41960) env = spec.make(kwargs)
(pid=41960) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/gym/envs/registration.py", line 76, in make
(pid=41960) env = cls(_kwargs)
(pid=41960) File "/media/kemove/16T/Jupyter/Electronics/rl4rs/server/httpEnv.py", line 12, in init
(pid=41960) self.instance_id = self.client.env_create(env_id, config)
(pid=41960) File "/media/kemove/16T/Jupyter/Electronics/rl4rs/server/gymHttpClient.py", line 55, in env_create
(pid=41960) resp = self._post_request(route, data)
(pid=41960) File "/media/kemove/16T/Jupyter/Electronics/rl4rs/server/gymHttpClient.py", line 43, in _post_request
(pid=41960) data=json.dumps(data))
(pid=41960) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/requests/sessions.py", line 577, in post
(pid=41960) return self.request('POST', url, data=data, json=json, **kwargs)
(pid=41960) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/requests/sessions.py", line 529, in request
(pid=41960) resp = self.send(prep, **send_kwargs)
(pid=41960) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/requests/sessions.py", line 645, in send
(pid=41960) r = adapter.send(request, **kwargs)
(pid=41960) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/requests/adapters.py", line 519, in send
(pid=41960) raise ConnectionError(e, request=request)
(pid=41960) requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /v1/envs/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f21554e6518>: Failed to establish a new connection: [Errno 111] Connection refused',))
(pid=41942) 2024-03-11 15:06:38,686 ERROR worker.py:421 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::RolloutWorker.init() (pid=41942, ip=192.168.1.4)
(pid=41942) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/util/connection.py", line 96, in create_connection
(pid=41942) raise err
(pid=41942) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/util/connection.py", line 86, in create_connection
(pid=41942) sock.connect(sa)
(pid=41942) ConnectionRefusedError: [Errno 111] Connection refused
(pid=41942)
(pid=41942) During handling of the above exception, another exception occurred:
(pid=41942)
(pid=41942) ray::RolloutWorker.init() (pid=41942, ip=192.168.1.4)
(pid=41942) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
(pid=41942) chunked=chunked,
(pid=41942) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/connectionpool.py", line 394, in _make_request
(pid=41942) conn.request(method, url, **httplib_request_kw)
(pid=41942) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/connection.py", line 234, in request
(pid=41942) super(HTTPConnection, self).request(method, url, body=body, headers=headers)
(pid=41942) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/http/client.py", line 1287, in request
(pid=41942) self._send_request(method, url, body, headers, encode_chunked)
(pid=41942) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/http/client.py", line 1333, in _send_request
(pid=41942) self.endheaders(body, encode_chunked=encode_chunked)
(pid=41942) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/http/client.py", line 1282, in endheaders
(pid=41942) self._send_output(message_body, encode_chunked=encode_chunked)
(pid=41942) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/http/client.py", line 1042, in _send_output
(pid=41942) self.send(msg)
(pid=41942) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/http/client.py", line 980, in send
(pid=41942) self.connect()
(pid=41942) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/connection.py", line 200, in connect
(pid=41942) conn = self._new_conn()
(pid=41942) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/connection.py", line 182, in _new_conn
(pid=41942) self, "Failed to establish a new connection: %s" % e
(pid=41942) urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7ef8abef6550>: Failed to establish a new connection: [Errno 111] Connection refused
(pid=41942)
(pid=41942) During handling of the above exception, another exception occurred:
(pid=41942)
(pid=41942) ray::RolloutWorker.init() (pid=41942, ip=192.168.1.4)
(pid=41942) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/requests/adapters.py", line 450, in send
(pid=41942) timeout=timeout
(pid=41942) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
(pid=41942) method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
(pid=41942) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/util/retry.py", line 574, in increment
(pid=41942) raise MaxRetryError(_pool, url, error or ResponseError(cause))
(pid=41942) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /v1/envs/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ef8abef6550>: Failed to establish a new connection: [Errno 111] Connection refused',))
(pid=41942)
(pid=41942) During handling of the above exception, another exception occurred:
(pid=41942)
(pid=41942) ray::RolloutWorker.init() (pid=41942, ip=192.168.1.4)
(pid=41942) File "python/ray/_raylet.pyx", line 523, in ray._raylet.execute_task
(pid=41942) File "python/ray/_raylet.pyx", line 530, in ray._raylet.execute_task
(pid=41942) File "python/ray/_raylet.pyx", line 534, in ray._raylet.execute_task
(pid=41942) File "python/ray/_raylet.pyx", line 484, in ray._raylet.execute_task.function_executor
(pid=41942) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/_private/function_manager.py", line 563, in actor_method_executor
(pid=41942) return method(__ray_actor, *args, **kwargs)
(pid=41942) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/rllib/evaluation/rollout_worker.py", line 392, in init
(pid=41942) self.env = env_creator(env_context)
(pid=41942) File "", line 43, in
(pid=41942) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/gym/envs/registration.py", line 184, in make
(pid=41942) return registry.make(id, **kwargs)
(pid=41942) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/gym/envs/registration.py", line 106, in make
(pid=41942) env = spec.make(kwargs)
(pid=41942) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/gym/envs/registration.py", line 76, in make
(pid=41942) env = cls(_kwargs)
(pid=41942) File "/media/kemove/16T/Jupyter/Electronics/rl4rs/server/httpEnv.py", line 12, in init
(pid=41942) self.instance_id = self.client.env_create(env_id, config)
(pid=41942) File "/media/kemove/16T/Jupyter/Electronics/rl4rs/server/gymHttpClient.py", line 55, in env_create
(pid=41942) resp = self._post_request(route, data)
(pid=41942) File "/media/kemove/16T/Jupyter/Electronics/rl4rs/server/gymHttpClient.py", line 43, in _post_request
(pid=41942) data=json.dumps(data))
(pid=41942) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/requests/sessions.py", line 577, in post
(pid=41942) return self.request('POST', url, data=data, json=json, **kwargs)
(pid=41942) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/requests/sessions.py", line 529, in request
(pid=41942) resp = self.send(prep, **send_kwargs)
(pid=41942) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/requests/sessions.py", line 645, in send
(pid=41942) r = adapter.send(request, **kwargs)
(pid=41942) File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/requests/adapters.py", line 519, in send
(pid=41942) raise ConnectionError(e, request=request)
(pid=41942) requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /v1/envs/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ef8abef6550>: Failed to establish a new connection: [Errno 111] Connection refused',))

RayActorError Traceback (most recent call last)
in
83 **cfg)
84 print('rllib_config', rllib_config)
---> 85 trainer = get_rl_model(algo, rllib_config)
86
87 # restore_file = ''

/media/kemove/16T/Jupyter/Electronics/script/modelfree_trainer.py in get_rl_model(algo, rllib_config)
12 trainer = None
13 if algo == "PPO":
---> 14 trainer = ppo.PPOTrainer(config=rllib_config, env="rllibEnv-v0")
15 elif algo == "DQN":
16 trainer = dqn.DQNTrainer(config=rllib_config, env="rllibEnv-v0")

~/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/rllib/agents/trainer_template.py in init(self, config, env, logger_creator)
121
122 def init(self, config=None, env=None, logger_creator=None):
--> 123 Trainer.init(self, config, env, logger_creator)
124
125 def _init(self, config: TrainerConfigDict,

~/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/rllib/agents/trainer.py in init(self, config, env, logger_creator)
582 logger_creator = default_logger_creator
583
--> 584 super().init(config, logger_creator)
585
586 @classmethod

~/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/tune/trainable.py in init(self, config, logger_creator)
101
102 start_time = time.time()
--> 103 self.setup(copy.deepcopy(self.config))
104 setup_time = time.time() - start_time
105 if setup_time > SETUP_TIME_THRESHOLD:

~/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/rllib/agents/trainer.py in setup(self, config)
729
730 with get_scope():
--> 731 self._init(self.config, self.env_creator)
732
733 # Evaluation setup.

~/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/rllib/agents/trainer_template.py in _init(self, config, env_creator)
150 policy_class=self._policy_class,
151 config=config,
--> 152 num_workers=self.config["num_workers"])
153 self.execution_plan = execution_plan
154 self.train_exec_impl = execution_plan(self.workers, config)

~/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/rllib/agents/trainer.py in _make_workers(self, env_creator, validate_env, policy_class, config, num_workers)
817 trainer_config=config,
818 num_workers=num_workers,
--> 819 logdir=self.logdir)
820
821 @DeveloperAPI

~/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/rllib/evaluation/worker_set.py in init(self, env_creator, validate_env, policy_class, trainer_config, num_workers, logdir, _setup)
84 remote_spaces = ray.get(self.remote_workers(
85 )[0].foreach_policy.remote(
---> 86 lambda p, pid: (pid, p.observation_space, p.action_space)))
87 spaces = {
88 e[0]: (getattr(e[1], "original_space", e[1]), e[2])

~/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/_private/client_mode_hook.py in wrapper(*args, **kwargs)
80 if client_mode_should_convert():
81 return getattr(ray, func.name)(*args, **kwargs)
---> 82 return func(*args, **kwargs)
83
84 return wrapper

~/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/worker.py in get(object_refs, timeout)
1564 raise value.as_instanceof_cause()
1565 else:
-> 1566 raise value
1567
1568 if is_individual_id:

RayActorError: The actor died because of an error raised in its creation task, ray::RolloutWorker.init() (pid=41942, ip=192.168.1.4)
File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/util/connection.py", line 96, in create_connection
raise err
File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/util/connection.py", line 86, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

ray::RolloutWorker.init() (pid=41942, ip=192.168.1.4)
File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
chunked=chunked,
File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/connection.py", line 234, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/http/client.py", line 1287, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/http/client.py", line 1333, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/http/client.py", line 1282, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/http/client.py", line 1042, in _send_output
self.send(msg)
File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/http/client.py", line 980, in send
self.connect()
File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/connection.py", line 200, in connect
conn = self._new_conn()
File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/connection.py", line 182, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7ef8abef6550>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

ray::RolloutWorker.init() (pid=41942, ip=192.168.1.4)
File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/requests/adapters.py", line 450, in send
timeout=timeout
File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/urllib3/util/retry.py", line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /v1/envs/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ef8abef6550>: Failed to establish a new connection: [Errno 111] Connection refused',))

During handling of the above exception, another exception occurred:

ray::RolloutWorker.init() (pid=41942, ip=192.168.1.4)
File "python/ray/_raylet.pyx", line 523, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 530, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 534, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 484, in ray._raylet.execute_task.function_executor
File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/_private/function_manager.py", line 563, in actor_method_executor
return method(__ray_actor, *args, **kwargs)
File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/ray/rllib/evaluation/rollout_worker.py", line 392, in init
self.env = env_creator(env_context)
File "", line 43, in
File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/gym/envs/registration.py", line 184, in make
return registry.make(id, **kwargs)
File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/gym/envs/registration.py", line 106, in make
env = spec.make(kwargs)
File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/gym/envs/registration.py", line 76, in make
env = cls(_kwargs)
File "/media/kemove/16T/Jupyter/Electronics/rl4rs/server/httpEnv.py", line 12, in init
self.instance_id = self.client.env_create(env_id, config)
File "/media/kemove/16T/Jupyter/Electronics/rl4rs/server/gymHttpClient.py", line 55, in env_create
resp = self._post_request(route, data)
File "/media/kemove/16T/Jupyter/Electronics/rl4rs/server/gymHttpClient.py", line 43, in _post_request
data=json.dumps(data))
File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/requests/sessions.py", line 577, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/requests/sessions.py", line 529, in request
resp = self.send(prep, **send_kwargs)
File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/requests/sessions.py", line 645, in send
r = adapter.send(request, **kwargs)
File "/home/kemove/anaconda3/envs/rl4rs/lib/python3.6/site-packages/requests/adapters.py", line 519, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /v1/envs/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ef8abef6550>: Failed to establish a new connection: [Errno 111] Connection refused',))

​