daochenzha / autoshard Goto Github PK

[ERROR:640844 training:471 2022-10-12 11:16:25,954] Exception in worker process 0
Traceback (most recent call last):
  File "/home/lcfjr/codes/autoshard/autoshard/training.py", line 437, in act
    env_output = env.step(agent_output["action"])
  File "/home/lcfjr/codes/autoshard/autoshard/utils.py", line 72, in step
    frame, reward, done, info = self.gym_env.step(action.item())
  File "/home/lcfjr/codes/autoshard/autoshard/bench.py", line 310, in step
    reward = self._get_reward()
  File "/home/lcfjr/codes/autoshard/autoshard/bench.py", line 438, in _get_reward
    latencies = self._get_cost(self.plan, predictor=False)
  File "/home/lcfjr/codes/autoshard/autoshard/bench.py", line 323, in _get_cost
    costs = [self._envs[self.env_id].single_step([self.table_indices[self.env_id][i] for i in indices]) for j, indices in enumerate(indices_batch) if j in non_empty]
  File "/home/lcfjr/codes/autoshard/autoshard/bench.py", line 323, in <listcomp>
    costs = [self._envs[self.env_id].single_step([self.table_indices[self.env_id][i] for i in indices]) for j, indices in enumerate(indices_batch) if j in non_empty]
  File "/home/lcfjr/codes/autoshard/autoshard/bench.py", line 136, in single_step
    op = split_table_batched_embeddings_ops.SplitTableBatchedEmbeddingBagsCodegen(
  File "/home/lcfjr/.local/lib/python3.8/site-packages/fbgemm_gpu/split_table_batched_embeddings_ops.py", line 232, in __init__
    self.dummy_tensor: Tensor = torch.zeros(0, device=device)
  File "/home/lcfjr/miniconda3/envs/dev/lib/python3.8/site-packages/torch/cuda/__init__.py", line 207, in _lazy_init
    raise RuntimeError(
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
```