[Feature]: run sub wandb run inside an wandb run about wandb HOT 10 CLOSED

Hi @LianhaoYin thank you for writing in. Could you please provide more context on the use case why'd you wish to start one run inside another one? would you additionally wish to resume to the same parent run that you had started? this isn't supported but you may be interested to use runs forking for your use case.

Regarding the x-axis step, you could make use of define_metric to create a customized step (docs) and have different step for x, sub_x metrics.

Would this code example below work for you?

import wandb
import random
import multiprocessing

def main():
    wandb.init(project='define-test')
    
    # Define custom metrics with step requirements
    wandb.define_metric("global_step")
    wandb.define_metric("sub_step")
    wandb.define_metric("x", step_metric="global_step")
    wandb.define_metric("sub_x", step_metric="sub_step")
    
    for global_step in range(1000):
        wandb.log({"x": random.random(), "global_step": global_step})
        if global_step % 1000 == 0:
            p = multiprocessing.Process(target=sub_function)
            p.start()
            p.join()

    wandb.finish()

def sub_function():    
    for sub_step in range(10):
        wandb.log({"sub_x": random.random(), "sub_step": sub_step})
    
if __name__ == "__main__":
    main()

from wandb.

Thanks for your example code. The reason why I want this is that the sub_function in my example code is a big project and it use the one input from the main function. The sub_function is an already finished project with its own wandb init(), log(), finish and everything. I would like to integrate the wandb logging of the sub_fuction to the main without major change of sub_function which is already done. The problem is that the the self.log() in main and sub_function will add step by 1 everything it is called. This make the integration very complex. There are also other problem like if I want the sub function to login the same run of main function, it is not possible, since the wandb.init() will start a new run with same name. Hope this is clear.

from wandb.

Great, thank you for providing a detailed use case. In that case, would it not work for you that the sub_function is being called as a subprocess as below (based on your example)?

import wandb
import multiprocessing

wandb.login()

def main():
  wandb.init(project='subrun')
  for i in range(100):
    wandb.log({'metric_1': i})
    if i % 10 == 0:
      p = multiprocessing.Process(target=sub_function)
      p.start()
      p.join()
  wandb.finish()

def sub_function():
  wandb.init(project='subrun')
  for j in range(2):
    wandb.log({'metric_2': 0.1*j})
  wandb.finish()

if __name__ == '__main__':
  main()

from wandb.

from wandb.

Hi @LianhaoYin thank you for the clarification. In that case, you could share the run object in a subprocess, similarly to what's done in this multiprocessing example:
https://docs.wandb.ai/guides/track/log/distributed-training#share-a-wb-run

Would this option work for you?

from wandb.

Hi @LianhaoYin we wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved.

from wandb.

Hi, it solve the problem to log into same run, However, as stated in the problem:
The problem is that the the self.log() in main and sub_function will add step by 1 everything it is called. This make the integration very complex. ( the step_sub and step_main should be independent)

from wandb.

Hi @LianhaoYin thank you for the clarification. That's correct, we will +1 the internal wandb step in every wandb.log() call, unless you're passing the commit=False arg: https://docs.wandb.ai/ref/python/log. Alternatively, you could use wandb.define_metric method to create customized x-axis steps: https://docs.wandb.ai/guides/track/log/customize-logging-axes. Have you tried either of these options, and would they not work for your use case?

from wandb.

Hi @LianhaoYin just wanted to follow up with you regarding your support request as we have not heard back from you. Please let me know if define_metric would prevent this issue for you by having a customised x-axis for the subfunction?

from wandb.

Hi @LianhaoYin since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know, and we will be more than happy to keep investigating!

from wandb.