[App]: Duplicate Runs in the Sweep with Parallel Agents about wandb HOT 7 CLOSED

Hi @marsh312, thanks for flagging this! Would you mind sharing your sweep config so we can take a look?

from wandb.

Hi @marsh312, thanks for flagging this! Would you mind sharing your sweep config so we can take a look?

Of course！This is my config：

sweep_config = {
    'method': 'bayes',
    'name': 'mind_deforec_bayes_sweeps_newer',
    'metric': {
        'name': 'Test/f_measure',
        'goal': 'maximize',
    },
    'parameters': {
        'train_leaf_layer_epoch': {
            'values': [0, 2, 4]
        },
        'wd': {
            'values': [1e-4, 1e-3, 1e-2]
        },
        'batch_gap_to_rebuild_tree': {
            'values': [60000, 80000, 100000]
        },
        'dropout_p': {
            'values': [0.0, 0.1, 0.2, 0.3]
        },
        'sample_num': {
            'values': [50, 60, 70, 80, 90]
        },
        'emb_dim': {
            'values': [22, 24, 28]
        },
        'min_lr': {
            'values': [1.0e-4, 1.0e-5, 1.0e-6]
        },
        'scheduler_mode': {
            'values': ['cosine', 'exponential']
        },
    }
}

from wandb.

Thanks @marsh312! I did some tests on my end but was unable to reproduce the issue. Would you mind sharing if you're using Hydra? We have seen this issue happening when using it so wanted to double-check

from wandb.

Thanks @marsh312! I did some tests on my end but was unable to reproduce the issue. Would you mind sharing if you're using Hydra? We have seen this issue happening when using it so wanted to double-check

Of course, @luisbergua! I am using a shell script to run the sweep, not Hydra. The script is listed above.

from wandb.

Thanks @marsh312! I haven't been able to repro this issue yet so wanted to ask if you could try to avoid passing --gpu_id and only execute the command once. I know you mentioned that repeating runs have the same --gpu_id and agree that this doesn't seem to be associated with the use of multiple GPUs but would like to double-check. Also sharing our docs about sweeps on launch since those could be helpful for running sweeps in distributed environments

from wandb.

Sorry, @luisbergua! Due to resource limitations, I was only able to conduct some experiments based on your instructions and haven't witnessed the same issue again. Here is my hypothesis regarding the issue: because the sweep space I set is small and discrete, using the Bayesian sweep method might lead to duplicate runs. Do you think this hypothesis is reasonable?

from wandb.

Thanks for sharing this @marsh312! That could be a potential cause but not necessarily. Will close this request for now as it sounds like the issue isn't appearing anymore but feel free to re-open it if you see it again!

from wandb.