Epoch length incorrectly calculated when using DDP about composer HOT 5 CLOSED

After further investigation, I've realised this was because we were using the "size" rather than "length" of the streaming dataset.
I have confirmed this now fixes the issues and will close the issue.
Thank you for the input!

from composer.

@JAEarly would you mind adding streaming and composer versions? We've fixed a similar issue recently so want to confirm it's not that

from composer.

@JAEarly would you mind adding streaming and composer versions? We've fixed a similar issue recently so want to confirm it's not that

Sure, apologies for not providing those.

composer: 0.23.2
streaming: 0.7.6

What version was the similar issue fixed in? I believe these are the correct versions; we may be using slightly older ones.

from composer.

@JAEarly looks different.. can you print your dataloader length and compare to number of steps composer thinks? If you have a repro that would be helpful as well

from composer.

@JAEarly looks different.. can you print your dataloader length and compare to number of steps composer thinks? If you have a repro that would be helpful as well

Apologies for the delay. The dataloader length and number of steps per epoch is the same for both single GPU and distributed (2 x 8):

1 x 1:
  Train dataloader length: 834882
  Composer number of steps per epoch: 834882
 
2 x 8:
  Train dataloader length: 834882
  Composer number of steps per epoch: 834882

Note in both cases I'm passing batch size = 64 to the dataloader.

from composer.