extending torchdash about tensordash HOT 8 CLOSED

Hey Sai Krishna! Thanks for bringing this issue to light. We have fixed it in the latest version of the tensor-dash package, tensor-dash 1.8.1. Please use this and let us know if you face any more problems.
Thank you!

from tensordash.

Hi! Thanks for your suggestion. We will add an option to include custom metrics in the next version.

from tensordash.

Also noticed that the logging stops after a while. even though the model is still training. any hints as to why this could happen? i have replicated this behavior on multiple models, running from multiple servers. typically deep learning models train for a larger period of time, so it will be great to resolve this.

from tensordash.

Hey Sai Krishna! Can you check if the data is updated in the app, by dragging down to refresh? One possible reason might be that data is being uploaded by the model but is not displaying since the app does not automatically sync. We are currently working on this specific issue and it would be fixed in the next release.

from tensordash.

thanks for the prompt response. Yes. I have tried refreshing, killing the app completely and restarting as well. Here is an observation that might help: I ran two models concurrently, one that runs 40 epochs per hour and another that runs 200 per hour. The first model seems to stop updating after 30 epochs, and the latter after 160 epochs.

from tensordash.

Here is a script with a sample integration of tensordash. This is example of former case. Here is an example script of the latter

from tensordash.

Okay. That behaviour is not expected. Give us time, we will try to figure it out and fix it by the next release.

from tensordash.

thanks. closing the issue for now

from tensordash.