Continual data stream coming about comstream HOT 4 CLOSED

Hi,
In which config are you training?

from comstream.

There are two important parameters that you should config well to control the number of agents as the data comes its number increases.
agent_fading_rate and del_agent_weight
if you increase the fading rate or del_agent_weight the number of agents will decrease in time so it would not get slower.
these parameters are already explained in the paper!

from comstream.

There are two important parameters that you should config well to control the number of agents as the data comes its number increases. agent_fading_rate and del_agent_weight if you increase the fading rate or del_agent_weight the number of agents will decrease in time so it would not get slower. these parameters are already explained in the paper!

But I would like to keep all gathered clusters instead of deleting some of them. Is possible to use Faiss to accelerate as the data stream comes continually?

from comstream.

There are two important parameters that you should config well to control the number of agents as the data comes its number increases. agent_fading_rate and del_agent_weight if you increase the fading rate or del_agent_weight the number of agents will decrease in time so it would not get slower. these parameters are already explained in the paper!

But I would like to keep all gathered clusters instead of deleting some of them. Is possible to use Faiss to accelerate as the data stream comes continually?

This algorithm aims to delete redundant clusters (outliers) during the time and also we cannot keep whole data in our model (imagine you had 10TB data from 2010 to 2022) so they have to get deleted!
About the Faiss, I believe it is possible to integrate it with our method as we are searching for the best cluster (Cluster representation) in communication time. But the problem is that the indexes should get updated as the clusters change!

from comstream.