chyikwei / bnp Goto Github PK

Hello,

This question my be a result of my lack of understanding, but I have noticed that changing the parameter n_topic_truncate, which controls the total number of topics returned, also changes the number of live and dead topics predicted by the model. For example, if I choose n_topic_truncate = 200, it may return 200 topics where 10 have 'useful information' and the 190 do not. What I mean is the 10 topics will make up over 99% of the corpus, and all have unique distributions of topic-words. But the remaining 190 topics will all have identical topic-word distributions and make up a small fraction of the corpus.

Now, if I change n_topic_truncate = 20, instead of obtaining 10 live topics and 10 dead topics, the model may return, for example, 3 live topics and 17 dead ones. Why is this? Why are the same 10 live topics not identified with different n_topic_truncate values? I can understand that if n_topic_truncate is too small, then the model may contain too few topics to represent the corpus. But I have found that even n_topic_truncate=2 returns one topic that represents the majority of the data and the second topic hardly presents any (based on hdp.topic_distribution()).

Conversely, increasing n_topic_truncate seems to always increase the number of live topics. Then, how does one determine an appropriate value for n_topic_truncate? In this way, this HDP seems like LDA, in which one needs to designate K beforehand. Am I missing something?

Thanks for your time and I appreciate any help or insight.

It is not quite clear, how the number of topics is obtained. How the 3 topics were eliminated? and what is topic distribution? since each topic is represented by one value, where the sum of all of them =1.