Comments (6)
Yes that is correct.
Lda.docToTop & lda.worToTop are local to each machine. Essentially topic assignments for documents in the chunk assigned to a machine.
Lda.topToWor is expected to be similar across the 3 machines. For an interpretation of the topic model you can use any one of them.
But there is only one global model built which is stored in the global folder along with the global dictionary. This is the one used while testing.
--Shravan
-----Original Message-----
From: yanbo68 [mailto:[email protected]]
Sent: Thursday, April 12, 2012 1:06 PM
To: Shravan Narayanamurthy
Subject: [Yahoo_LDA] the results of Y!LDA with multi machines (#10)
Hi,
I am using Y!LDA in Hadoop with 3 computers.
I got the results of "train mode" and found it a little bit confusion. I ran the script with --topics=20, and found that the files "lda.docToTop.txt, lda.topToWor.txt, lda.worToTop.txt" exist in 3 different directories. Each directory has 20 topics. Is it correct?
What am I supposed to get the "test" result from the "trained model"? Still 3 different directories?
Hope somebody can help me. Thanks a lot!
Yanbo
Reply to this email directly or view it on GitHub:
https://github.com/shravanmn/Yahoo_LDA/issues/10
from yahoo_lda.
Thanks a lot!
I checked the lda.topToword file.
For the result of "train mode", each topic has almost 4 different words for different machine.
But "test mode" is much better, only 1 different word for each topic. I think I can interpret the topic model using "test mode" result.
Btw, for the topic counts table, though there are 3 tables after "train mode", I found that it seems the system will merge the 3 tables together during the "test mode"?
The LOG says :"Initializing Word-Topic counts table from 3 dumps with topic_counts/lda.ttc.dump as prefix ......"
So each machine is using the same big table?
from yahoo_lda.
In line...
-----Original Message-----
From: yanbo68 [mailto:[email protected]]
Sent: Friday, April 13, 2012 8:29 AM
To: Shravan Narayanamurthy
Subject: Re: [Yahoo_LDA] the results of Y!LDA with multi machines (#10)
Thanks a lot!
I checked the lda.topToword file.
For the result of "train mode", each topic has almost 4 different words for different machine.
[shrav] How many iterations did you run?
But "test mode" is much better, only 1 different word for each topic. I think I can interpret the topic model using "test mode" result.
Btw, for the topic counts table, though there are 3 tables after "train mode", I found that it seems the system will merge the 3 tables together during the "test mode"?
The LOG says :"Initializing Word-Topic counts table from 3 dumps with topic_counts/lda.ttc.dump as prefix ......"
So each machine is using the same big table?
[shrav] Yes. A global table is created and a local table per machine is induced using the global table.
--Shravan
Reply to this email directly or view it on GitHub:
https://github.com/shravanmn/Yahoo_LDA/issues/10#issuecomment-5107635
from yahoo_lda.
I ran 200 iterations
from yahoo_lda.
If you run about 500 to 600 iterations, the words will look similar in the different topToWor files. This is what we have observed.
--Shravan
-----Original Message-----
From: yanbo68 [mailto:[email protected]]
Sent: Friday, April 13, 2012 4:07 PM
To: Shravan Narayanamurthy
Subject: Re: [Yahoo_LDA] the results of Y!LDA with multi machines (#10)
I ran 200 iterations
Reply to this email directly or view it on GitHub:
https://github.com/shravanmn/Yahoo_LDA/issues/10#issuecomment-5112216
from yahoo_lda.
Thanks a lot! I will try more iterations.
from yahoo_lda.
Related Issues (14)
- streaming mode broken HOT 3
- Batch vs. Streaming Classification - "Using the Model" HOT 1
- a question about different topics from same corpus
- traning attempt failed run on hadoop HOT 1
- Maxmem parameter in learntopics
- Yahoo LDA and Hadoop
- Multi machine hadoop setup HOT 3
- A synchronization problem and ICE networking error HOT 1
- Failed stream-job while run Yahoo!LDA on hadoop distribution system
- unable to run formatter HOT 12
- IceUtil::Exception HOT 1
- std::bad_alloc exception
- Choosing count of topics
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from yahoo_lda.