#plsa

This is a tiny PLSA tool written in Java, you can use or modify it as you wish.

1. Compile the project

This project is maintained by Ant. If you want to re-compile it, just execute "./build.sh" in your termial cmdline.

The ".jar" file is already included in the "dist/lib" directory, so you can also use this tool without re-compiling.

2. How to use this tool

There are only 2 shell scripts under the root folder of this project:

process

plsa

2.1 Usage

2.1.1 process

process is used for pre-processing the raw corpus. It will transfer the raw corpus files into the sparse matrix file for PLSA training.

Usage

./process -corpusfolder <folder> -stopwordfile <file> -vocfile <file> -matrixfile <file> -lowfreq <int>

2.1.2 plsa

plsa is used for PLSA training.

Usage

./plsa -matrixfile <file> -vocfile <file> -topic <int> -iter <int> -eps <double> -topk <int> -resultfolder <folder>

2.2 Parameters of process and plsa

-corpusfolder: raw corpus folder.

-stopwordfile: stopwords vocabulary file.

-vocfile: the file for the generated vocabulary of corpus.

-matrixfile: the matrix file generated, this file will be used for train.

-lowfreq: low freq(<lowfreq) word will be discarded.

-topic: the topic #.

-iter: the max train iterate times.

-eps: early stop threshold(tool will sotp when the likelihood diff of 2 iterate < eps).

-topk: the topK words of a topic will show.

-resultfolder: train reslut folder.

3. Input Data Format

Plain english text files.

4. Output Data Format

4.1. Files generated by process

4.1.1 maxtrixfile is sparse matrix for raw corpus

file_0 word:freq word:freq ...

file_1 word:freq word:freq ...

...

file_D word:freq word:freq ...

file_i is the full pathname of the i-th file

4.1.2 vocfile includes the vocabulary got from the corpus

word0

word1

...

wordV

file_i is the full file name of the i-th file

4.2. Files generated by plsa

4.2.1 p_z is for p(z)

prob_z0 prob_z1 ... prob_zK

4.2.2 p_d_z is for p(d|z)

p(d0|z0) p(d0|z1)... p(d0|zK)

p(d1|z0) p(d1|z1)... p(d1|zK)

...

p(dD|z0) p(dD|z1)... p(dD|zK)

4.2.3 p_w_z is for p(w|z)

p(w0|z0) p(w0|z1)... p(w0|zK)

p(w1|z0) p(w1|z1)... p(w1|zK)

...

p(wV|z0) p(wV|z1)... p(wV|zK)

4.2.4 docTopics is for p(z|d)

p(z0|d0) p(z1|d0)... p(zK|d0)

p(z0|d1) p(z1|d1)... p(zK|d1)

...

p(z0|dD) p(z1|dD)... p(zK|dD)

4.2.5 wordTopics is for p(z|w)

p(z0|w0) p(z1|w0)... p(zK|w0)

p(z0|w1) p(z1|w1)... p(zK|w1)

...

p(z0|wV) p(z1|wV)... p(zK|wV)

4.2.6 topicTopKWords is for top K words of each topic

[Topic0]

[word0]

[word1]

...