suhitaghosh10 / fictionretrieval Goto Github PK

We apply the standard topic mining algorithms like LDA, pLSA - to determine topics per chunk. The idea is to create topics per 10k chunk and then roll them up.

L1,L2, Cosine

Chunking

• Segment the content in chunks of 10,000 words.
  • Perform Lemmatization
  • Chunking - with stopwords -  using porter stemmer plus the words mentioned in - http://www.matthewjockers.net/macroanalysisbook/expanded-stopwords-list/
  • Chunking - without stop word removal

Genre assignment done manually

1.Writing style,
2.Complexity

Pronoun - http://www.ucl.ac.uk/internet-grammar/nouns/pronoun.htm
1) Personal
2) Possesive
3) Male (personal+possesive)
4) Female (personal+possesive)

Punctuation-
1.comma
2.period
3.colon
4.semicolon
5.hyphen
6.Exclamation (combined with interjection POS)

Title,author,genre extraction from gutenberg epub

Locative preposition - n, at, on, by, next to, beside, near,in front of,towards
scene-setting - come,there(existential)

coordinating conjunctions

• remove gutenberg related data
• remove metadata
• remove other non-content related stuff

Double quotes - give an idea of the amount of conversation between people in the book.

Some common character names (as per SMEs) have been added to stopword list.
Need to get the character names from the current corpus and add further to that existing stoplist.