ink-usc / clustype Goto Github PK

Automatic Entity Recognition and Typing for Domain-Specific Corpora (KDD'15)

License: GNU General Public License v3.0

Python 99.26% Shell 0.74%

natural-language-processing information-extraction entity-extraction

clustype's Introduction

ClusType

Source code for SIGKDD'15 paper ClusType: Effective Entity Recognition and Typing by Relation Phrase-Based Clustering (Slides).

Given a text corpus (e.g., a collection of news articles), it performs automatically entity extraction and typing using distant supervision (i.e., examples from external knowledge bases like Freebase). For example, from the sentence "The best BBQ I’ve tasted in Phoenix " the system will recognize BBQ as food and phoenix as location. More background can be found in our WWW'16 tutorial.

ClusType works on coarse-grained entity types (e.g., Person, Location, Organization); for more fine-grained entity typing, please use AFET (Ren et al., EMNLP'16).

Data

NYT:
- Corpus: 110k New York Times news articles (download)
- Seed entities: entity linking result by DBpediaSpotlight (download)
Yelp:
- Corpus: 230k Yelp reviews sampled from Yelp Dataset (download)
- Seed entities: entity linking result by DBpediaSpotlight (download)
Tweet:
- Corpus: 302k tweets from May 2011 (download)
- Seed entities: entity linking result by DBpediaSpotlight (download)

System Output & Evaluation

The system output on NYT dataset can be downloaded from here. We evaluated the result over ~1k (20,874 annotated entity mentions) gold standard set. Sample output on 50k Yelp reviews can be download from here.

Evaluate the result:

python src/evaluation.py -ResultPath -GroundTruthPath

Dependencies

python 2.7
numpy, scipy, scikit-learn, lxml, TextBlob and related corpora

$ sudo pip install numpy scipy sklearn lxml textblob
$ sudo python -m textblob.download_corpora

Default Run

$ ./run.sh

Run.sh - File path setup

We take Yelp dataset as an example.

Input: text corpus path.

RawText='data/yelp/yelp_230k.txt'

format: "docId \TAB document \n"

Input: type mapping file path.

TypeFile='data/yelp/type_tid.txt'

format: "type name \TAB typeId \n". "NIL" means "Not-of-Interest"

Input: mapping between Freebase and DBpedia entities.

FreebaseMap='data/freebase_links.nt'

Download Freebase-to-DBpedia mapping file. Place it under "data/" directory

Output: output file from candidate generation (format: "docId \TAB segmented sentence \n").

SegmentOutFile='result/segment.txt'

Segments are separated by ",". Entity mention candidates are marked with ":EP". Relation phrases are marked with ":RP".

Output: entity linking result (please download the corresponding seed entity files).

SeedFile='data/yelp/seed_yelp.txt'

Format: "docId \TAB entity name \TAB Original Freebase Type \TAB Refined Type \TAB Freebase EntityID \TAB Similarity Score \TAB Relative Rank \n".
NOTE: Our entity linking module calls DBpediaSpotLight Web service, which has limited querying speed. This process can be largely accelarated by installing the tool on your local machine Link.

Output: entity mentions found in each document.

ResultFile='result/yelp/results.txt'

Format: "docId \TAB entity mention \TAB entity type \n".

Output: In-text annotation of entity mentions.

ResultFileInText='result/yelp/resultsInText.txt'

Run.sh - Model parameters

Threshold on significance score for candidate generation.

significance="2"

Switch on capitalization feature for candidate generation.

capitalize="1"

Maximal phrase length for candidate generation.

maxLength='4'

Minimal support of phrases for candidate generation.

minSup='30'

Number of relation phrase clusters.

NumRelationPhraseClusters='500'

Reference

@inproceedings{ren2015clustype,
  title={Clustype: Effective entity recognition and typing by relation phrase-based clustering},
  author={Ren, Xiang and El-Kishky, Ahmed and Wang, Chi and Tao, Fangbo and Voss, Clare R and Han, Jiawei},
  booktitle={Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining},
  pages={995--1004},
  year={2015},
  organization={ACM}
}

clustype's People

Contributors

Stargazers

Watchers

clustype's Issues

Does the freebase support Chinese?

clustype_noClus vs. clustype_inexact vs. clustype_exact

Hello,

I notice that step1-entity_recognition.py calls clustype_noClus but has commented out a call toclustype_inexact. Looking in clustype.py I see that there is also a clustype_exact function. These functions seem quite similar -- could you explain what is the difference between them?

Thanks!

How to run eval?

I'm trying to figure out how to run eval. From what I can tell, run.sh doesn't perform eval but there is code in evaluation.py that seems to be for evaluation. However, functions like test_method in evaluation.py aren't called anywhere else in the code.

Could you provide some information about how to run evaluation?

Thanks!

Issues

I ran ClusType on the yelp corpus that was provided but somehow it failed and provided me with this message.

links in S_L: 141185 , #links in S_R: 168899

S_M dims 277337 277337 #links in S_M: 1685222
graph loading DONE
No target type entity seeded!!!!
Percent Seeded Mention: 0.0 % of 277337 , #target/All =
Traceback (most recent call last):
File "src/step1-entity_recognition.py", line 80, in
Y0 = set_Y(seed_mid, seedMention_tid_score, mid_mention, m, T).todense()
File "/Text_Mining/Method/shanzhenren-ClusType-3669a07/shanzhenren-ClusType-3669a07/src/algorithm.py", line 66, in set_Y
', #target/All = ', num_target/(Y.nnz) * 100
ZeroDivisionError: integer division or modulo by zero

Do you know what might have happened?

FreeBase deprecated

As I'm trying the entity linking using dBpedia-Spotlight, I realize I can not do entity typing anymore as the FreeBase is deprecated.

More specifically, I can not find a good alternative to find "notable_types" now.
And the other APIs does not seem to show good performance on entity typing:

I would appreciate if you guys can provide some help on entity typing during seed entity generation.

Thanks.

Segment file has different Entity phrase than the results file and gold standard training data

Hi Xiang,

I am using clustype and I am following the instructions in the readme file. However I don't get the same entity phrases as the system results or the gold standard file. I am running the nyt dataset.

An example of the issue is Alan Kazdin in article ID 4. Both the results file and gt_nyt file has that entity phrase, but my segment file labels "GAME Alan Kazdin" as an entity phrase.

Thank you,
Neil

Incorrect SeedFile value

Hi, thanks for your research and all the work, I find the code quite useful for me to learn more about unsupervised NER!

There are two little issues that I want to address:

In terms of the example run, I realize the correct value for SeedFile should be 'data/yelp/seed_yelp.txt' instead of 'result/yelp/seed_yelp.txt'.
I also don't think the notes in this page is still valid. (However, I think the installation process in this page is really helpful)

Does the type will propagate to each entity mention?

I notice that when clustype mining the entity mention , the recall is the mainly to consider.But not each mention have a entity type. So if an entity mention is not a real entity, clustype will not assign a entity type to it, is that right？Or,clustype will try to assign a entity type for each mention, no matter the entity mention is a real mention or not.