How to extract phrases from Wikipedia? about densephrases HOT 5 CLOSED

Hi @Albert-Ma,

if you are looking for getting phrase representations from the documents, please refer here.

The code that extracts phrases is https://github.com/princeton-nlp/DensePhrases/blob/main/generate_phrase_vecs.py and also see

from densephrases.

Hi @jhyuklee,
I am looking for how to get phrases from raw documents like Wikipedia or squad?
This is the very first step of the phrase retrieval and I think this may occur before training the model.
Or the phrase retrieval didn't extract phrases from documents explicitly and it did it on the fly?
I'll check the generate_phrase_vecs.py function.
Thanks.

from densephrases.

Phrase retrieval is trained with QA datasets which contain phrase-level answer annotations. So we don't need to explicitly extract phrases before training. After training, generate_phrase_vecs.py filters out irrelevant phrases (i.e., start/end tokens) and stores only relevant phrases that could be used for downstream tasks. This filtering model was also trained on QA datasets so that these phrases serve as answer candidates. In embed_utils.py, there is a function that applies this filtering:

Here, metadata means phrase vector related outputs for each document (phrase start/end vectors, start2end mapper, etc).

from densephrases.

Got it, thanks

from densephrases.

You can also check this issue! #17
I think it's related.

from densephrases.