[ACL 2021] Learning Dense Representations of Phrases at Scale; EMNLP'2021: Phrase Retrieval Learns Passage Retrieval, Too https://arxiv.org/abs/2012.12624
Home Page: https://arxiv.org/abs/2012.12624
License: Apache License 2.0
Hi,
I'm trying to utilize DensePhrases as Korean version, not English version. The trained model is bert-base-multilingual-cased model (not span-bert model because it doesn't support Korean) on korquad dataset. The dataset sourced from Korean Wikipedia was manually preprocessed and used to create a custom phrase index.
The error that I encountered when running run_demo.py is as shown below:
02/14/2023 07:08:04 - ERROR - tornado.access - 500 POST /query2vec_api (127.0.0.1) 11.09ms
02/14/2023 07:08:04 - INFO - densephrases.utils.embed_utils - ---feature : {'input_ids': [101, 48556, 16617, 119202, 9323, 51431, 14040, 34907, 10892, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'qas_id': 0, 'question_text': '한국전쟁 발발시점은'} ,
example['question_text'] : 한국전쟁 발발시점은
02/14/2023 07:08:04 - INFO - densephrases.utils.embed_utils - ---query_eval_features [{'input_ids': [101, 48556, 16617, 119202, 9323, 51431, 14040, 34907, 10892, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'qas_id': 0, 'question_text': '한국전쟁 발발시점은', 'unique_id': 1000000000}]---
02/14/2023 07:08:04 - INFO - densephrases.utils.embed_utils - ---question_dataloader <torch.utils.data.dataloader.DataLoader object at 0x7f76d1e7da60>---
02/14/2023 07:08:04 - INFO - densephrases.utils.embed_utils - ---DEVICE cpu---
02/14/2023 07:08:04 - INFO - densephrases.utils.embed_utils - ---BATCH [tensor([[ 101, 48556, 16617, 119202, 9323, 51431, 14040, 34907, 10892,
102, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0]]), tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0]]), tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0]]), tensor([0])]---
02/14/2023 07:08:04 - INFO - densephrases.utils.embed_utils - ---len(batch) 4---
02/14/2023 07:08:04 - INFO - densephrases.utils.embed_utils - --tmp 1---
02/14/2023 07:08:04 - ERROR - run_demo - Exception on /query2vec_api [POST]
Traceback (most recent call last):
File "/opt/conda/envs/densephrases-v1.1.0/lib/python3.8/site-packages/flask/app.py", line 2073, in wsgi_app
response = self.full_dispatch_request()
File "/opt/conda/envs/densephrases-v1.1.0/lib/python3.8/site-packages/flask/app.py", line 1518, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/opt/conda/envs/densephrases-v1.1.0/lib/python3.8/site-packages/flask_cors/extension.py", line 165, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/opt/conda/envs/densephrases-v1.1.0/lib/python3.8/site-packages/flask/app.py", line 1516, in full_dispatch_request
rv = self.dispatch_request()
File "/opt/conda/envs/densephrases-v1.1.0/lib/python3.8/site-packages/flask/app.py", line 1502, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "run_demo.py", line 59, in query2vec_api
outs = list(query2vec(batch_query))
File "/data/code-server/baseline/DensePhrases/densephrases/utils/embed_utils.py", line 412, in get_question_results
outputs = model(**inputs)
File "/opt/conda/envs/densephrases-v1.1.0/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/data/code-server/baseline/DensePhrases/densephrases/encoder.py", line 177, in forward
query_start, query_end = self.embed_query(input_ids_, attention_mask_, token_type_ids_)
File "/data/code-server/baseline/DensePhrases/densephrases/encoder.py", line 132, in embed_query
outputs_s_ = self.query_start_encoder(
File "/opt/conda/envs/densephrases-v1.1.0/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/opt/conda/envs/densephrases-v1.1.0/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py", line 992, in forward
embedding_output = self.embeddings(
File "/opt/conda/envs/densephrases-v1.1.0/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/opt/conda/envs/densephrases-v1.1.0/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py", line 214, in forward
inputs_embeds = self.word_embeddings(input_ids)
File "/opt/conda/envs/densephrases-v1.1.0/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/opt/conda/envs/densephrases-v1.1.0/lib/python3.8/site-packages/torch/nn/modules/sparse.py", line 158, in forward
return F.embedding(
File "/opt/conda/envs/densephrases-v1.1.0/lib/python3.8/site-packages/torch/nn/functional.py", line 2044, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self
02/14/2023 07:08:04 - ERROR - tornado.access - 500 POST /query2vec_api (127.0.0.1) 7.52ms
I tried to figure out input and output shape, but still don't know what to do to get them.
It would be appreciated if you could help me. Thanks in advance.
HI,
I'm trying to create a dense representations from my corpus and search paragraphs/phrases by keywords or a question. I don't have labeled Questions and Answers and I don't need for now to get answers, just retrieve documents possibly containing the answer.
I build a JSON with my corpus (pt-br) like this:
{
"data": [
{
"title": "Radicais livres: o que são, efeitos no corpo e como se proteger",
"paragraphs": [
{
"context": "Os radicais livres ...""
},
{
"context": "Desta forma, quanto menos radicais livres, ..."
}, ...
then I ran the following commands:
python generate_phrase_vecs.py \
--pretrained_name_or_path SpanBERT/spanbert-base-cased \
--data_dir ./data \
--cache_dir ./cache \
--test_file ../tua-saude/all_data.json \
--do_dump \
--max_seq_length 512 \
--fp16 \
--filter_threshold -2.0 \
--append_title \
--output_dir ./data/densephrases-multi_sample \
--load_dir princeton-nlp/densephrases-multi
python build_phrase_index.py \
--dump_dir ./data/densephrases-multi_sample/dump \
--stage all \
--replace \
--num_clusters 128 \
--fine_quant OPQ96 \
--doc_sample_ratio 0.3 \
--vec_sample_ratio 0.3 \
--cuda
python scripts/preprocess/compress_metadata.py \
--input_dump_dir ./data/densephrases-multi_sample/dump/phrase \
--output_dir ./data/densephrases-multi_sample/dump
Those commads looks like working fine. Here the contents of output_dir
Now, when I try to use the model:
model = DensePhrases(
load_dir='princeton-nlp/densephrases-multi',
dump_dir='./data/densephrases-multi_sample/dump/',
index_name='start/128_flat_OPQ96'
)
This error raises:
>>>
This could take up to 15 mins depending on the file reading speed of HDD/SSD
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/projetos/u4vn/DensePhrases/densephrases/model.py", line 52, in __init__
self.truecase = TrueCaser(os.path.join(os.environ['DATA_DIR'], self.args.truecase_path))
File "/projetos/u4vn/DensePhrases/densephrases/utils/data_utils.py", line 366, in __init__
with open(dist_file_path, "rb") as distributions_file:
FileNotFoundError: [Errno 2] No such file or directory: './data/truecase/english_with_questions.dist'
What am I missing? What file is this?
Hi
I create a custom index and finished all steps but when I test it i faced this error
File "/DensePhrases/densephrases/index.py", line 507, in search
outs = self.search_phrase(
File "/DensePhrases/densephrases/index.py", line 428, in search_phrase
out = [{
File "/DensePhrases/densephrases/index.py", line 431, in <listcomp>
'end_pos': (groups_all[doc_idx]['word2char_end'][groups_all[doc_idx]['f2o_start'][end_idx]].item()
IndexError: index 99 is out of bounds for axis 0 with size 35
Hi Jhyuklee,
Thank you for good works and support.
I have one query. Here I want use my custom pdf statements as a dump in place of Wikipedia dump, and want a model to get information from pdf data rather than getting it from wikipedia.
Do I need to freshly train our whole dump data or is there a way where I can fine tune this model based on checkpoints trained by you.
Pls guide.
Hi~ Thanks a lot for your open source work.
When I run your code for SQuAD dataset in one passage training, I got 77.3 EM and 85.7 F1. I ran code in this script - python train_rc.py --model_type bert --pretrained_name_or_path SpanBERT/spanbert-base-cased --data_dir densephrases/densephrases-data/single-qa --cache_dir densephrases/cache --train_file squad/train-v1.1_qg_ents_t5large_3500_filtered.json --predict_file squad/dev-v1.1.json --do_train --do_eval --per_gpu_train_batch_size 24 --learning_rate 3e-5 --num_train_epochs 3.0 --max_seq_length 384 --seed 42 --fp16 --fp16_opt_level O1 --lambda_kl 4.0 --lambda_neg 2.0 --lambda_flt 1.0 --filter_threshold -2.0 --append_title --evaluate_during_training --overwrite_output_dir --teacher_dir densephrases/outputs/spanbert-base-cased-squad
I also train this model for another 2 epochs like your makefile using pre-batch negative and train-v1.1.json (the real squad data), but the results is still below the paper results.
(1) Does I should use different hyperparameters? I found your paper use different parameters with your script, such as batch size (84 vs 24) or lambda weight, etc.
(2) In the paper, the results are the average of random seeds?
(3) Do you use the whole nq and squad datasets to train the model?
logs as following, thanks
convert squad examples to features: 100%|█████████████████████████████████████████████████████████████████████████| 902/902 [00:00<00:00, 2092.37it/s]
add example index and unique id: 100%|██████████████████████████████████████████████████████████████████████████| 902/902 [00:00<00:00, 439863.06it/s]
06/14/2022 22:26:05 - INFO - main - Number of trainable params: 258,127,108
Selected optimization level O1: Insert automatic casts around Pytorch functions and Tensor methods.
Defaults for this optimization level are:
enabled : True
opt_level : O1
cast_model_type : None
patch_torch_functions : True
keep_batchnorm_fp32 : None
master_weights : None
loss_scale : dynamic
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled : True
opt_level : O1
cast_model_type : None
patch_torch_functions : True
keep_batchnorm_fp32 : None
master_weights : None
loss_scale : dynamic
Warning: multi_tensor_applier fused unscale kernel is unavailable, possibly because apex was installed without --cuda_ext --cpp_ext. Using Python fallback. Original ImportError was: ModuleNotFoundError("No module named 'amp_C'")
06/14/2022 22:26:05 - INFO - main - ***** Running training *****
06/14/2022 22:26:05 - INFO - main - Num examples = 1218
06/14/2022 22:26:05 - INFO - main - Num Epochs = 2
06/14/2022 22:26:05 - INFO - main - Instantaneous batch size per GPU = 48
06/14/2022 22:26:05 - INFO - main - Total train batch size (w. parallel, distributed & accumulation) = 384
06/14/2022 22:26:05 - INFO - main - Gradient Accumulation steps = 1
06/14/2022 22:26:05 - INFO - main - Total optimization steps = 8
Epoch: 0%| | 0/2 [00:00<?, ?it/s]06/14/2022 22:26:05 - INFO - main -
[Epoch 1]
06/14/2022 22:26:05 - INFO - main - Initialize pre-batch of size 2 for Epoch 1
raceback (most recent call last): | 0/4 [00:00<?, ?it/s]
File "train_rc.py", line 593, in
main()
File "train_rc.py", line 537, in main
global_step, tr_loss = train(args, train_dataset, model, tokenizer)
File "train_rc.py", line 222, in train
outputs = model(**inputs)
File "/ssd3/wangxiao/anaconda3/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/ssd3/wangxiao/anaconda3/envs/py38/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 168, in forward
outputs = self.parallel_apply(replicas, inputs, kwargs)
File "/ssd3/wangxiao/anaconda3/envs/py38/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 178, in parallel_apply
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
File "/ssd3/wangxiao/anaconda3/envs/py38/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 86, in parallel_apply
output.reraise()
File "/ssd3/wangxiao/anaconda3/envs/py38/lib/python3.8/site-packages/torch/_utils.py", line 425, in reraise
raise self.exc_type(msg)
StopIteration: Caught StopIteration in replica 0 on device 0.
Original Traceback (most recent call last):
File "/ssd3/wangxiao/anaconda3/envs/py38/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
output = module(*input, **kwargs)
File "/ssd3/wangxiao/anaconda3/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/ssd1/zhangyiming/densephrase/DensePhrases/densephrases/encoder.py", line 132, in forward
start, end = self.embed_phrase(input_ids, attention_mask, token_type_ids)
File "/ssd1/zhangyiming/densephrase/DensePhrases/densephrases/encoder.py", line 94, in embed_phrase
outputs = self.phrase_encoder(
File "/ssd3/wangxiao/anaconda3/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/ssd3/wangxiao/anaconda3/envs/py38/lib/python3.8/site-packages/transformers/modeling_bert.py", line 707, in forward
attention_mask, input_shape, self.device
File "/ssd3/wangxiao/anaconda3/envs/py38/lib/python3.8/site-packages/transformers/modeling_utils.py", line 113, in device
return next(self.parameters()).device
StopIteration
Hi,
Thanks for the interesting project!
One question: If I want to get only phrase representations from your pre-trained model, how can I do that? I plan to use them as baselines. Thank you!
Best,
Jiacheng
I tried to run index-vecs using custom wikidump, dataset and model, but got this error
Modifying num_clusters flags to 96 doesn't seem to help, the k in error message is still 256.
I notice the EM score of densephrases on SQuAD is 78.3, and I want to reproduce this experiment but I cannot find a script available.
Hi! Thanks for this amazing work, and for making your code open-source.
I'm trying to figure out where in the code is non-unique passage retrieval handled that ensures that the final k results are unique. According to this footnote on page 3 in your paper "Phrase Retrieval Learns Passage Retrieval, Too", it seems that you perform iterative retrieval to achieve this. Could you point me to the code where this is happening?
Hi!
First of all thanks a lot for this solid project!
I just want to figure out how to extract phrases from Wikipedia? Which script is the right one?
I am a little confused when I see so many scripts in the preprocess folder.
Nice to meet you , I am a newcomer to NLP and I am very intersted in your code , but I am not good at coding ,I havd tried a week to find the code where to get the score of query embedding and the phrases embedding , I have found many score but do not know which is I need .
I would appreciate it if you could answer this question，thanks for your time !
eval_phrase_retrieval.zip
eval_phrase_retrieval.zip
Hi,
I am trying to make the dataset for natural questions. There seems to be a file missing according to the Makefile.
Is there any plan to including the file as another release?
Missing file scripts/preprocess/create_nq_reader.py
https://github.com/princeton-nlp/DensePhrases/blob/main/Makefile#L384
Hi,
Thanks for the amazing work! May I ask how do you choose the parameter for faiss index? Like the number of clusters and quant type OPQ96? It seems that the number of clusters varies with the number of phrases to save.
Thanks!
Last night when I tried to train with custom pre-trained model I found out that densephrases also use another pre-trained model for cross-encoder, which is spanbert by default($TEACHER_MODEL).
When I tried to change it with bert-base-cased it failed to run. Can I change this or not? If I can, which model from huggingface I could use as teacher model?
Hi there,
In this issue, you point out that you use span answers in the dataset as phrases for training, how about in the inference step when you have a new corpus, how do you choose phrases from a document to encode (you extract entities in the document as phrases or just any contiguous segment of text up to L words ?)
Hello!
Thanks for the amazing work on phrase retrieval. I was recently working on a question answering project and stumbled upon this work. Our project however involves non-answerable questions as in Squad2.0.
To prevent answering, I tried to threshold the score obtained in the meta data of the results, but it seemed that it was not well calibrated (since it is not trained on non-answerable sets)
Could you please guide me on how I could train this model to perform a good distinction between answerable and non-answerable questions. Otherwise, advice on properly using score to achieve this would also be great.
Hi,
I was going through the code for query finetuning and I am not able to understand one condition in the code:
Is the above highlighted line redundant and if not what is the significance (I feel we can directly update the encoder). Just wanted to make sure that I am not missing anything.
Thanks for sharing interesting paper and code.
As far as I understand, all the loss function used in the paper is implemented in encoder.py. However, it is unclear for me in which part in code address in-passage negtive (defined in section 4.1 of EMNLP21 paper). Could you provide me the related parts in code? Thank you.
Hi
While running this code :
python generate_phrase_vecs.py --model_type bert --pretrained_name_or_path SpanBERT/spanbert-base-cased --data_dir ./ --cache_dir $CACHE_DIR --predict_file sample/articles.json --do_dump --max_seq_length 512 --doc_stride 500 --fp16 --filter_threshold -2.0 --append_title --load_dir $SAVE_DIR/densephrases-multi --output_dir $SAVE_DIR/densephrases-multi_sample
I am getting below error:
File "generate_phrase_vecs.py", line 396, in
main()
File "generate_phrase_vecs.py", line 392, in main
dump_phrases(args, model, tokenizer)
File "generate_phrase_vecs.py", line 85, in dump_phrases
args, tokenizer, evaluate=True, output_examples=True, context_only=True
File "/home/neerajku/neerajAvatar/DensePhrases/densephrases/utils/squad_utils.py", line 1199, in load_and_cache_examples
context_only=context_only, args=args)
File "/home/neerajku/neerajAvatar/DensePhrases/densephrases/utils/squad_utils.py", line 792, in get_dev_examples
return self._create_examples(input_data, "dev", draft, context_only=context_only, args=args)
File "/home/neerajku/neerajAvatar/DensePhrases/densephrases/utils/squad_utils.py", line 805, in _create_examples
truecase = TrueCaser(os.path.join(os.environ['DATA_DIR'], args.truecase_path))
File "/home/neerajku/neerajAvatar/DensePhrases/densephrases/utils/squad_utils.py", line 1301, in init
with open(dist_file_path, "rb") as distributions_file:
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/neerajku/DensePhrases//densephrases-data/truecase/english_with_questions.dist'
Could you help me , how to resolve this
Hi,
What is the all in this command, I am getting unrecognized command error when i remove all.
python build_phrase_index.py \
$SAVE_DIR/densephrases-multi_sample/dump all \
--replace \
--num_clusters 32 \
--fine_quant OPQ96 \
--doc_sample_ratio 1.0 \
--vec_sample_ratio 1.0 \
--cuda
I corrected that by giving --dump_dir before but its not creating anything. Please find the screenshot below,
It seems that I cannot find the code to train the cross-encoder teacher models, such as
spanbert-base-cased-nq
spanbert-base-cased-sqdnq
spanbert-base-cased-squad
Can you tell me how to train these model If I change the train data?
Hi Jinhyuk,
I was trying to reproduce the third row of Table 1 in your paper (https://arxiv.org/pdf/2109.08133.pdf). I'm using the index and pre-trained ckpt on NQ you gave me several days ago. Here's my results:
Top-1 = 34.32%
Top-5 = 54.13%
Top-20 = 66.59%
Top-100 = 76.43%
Acc@1 when Acc@100 = 44.91%
MRR@20 = 43.12
P@20 = 14.61
Here's the command I use:
make eval-index-psg MODEL_NAME=densephrases-nq-query-nq DUMP_DIR=densephrases-nq_wiki-20181220-p100/dump/ TEST_DATA=open-qa/nq-open/test_preprocessed.json
Any idea what I might do wrong? Thanks in advance.
Minghan
Hello
thank you for you sharing code but i have question becuase the results of Nq-open and squad have error when i tested them
i changed the code of run demo to make evaultion for the test dataset without using server and using this commad line
python run_demo_edit.py --run_mode eval_request --index_port 51997 --test_path $DATA_DIR/open-qa/squad/test_preprocessed.json --eval_batch_size 64 --save_pred --truecase --cuda --index_name start/1048576_flat_OPQ96 --dump_dir $SAVE_DIR/densephrases-multi_wiki-20181220/dump/
the code work fine but the result is very bad i dont know what the reason
03/07/2023 10:46:24 - INFO - eval_phrase_retrieval - EM: 0.22, F1: 0.45
03/07/2023 10:46:24 - INFO - eval_phrase_retrieval - 1) Who got the first Nobel Prize in physics
03/07/2023 10:46:24 - INFO - eval_phrase_retrieval - => groundtruths: ['Wilhelm Conrad Röntgen'], top 5 prediction: ['G', 'He', 'Chad', 'W', 'Davis']
03/07/2023 10:46:24 - INFO - eval_phrase_retrieval - 2) When is the next Deadpool movie being released
03/07/2023 10:46:24 - INFO - eval_phrase_retrieval - => groundtruths: ['May 18 , 2018'], top 5 prediction: ['214', '[', '2099', 'Pet', '2010']
03/07/2023 10:46:24 - INFO - eval_phrase_retrieval - 3) Which mode is used for short wave broadcast service
03/07/2023 10:46:24 - INFO - eval_phrase_retrieval - => groundtruths: ['MFSK', 'Olivia'], top 5 prediction: ['98', 'O', '136', "Schöbel's", '990",']
03/07/2023 10:46:26 - INFO - eval_phrase_retrieval - {'exact_match_top1': 0.22160664819944598, 'f1_score_top1': 0.4542936288088642}
03/07/2023 10:46:26 - INFO - eval_phrase_retrieval - {'exact_match_top10': 0.9418282548476454, 'f1_score_top10': 2.4090489381348115}
03/07/2023 10:46:26 - INFO - eval_phrase_retrieval - {'redundancy of top10': 1.4556786703601108}
and i notice that the prediction is only one word here is an example of the predication and true label
Bond ['lament on various worldwide problems']
mon ['peptide bond']
1943 ['1952']
Shahi ['Akbar the Great', 'Babur']
some time give one character
here is my edit code
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0,1"
import json
import argparse
import torch
import os
import random
import numpy as np
import requests
import logging
from tqdm import tqdm
from time import time
from flask import Flask, request, jsonify, render_template, redirect
from flask_cors import CORS
from tornado.wsgi import WSGIContainer
from tornado.httpserver import HTTPServer
from tornado.ioloop import IOLoop
from requests_futures.sessions import FuturesSession
from eval_phrase_retrieval import evaluate_results, evaluate_results_kilt
from densephrases.utils.single_utils import load_encoder
from densephrases.utils.data_utils import TrueCaser
from densephrases.utils.open_utils import load_phrase_index, load_cross_encoder, load_qa_pairs, get_query2vec
from densephrases import Options
from concurrent.futures import Future
logging.basicConfig(format='%(asctime)s - %(levelname)s - %(name)s - %(message)s', datefmt='%m/%d/%Y %H:%M:%S',
level=logging.INFO)
logger = logging.getLogger(__name__)
class DensePhrasesDemo(object):
def __init__(self, args, inmemory=False, batch_size=64, query_encoder=None, tokenizer=None):
self.args = args
self.base_ip = args.base_ip
self.query_port = args.query_port
self.index_port = args.index_port
self.truecase = TrueCaser(os.path.join(os.environ['DATA_DIR'], args.truecase_path))
self.args.examples_path = os.path.join('densephrases/demo/static', args.examples_path)
self.mips = load_phrase_index(args)
self.device = 'cuda' if args.cuda else 'cpu'
if query_encoder is None:
self.query_encoder, self.tokenizer, _ = load_encoder(self.device, args, query_only=True)
self.query2vec = get_query2vec(
query_encoder=self.query_encoder, tokenizer=self.tokenizer, args=args, batch_size=batch_size
)
def query2vec_api(self,x):
batch_query = json.loads(x['query'])
#print(batch_query)
# start_time = time()
outs = list(self.query2vec(batch_query))
#print(outs)
# logger.info(f'query2vec {time()-start_time:.3f} for {len(batch_query)} queries: {batch_query[0]}')
#outs = json.dumps(outs)
return outs #json.loads(outs)#jsonify(outs)
def api(self,x):
query = x['query']
query = query[:-1] if query.endswith('?') else query
if args.truecase:
if query[1:].lower() == query[1:]:
query = self.truecase.get_true_case(query)
out = batch_search(
[query],
max_answer_length=args.max_answer_length,
top_k=args.top_k,
nprobe=args.nprobe,
)
out['ret'] = out['ret'][0]
#outs = json.loads(json.dumps(outs))
return outs
#return jsonify(out)
def get_examples():
with open(args.examples_path, 'r') as fp:
examples = [line.strip() for line in fp.readlines()]
examples = json.loads(json.dumps(examples))
return #jsonify(examples)
def batch_search(self, batch_query, max_answer_length=20, top_k=10,
nprobe=64, return_vecs=False):
t0 = time()
outs, _ = self.embed_query(batch_query)
start = np.concatenate([out[0] for out in outs], 0)
end = np.concatenate([out[1] for out in outs], 0)
query_vec = np.concatenate([start, end], 1)
rets = self.mips.search(
query_vec, q_texts=batch_query, nprobe=nprobe,
top_k=top_k, max_answer_length=max_answer_length,
return_vecs=return_vecs, aggregate=True,
)
for ret_idx, ret in enumerate(rets):
for rr in ret:
rr['query_tokens'] = outs[ret_idx][2]
t1 = time()
out = {'ret': rets, 'time': int(1000 * (t1 - t0))}
return out
def batch_api(self, post_data):
batch_query = json.loads(post_data['query'])
max_answer_length = int(post_data['max_answer_length'])
top_k = int(post_data['top_k'])
nprobe = int(post_data['nprobe'])
out = self.batch_search(
batch_query,
max_answer_length=max_answer_length,
top_k=top_k,
nprobe=nprobe,
)
return out #json.loads(json.dumps(out)) #jsonify(out)
def get_address(self, port):
assert self.base_ip is not None and len(port) > 0
return self.base_ip + ':' + port
"""def embed_query(self, batch_query):
#emb_session = FuturesSession()
data={'query': json.dumps(batch_query)}
r = self.query2vec_api(data) #emb_session.post(self.get_address(self.query_port) + '/query2vec_api',data={'query': json.dumps(batch_query)})
print(type(r))
def map_():
result = r.result()
emb = result.json()
return emb, result.elapsed.total_seconds() * 1000
return map_"""
def embed_query(self, batch_query):
data = {'query': json.dumps(batch_query)}
r = self.query2vec_api(data)
return r,0
def query(self, query):
params = {'query': query}
res = requests.get(self.get_address(self.index_port) + '/api', params=params)
try:
outs = json.loads(res)
except Exception as e:
logger.info(f'no response or error for q {query}')
logger.info(res)
return outs
def batch_query(self, batch_query, batch_context=None, max_answer_length=20, top_k=10, nprobe=64):
post_data = {
'query': json.dumps(batch_query),
'context': json.dumps(batch_context) if batch_context is not None else json.dumps(batch_query),
'max_answer_length': max_answer_length,
'top_k': top_k,
'nprobe': nprobe,
}
res = self.batch_api(post_data)#requests.post(self.get_address(self.index_port) + '/batch_api', data=post_data)
"""if res.status_code != 200:
logger.info('Wrong behavior %d' % res.status_code)"""
try:
outs = json.loads(res)
except Exception as e:
logger.info(f'no response or error for q {batch_query}')
logger.info(res)
return res
def eval_request(self, args):
# Load dataset
qids, questions, answers, _ = load_qa_pairs(args.test_path, args)
# Run batch_query and evaluate
step = args.eval_batch_size
predictions = []
evidences = []
titles = []
scores = []
start_time = None
num_q = 0
for q_idx in tqdm(range(0, len(questions), step)):
if q_idx >= 5*step: # exclude warmup
if start_time is None:
start_time = time()
num_q += len(questions[q_idx:q_idx+step])
result = self.batch_query(
questions[q_idx:q_idx+step],
max_answer_length=args.max_answer_length,
top_k=args.top_k,
nprobe=args.nprobe,
)
prediction = [[ret['answer'] for ret in out] if len(out) > 0 else [''] for out in result['ret']]
evidence = [[ret['context'] for ret in out] if len(out) > 0 else [''] for out in result['ret']]
title = [[ret['title'] for ret in out] if len(out) > 0 else [''] for out in result['ret']]
score = [[ret['score'] for ret in out] if len(out) > 0 else [-1e10] for out in result['ret']]
predictions += prediction
evidences += evidence
titles += title
scores += score
logger.info(f'{time()-start_time:.3f} sec for {num_q} questions => {num_q/(time()-start_time):.1f} Q/Sec')
eval_fn = evaluate_results if not args.is_kilt else evaluate_results_kilt
eval_fn(
predictions, qids, questions, answers, args, evidences=evidences, scores=scores, titles=titles,
)
if __name__ == '__main__':
# See options in densephrases.options
options = Options()
options.add_model_options()
options.add_index_options()
options.add_retrieval_options()
options.add_data_options()
options.add_demo_options()
args = options.parse()
# Seed for reproducibility
random.seed(args.seed)
np.random.seed(args.seed)
torch.manual_seed(args.seed)
if torch.cuda.is_available():
torch.cuda.manual_seed_all(args.seed)
server = DensePhrasesDemo(args)
if args.run_mode == 'q_serve':
logger.info(f'Query address: {server.get_address(server.query_port)}')
server.serve_query_encoder(args.query_port, args)
elif args.run_mode == 'p_serve':
logger.info(f'Index address: {server.get_address(server.index_port)}')
server.serve_phrase_index(args.index_port, args)
elif args.run_mode == 'query':
query = 'Name three famous writers'
result = server.query(query)
logger.info(f'Answers to a question: {query}')
logger.info(f'{[r["answer"] for r in result["ret"]]}')
elif args.run_mode == 'batch_query':
queries = [
'Where',
'when did medicare begin in the united states',
'who sings don\'t stand so close to me',
'Name three famous writers',
'Who was defeated by computer in chess game?'
]
contexts = [
'Uncle jesse\'s original full name was James Lee. And he was born in South Korea.',
'Uncle jesse\'s original name was James Lee. And he was born in South Korea in 1333. US medicare started in 1222.',
'Uncle jesse\'s original name was James Lee. And he sang this song.',
'Uncle jesse\'s original name was James Lee. And Jens wrote this novel.',
'Uncle jesse\'s original name was James Lee. The man was defeated by Alphago.'
]
result = server.batch_query(
queries,
contexts, # feed context for cross encoders
max_answer_length=args.max_answer_length,
top_k=args.top_k,
nprobe=args.nprobe,
)
for query, re in zip(queries, result['ret'].values()):
logger.info(f'Answers to a question: {query}')
logger.info(f'{re}')
elif args.run_mode == 'eval_request':
print("sadadadada")
server.eval_request(args)
else:
raise NotImplementedError
Hi!
First of all thanks a lot for this amazing project!!
When going through the repo I have noticed some missing scripts in the scripts\preprocess folder. Namely:
build_db.pyprep_wikipedia.pybuild_wikisquad.pyThanks in advance :)
Hi, First of all many thanks for work.
I am trying to test this.
As per documentation I downloaded all 4 tar files (datasets, wikipediadump, pretrained models and phrase index). but while running getting the below mentioned error:
which seems to be finding some phrase folder in wikidump, which is not available at all.
Can u suggest the reason for same.
I have given correct path for all folders.
Thanks for the great work! I have read the paper and understood the Gist of the approach but I am out-of-depth on few things so I wanted to clarify few things here and get some suggestion.
My problem statement is straight-forward, I have thousands of documents as context and I need to retrieve an answer for query in real-time. I came across your approach and I wanted to try it out and possibly adapt this in industry setting. One other constrain is that the context grows, that is, new documents get added on and the retrieval needs to support that.
Do you think your approach is suitable for this use-case?
Thanks in Advance
I've built the compressed DensePhrase index on SQuAD using OPQ96. I haven't run any query-side finetuning yet but here are the results:
11/22/2021 19:50:57 - INFO - main - no_ans/all: 0, 10570
11/22/2021 19:50:57 - INFO - main - Evaluating 10570 answers
11/22/2021 19:50:58 - INFO - main - EM: 21.63, F1: 27.96
11/22/2021 19:50:58 - INFO - main - 1) Which NFL team represented the AFC at Super Bowl 50
11/22/2021 19:50:58 - INFO - main - => groundtruths: ['Denver Broncos', 'Denver Broncos', 'Denver Broncos'], top 5 prediction: ['Denver Broncos', 'Pittsburgh Steelers', 'Pittsburgh Steelers', 'Pittsburgh Steelers', 'Pittsburgh Steelers']
11/22/2021 19:50:58 - INFO - main - 2) Which NFL team represented the NFC at Super Bowl 50
11/22/2021 19:50:58 - INFO - main - => groundtruths: ['Carolina Panthers', 'Carolina Panthers', 'Carolina Panthers'], top 5 prediction: ['San Francisco 49ers', 'Chicago Bears', 'Seattle Seahawks', 'Tampa Bay Buccaneers', 'Green Bay Packers']
11/22/2021 19:50:58 - INFO - main - 3) Where did Super Bowl 50 take place
11/22/2021 19:50:58 - INFO - main - => groundtruths: ['Santa Clara, California', "Levi's Stadium", "Levi's Stadium in the San Francisco Bay Area at Santa Clara, California."], top 5 prediction: ['Tacoma, Washington, USA', "Levi's Stadium in Santa Clara, California", 'DeVault Vineyards in Concord, Virginia', "Levi's Stadium in Santa Clara", 'Jinan Olympic Sports Center Gymnasium in Jinan, China']
11/22/2021 19:53:44 - INFO - main - {'exact_match_top1': 21.62724692526017, 'f1_score_top1': 27.958255585698414}
11/22/2021 19:53:44 - INFO - main - {'exact_match_top200': 57.48344370860927, 'f1_score_top200': 73.28679644685603}
11/22/2021 19:53:44 - INFO - main - {'redundancy of top200': 5.308987701040681}
11/22/2021 19:53:44 - INFO - main - Saving prediction file to .//outputs/densephrases-squad-ddp/pred/test_preprocessed_10570_top200.pred
10570it [00:23, 448.84it/s]
11/22/2021 19:54:58 - INFO - main - avg psg len=124.84 for 10570 preds
11/22/2021 19:54:58 - INFO - main - dump to .//outputs/densephrases-squad-ddp/pred/test_preprocessed_10570_top200_psg-top100.json
ctx token length: 124.84
unique titles: 98.20
Top-1 = 27.02%
Top-5 = 42.80%
Top-20 = 56.40%
Top-100 = 69.20%
Acc@1 when Acc@100 = 39.05%
MRR@20 = 34.30
P@20 = 8.94
I understand that index compression results in accuracy loss w/o query-side finetuning. However, the score still looks a little bit too low to me. Could @jhyuklee confirm whether this looks alright?
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.