I really like your PICK project and want to thank you for sharing it, also I have a question about the code.

I am currently trying to export the prediction results from a trained model of PICK using the test.py script, but it just shows the predictions defined on the transcript of the .tsv file of the image provided, it doesn't export the other predictions, so my question is: 
Where are the predictions stored? I tried looking through the code but didn't manage to find the variable that contains them. I would be very grateful if you could give some insight about this.

A funny thing is that the script test.py just prints whatever I put in the .tsv, I replaced the transcripts of a .tsv with "asda sdas" and on the output folder got the same "asda sdas" on the .txt

Regards.

Hi is it possible to extract checkbox value. Like in document for gender there is checkbox male and female so how can we extract that. I am wondering that it can be done because google https://cloud.google.com/solutions/document-ai is able to identify checkbox and its key.

Hi Team,

Amazing work. Thank you.
Is it possible to train on DocBank dataset and use custom dataset to do fine tuning or transfer learning?
Because we have limited custom dataset which is around 200 to 300 pdf documents. Is this data enough to train from scratch or is it possible to use DocBank trained model as base model?

Thank you

While testing with out-of-sample images, i tried creating boxes_and_transcripts for new images using tesseract & prepare tsv files. but when i'm predicting output is not good as in-sample testing images.

kindly please let us know way to create bounding box files (tsv) for prediction for this model.

Also can we predict without tsv files?

Note: In tesseract i tried both WORD & TEXTLINE bounding boxes

Hi there. I have trained the model using ICDAR data and while using the checkpoint to train it on my own data with a different number of entities I have been getting the following error:

RuntimeError: Error(s) in loading state_dict for PICKModel:
size mismatch for decoder.bilstm_layer.mlp.mlp.0.weight: copying a param with shape torch.Size([11, 1024]) from checkpoint, the shape in current model is torch.Size([15, 1024]).
size mismatch for decoder.bilstm_layer.mlp.mlp.0.bias: copying a param with shape torch.Size([11]) from checkpoint, the shape in current model is torch.Size([15]).
size mismatch for decoder.crf_layer.transitions: copying a param with shape torch.Size([11, 11]) from checkpoint, the shape in current model is torch.Size([15, 15]).
size mismatch for decoder.crf_layer._constraint_mask: copying a param with shape torch.Size([13, 13]) from checkpoint, the shape in current model is torch.Size([17, 17]).
size mismatch for decoder.crf_layer.start_transitions: copying a param with shape torch.Size([11]) from checkpoint, the shape in current model is torch.Size([15]).
size mismatch for decoder.crf_layer.end_transitions: copying a param with shape torch.Size([11]) from checkpoint, the shape in current model is torch.Size([15]).

Does it mean that we can't train from the checkpoint? Is there any other way to train if from checkpoints using data with a different number of entities?
Thanks.

boxes_and_transcripts 每张图片对应一个 .tsv标签
在说明文档中, 每个标签都是如下格式:
index, box_coordinates, transcripts, box_entity_types
在给的两个例子中, index标签有矛盾
X00016469623.tsv 标签中所有的index都是1
asdf.tsv 标签中index又是递增的, 对应的是行数
所以这个 index 到底是指标签对应图片在train_sample_list.csv 文件中的 index, 还是指行数？

Hi man.
Your model very useful with me. I appreciate it, but length of output decoder just 50 character, how can i change it.

Is it possible to get the bounding box coordinates for each predicted labels in test set?

另外，支持不同排版的数据在同一个集合里训练吗？比如发票、收据、火车票在一个训练集里

Hi All

I have created training set which have around 64 tags (entities)
Which are the key configurations need to be changed for better result inside config.json

Hello,

In data/Readme.md, you say « if iob_tagging_type is set to box_level, this folder will not be used, then box_entity_types in file_name.tsv file of boxes_and_transcripts folder will be used as label of entity. otherwise, it must be provided.«  about the entities folder. So you mean entities folder is not used during evaluation? But how can you evaluate model without knowing the ground truth for entities? Or I don’t understand something

Thanks

是因为训练集不够，增加数据考虑吗？谢谢！

Hi,

Does anyone met this problem during distributed training?

subprocess.CalledProcessError: Command '['/opt/conda/bin/python', '-u', 'train.py', '--local_rank=3', '-c', 'config.json', '--local_world_size', '4']' returned non-zero exit status 1.

PLS give me some adivce, Tks

Hey, I saw you are using keys.txt file for encoding data. if i am wrong please correct me.

1. If you are using keys.txt then how you are making it and if any word is not in training data then it can be handled or not.
2. I have to train on another language then how can i do?

When testing, do we have to provide boxes and texts(scripts?) of the test images?

@wenwenyu @tengerye
Hey can you please upload pretrained weights of docbank dataset. Because this dataset is huge and i am using colab so i am not able to retrain this one. If you able to do it please it would be very helpful.

Hi wenwenyu,

It appears that the padding in the T dimension is not considered for x_gcn before adding with new_x.

Can you help confirm this?

Thanks

[2020-08-11 15:12:01,468 - trainer - INFO] - Train Epoch:[4/100] Step:[560/666] Total Loss: 0.163740 GL_Loss: 0.163414 CRF_Loss: 0.000326
[2020-08-11 15:13:22,293 - trainer - INFO] - Train Epoch:[4/100] Step:[570/666] Total Loss: 0.177532 GL_Loss: 0.177369 CRF_Loss: 0.000163
[2020-08-11 15:14:42,824 - trainer - INFO] - Train Epoch:[4/100] Step:[580/666] Total Loss: 0.171964 GL_Loss: 0.171639 CRF_Loss: 0.000326
[2020-08-11 15:16:05,332 - trainer - INFO] - Train Epoch:[4/100] Step:[590/666] Total Loss: 0.168039 GL_Loss: 0.167388 CRF_Loss: 0.000651
[2020-08-11 15:17:26,160 - trainer - INFO] - Train Epoch:[4/100] Step:[600/666] Total Loss: 0.184481 GL_Loss: 0.184319 CRF_Loss: 0.000163
[2020-08-11 15:24:51,474 - trainer - INFO] - [Step Validation] Epoch:[4/100] Step:[600/666]
+---------+-------+-------+-------+-------+
| name | mEP | mER | mEF | mEA |
+=========+=======+=======+=======+=======+
| overall | 0 | 0 | 0 | 0 |
+---------+-------+-------+-------+-------+
怎么各项指标都为0呢？是不是哪里设置不正确呢？

Chinese receipt may appear multi entities because of printing.

Seems like PICK fails to converge with increased amount of BiLSTM layers.

Changed: bilstm_kwargs "num_layers" from 2 to 4.

With num_layers = 2 it converges in 6 epochs, once I tried layers = 4, it fails to converge in 42 epochs.
Is the problem in hyper-parameters?

num_layers = 4, epoch = 42:

num_layers = 2, epoch = 6 (Converged):

Hi, that's a good work on IE, thanks.

Currently, I've tested your code on SROIE with the "document_level" setting. I used the OCR results to train and test the model, which downloaded from SROIE official website and extracted from the folder "task1&2_train(626p)" and "taks1&2_text_test(361p)". The performance looks not good as yours. May I ask how you prepared your dataset. Did you use other OCR tools to preprocess the dataset.

Thank you very much.

Hi, in one of the provided annotation file,

{
    "company": "BOOK TA .K (TAMAN DAYA) SDN BHD",
    "date": "25/12/2018",
    "address": "NO.53 55,57 & 59, JALAN SAGU 18, TAMAN DAYA, 81100 JOHOR BAHRU, JOHOR.",
    "total": "9.00"
}

I am wondering if a document has multiple entities of the same type, e.g., several companies' names, is the model able to find them all? Such as ["company": "company1", "company": "company2"].

If the model supports such case, how shall I prepare the data in entities folder?

Greetings, I wanted to point a conflict about the data format of training and test data:

The readme says that the direction of the coordinates of the bounding boxes must be clockwise, but looking at the examples given in the X00016469623.tsv file of this repo, the coordinates are counterclockwise.

1,83,41,331,41,331,78,83,78,TAN WOON YANN,other
1,109,171,330,171,330,191,109,191,MR D.I.Y. (M) SDN BHD,company

能否对config.json中的配置参数进行解释？！

请问训练过程中需要多少样本量，每个类别大概多少个图片？谢谢

# update hidden features between nodes, (B, N, N, in_dim） H = F.relu(x_i + x_j + alpha + self.bias_h)

Hi, thank you for your great job. I have some questions about your data file in your repository. In asdf.tsv file,
in the 5th line TAMAN DAYA, is marked as other, shouldn't it be address? And there is no total tag in the label file.

Looking forward to your reply.

照着readme用的
CUDA_VISIBLE_DEVICES=1,2 python -m torch.distributed.launch --nnode=1 --node_rank=0 --nproc_per_node=4
--master_addr=127.0.0.1 --master_port=5555
train.py -c config.json --local_world_size 2
报标题错误
是config里也要作对应改动吗？如何改。谢谢

Is it possible to train this model on colab? I have a small dataset.

Hi,

I was just exploring this model and found it useful for information extraction from scanned images. I need to know how can we make predictions on unlabelled data after we have trained the model?

By unlabelled data, I mean images where we do not have bounding box details.

Thanks in advance.

Hi, thank you for your great job.

May I ask two questions please?

1. Did you use external data for SROIE 2019 competition?
2. How did you evaluate on SROIE dataset?

whether already have a trained pytorch model or not?

Hi, kindly please suggest dataset preparation for extraction each item in table in invoice

I trained model for a while & cancelled training
Now i'm trying it test. i'm getting below issue. kindly please help. thanks in advance

Command:
!python test.py --checkpoint saved/models/PICK_Default/test_1012_064804/model_best.pth
--boxes_transcripts {out_box_path}
--images_path {out_img_path} --output_folder /content/output/
--gpu 0 --batch_size 2

Error:
Loading checkpoint: saved/models/PICK_Default/test_1012_064804/model_best.pth
with saved mEF 0.0000 ...

RuntimeError: Error(s) in loading state_dict for PICKModel:
size mismatch for decoder.bilstm_layer.mlp.mlp.0.weight: copying a param with shape torch.Size([49, 1024]) from checkpoint, the shape in current model is torch.Size([11, 1024]).
size mismatch for decoder.bilstm_layer.mlp.mlp.0.bias: copying a param with shape torch.Size([49]) from checkpoint, the shape in current model is torch.Size([11]).
size mismatch for decoder.crf_layer.transitions: copying a param with shape torch.Size([49, 49]) from checkpoint, the shape in current model is torch.Size([11, 11]).
size mismatch for decoder.crf_layer._constraint_mask: copying a param with shape torch.Size([51, 51]) from checkpoint, the shape in current model is torch.Size([13, 13]).
size mismatch for decoder.crf_layer.start_transitions: copying a param with shape torch.Size([49]) from checkpoint, the shape in current model is torch.Size([11]).
size mismatch for decoder.crf_layer.end_transitions: copying a param with shape torch.Size([49]) from checkpoint, the shape in current model is torch.Size([11]).

In your `documents.py' file, there is a snippet:

def sort_box_with_list(data: List[Tuple], left_right_first=True):
    def compare_key(x):
        #  x is (index, points, transcription, type) or (index, points, transcription)
        points = x[1]
        box = np.array([[points[0], points[1]], [points[2], points[3]], [points[4], points[5]], [points[6], points[7]]],
                       dtype=np.float32)
        rect = cv2.minAreaRect(box)  # Smallest rectangle around the box.
        # rect: center(x,y), (width, height), angle of rotation
        center = rect[0]
        if left_right_first:
            return center[1], center[0]
        else:
            return center[0], center[1]

    data = sorted(data, key=compare_key)
    return data

I think if you want left_right_first, you shall return center[0], center[1] instead of center[1], center[0]. Since center=(x, y).

Hi wenwenyu,

I am really interested in this work. I am just wondering is it possible for me to train it in a non-distributed way?

Thanks a lot!

Hi, @wenwenyu, is your code possible to output the box-level tags instead of entity_name-text pair please?

Hi, I encountered a problem with your code:

AssertionError: DistributedDataParallel device_ids and output_device arguments only work
with single-device CUDA modules, but got device_ids [0], output_device 0, and
module parameters {device(type='cuda', index=0), device(type='cpu')}.

But if I commented

if self.config['trainer']['sync_batch_norm']:
    self.model = torch.nn.SyncBatchNorm.convert_sync_batchnorm(self.model)

of your trainer.py, the code runs without errors. I used the recommended settings.
May I ask why?

Hi wenwenyu,

Is it possible to get score in the form of probabilities?

Output for file 004.txt:

address	LOT 1851-A & 1851-B\, JALAN KPB 6,addressKAWASAN PERINDUSTRIAN BALAKONG,address43300 SERI KEMBANGAN\, SELANGOR,address
address	WA45 /2A - 12,other1,other
address	WA44-A - 12,other1
address	9555916500126,othe
address	43-A - 24,other1,other11.23,other079567600084,otherX,other11.23,other
address	090822,othe
address	.91,other
date	18-11-18

Can you provide some intuition or resource how to post preprocess it so as to store it in key and value form in json format?

看了下最后输出关键词及类别的过程，好像是将全部文字组成了一个长序列，然后在里头提取出了各字的标签，再将连续同标签的字组成知名并打个此标签。这个过程中OCR部分识别为一个box里的词组的前后部分都可能被分配到其他组。
比如OCR出了三块"XXX","YYY","ZZZ“，序列标注的拆分结果可能变成了"XX","XYY","YZZ","Z"。
假设OCR的BOX组合的过程准确率很高，那如何更好利用这部分信息？

Hello,
thank you so much for providing code for great research paper. I am just wondering after I run python train.py --config config.json. All the values of mEP, mER, mEF, mEA are all 0 for every train Epoch. Is this normal?
2020-07-26 20:45:35.357166: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
[2020-07-26 20:45:40,891 - train - INFO] - Model trainable parameters: 68575842
[2020-07-26 20:45:40,892 - train - INFO] - Train datasets: 8 samples Validation datasets: 8 samples Max_epochs: 100 Log_per_step: 10 Validation_per_step: 50
[2020-07-26 20:45:40,892 - train - INFO] - Training start...
[2020-07-26 20:46:07,680 - trainer - INFO] - Train Epoch:[1/100] Step:[2/2] Total Loss: 1121.992798 GL_Loss: 8.390446 CRF_Loss: 1113.602295
[2020-07-26 20:46:09,988 - trainer - INFO] - [Epoch Validation] Epoch:[1/100] Total Loss: 1430.095032 GL_Loss: 0.091890 CRF_Loss: 1420.906006
+---------+-------+-------+-------+-------+
| name | mEP | mER | mEF | mEA |
+=========+=======+=======+=======+=======+
| date | 0 | 0 | 0 | 0 |
+---------+-------+-------+-------+-------+
| name | 0 | 0 | 0 | 0 |
+---------+-------+-------+-------+-------+
| overall | 0 | 0 | 0 | 0 |
+---------+-------+-------+-------+-------+
Also,
I ran

python test.py --checkpoint /content/PICK-pytorch/saved/models/PICK_Default/test2_0726_194721/model_best.pth --boxes_transcripts /content/PICK-pytorch/data/test_data_example/boxes_and_transcripts \
               --images_path /content/PICK-pytorch/data/test_data_example/images/ --output_folder /content/PICK-pytorch/output/test2 \
               --gpu 0 --batch_size 2

Output files are generated, but the files are actually empty. Can you please give me the right direction to test the model? Thank you!

Hi, I have the following entities I want to extract. During the training, some entities are not shown in the table and keep disappearing. When I restart the training process, the number of entities shown in the metric table is also different. It gives me an inconsistent result. Do you know what I can do to fix it?

Entities_list = [
"invoice_date",
"tax_rate",
"subtotal",
"total"
]
For example,

+--------------+-------+-------+-------+-------+
| name | mEP | mER | mEF | mEA |
+==============+=======+=======+=======+=======+
| total | 0 | 0 | 0 | 0 |
+--------------+-------+-------+-------+-------+
| invoice_date | 0 | 0 | 0 | 0 |
+--------------+-------+-------+-------+-------+
| overall | 0 | 0 | 0 | 0 |
+--------------+-------+-------+-------+-------+

+--------------+-------+-------+-------+-------+
| name | mEP | mER | mEF | mEA |
+==============+=======+=======+=======+=======+
| invoice_date | 0 | 0 | 0 | 0 |
+--------------+-------+-------+-------+-------+
| overall | 0 | 0 | 0 | 0 |

Hi wenwenyu,

Thanks this wonderful repo firstly.

One quick question, have you tried the documents/forms with variable layout?
I just wonder if GCN can still help and perform well if the (relative) positional features are not consistent.

Thanks.

Hi Wenwen,
In the training data set, I noticed a bounding box includes more than one word. Which program did you use to merge nearby bounding boxes based on distance? Currently, I have a bounding box around each word. Will the accuracy be affected if the bounding boxes are merged together? Thank you!

Is there a pretrained checkponit to use?