herobd / handwriting_line_generation Goto Github PK

Here are the steps I took:

input
python3 generate.py -c ./IAM_weights/IAMslant_noMask_charSpecSingleAppend_GANMedMT_autoAEMoPrcp2tightNewCTCUseGen_balB_hCF0.75_sMG/checkpoint-iteration175000.pth -d ./output1/out/jpg -s ./IAM_weights/IAMslant_noMask_charSpecSingleAppend_GANMedMT_autoAEMoPrcp2tightNewCTCUseGen_balB_hCF0.75_sMG/test_styles_175000.pkl

output
indexes/random interp/vae random/strech/author display/math/turk gen/from-to/umap-images/Random styles/help/quit?
input
from-to

output
image path 1?
input
./h1.jpg
output
image path 2?
input
./h2.jpg

output
text to generate?
input
hello!

output
indexes/random interp/vae random/strech/author display/math/turk gen/from-to/umap-images/Random styles/help/quit?

Hi, thanks a lot your efforts. It's a greatwork. I was trying to train your model with CVL dataset on Google Colab Pro. I've transform the image format from .tif to .png and feed to the model. But after 200 iterations, the error Cuda out of memory appears. This is the output:
`NumExpr defaulting to 4 threads.
/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py:481: UserWarning: This DataLoader will create 6 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary. cpuset_checked))
/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py:481: UserWarning: This DataLoader will create 6 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary. cpuset_checked))
model style_extrator Trainable parameters: 10136912 HWWithStyle( (hwr): CNNOnlyHWR( (cnn): Sequential( (conv0): Conv2d(1, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (relu0): ReLU(inplace=True) (pooling0): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False) (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (relu1): ReLU(inplace=True) (pooling1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False) (conv2): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (batchnorm2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu2): ReLU(inplace=True) (conv3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (relu3): ReLU(inplace=True) (pooling2): MaxPool2d(kernel_size=(2, 2), stride=(2, 1), padding=(0, 1), dilation=1, ceil_mode=False) (conv4): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (batchnorm4): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu4): ReLU(inplace=True) (conv5): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1)) (relu5): ReLU(inplace=True) (pooling3): MaxPool2d(kernel_size=(2, 2), stride=(2, 1), padding=(0, 1), dilation=1, ceil_mode=False) (conv6): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1)) (batchnorm6): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu6): ReLU(inplace=True) ) (cnn1d): Sequential( (0): Conv1d(512, 512, kernel_size=(3,), stride=(1,), padding=(2,), dilation=(2,)) (1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) (3): Conv1d(512, 512, kernel_size=(3,), stride=(1,), padding=(4,), dilation=(4,)) (4): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU(inplace=True) (6): Conv1d(512, 512, kernel_size=(3,), stride=(1,)) (7): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (8): ReLU(inplace=True) (9): Conv1d(512, 512, kernel_size=(3,), stride=(1,), padding=(8,), dilation=(8,)) (10): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (11): ReLU(inplace=True) (12): Conv1d(512, 80, kernel_size=(3,), stride=(1,)) (13): LogSoftmax(dim=1) ) ) )
Begin training
WARNING: upsampling image to fit size /usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at /pytorch/c10/core/TensorImpl.h:1156.) return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
WARNING: upsampling image to fit size
WARNING: upsampling image to fit size
WARNING: upsampling image to fit size
WARNING: upsampling image to fit size
WARNING: upsampling image to fit size
Train iteration: 100, loss: 3.290166, recogLoss: 3.290166, CER: 1.000000, WER: 1.000000,
Train iteration: 200, loss: 2.875033, recogLoss: 2.875033, CER: 1.000000, WER: 1.000000, sec_per_iter: 0.623328, avg_loss: 3.082350, avg_recogLoss: 3.082350, avg_CER: 1.000000, avg_WER: 1.000000, Traceback (most recent call last):
File "train.py", line 133, in main(config, args.resume) File "train.py", line 79, in main trainer.train()
File "/content/drive/.shortcut-targets-by-id/1gLhWu0Me1satHwX83jnDJLd9nHAl9Mp6/mockproject/paper1/base/base_trainer.py", line 219, in train result = self._train_iteration(self.iteration) File "/content/drive/.shortcut-targets-by-id/1gLhWu0Me1satHwX83jnDJLd9nHAl9Mp6/mockproject/paper1/trainer/hw_with_style_trainer.py", line 378, in _train_iteration pred, recon, losses = self.run(instance) File "/content/drive/.shortcut-targets-by-id/1gLhWu0Me1satHwX83jnDJLd9nHAl9Mp6/mockproject/paper1/trainer/hw_with_style_trainer.py", line 736, in run pred = self.model.hwr(image, style) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/content/drive/.shortcut-targets-by-id/1gLhWu0Me1satHwX83jnDJLd9nHAl9Mp6/mockproject/paper1/model/cnn_only_hwr.py", line 131, in forward conv = self.cnn(input) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/container.py", line 139, in forward input = module(input) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py", line 443, in forward return self._conv_forward(input, self.weight, self.bias) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py", line 440, in _conv_forward self.padding, self.dilation, self.groups)
RuntimeError: CUDA out of memory. Tried to allocate 3.33 GiB (GPU 0; 15.90 GiB total capacity; 11.91 GiB already allocated; 3.02 GiB free; 11.97 GiB reserved in total by PyTorch)

I don't change anything in your architechture and loss functions.
Could you advice something me to fix this issue.
Thank you again
Sorry for my bad english.

Hi, Great work. I was searching for generating handwritten text which looks very similar to human and I found your repo. I tried to use it but I am getting error of xml. Do i have to train it on my system.

python generate.py -c /home/ravi/Downloads/RIMES_weights/RIME_weights/RIMESLines_hwr_cnnOnly_batchnorm_aug/checkpoint-iteration425000.pth -d /home/ravi/Downloads/Output
loaded iteration 425000
Traceback (most recent call last):
File "/home/ravi/Downloads/handwriting_line_generation-master/generate.py", line 906, in
main(args.checkpoint, args.savedir, gpu=args.gpu, config=args.config, addToConfig=addtoconfig, test =args.test,arguments=arguments, style_loc=args.style_loc)
File "/home/ravi/Downloads/handwriting_line_generation-master/generate.py", line 139, in main
data_loader, valid_data_loader = getDataLoader(config,'train')
File "/home/ravi/Downloads/handwriting_line_generation-master/data_loader/data_loaders.py", line 39, in getDataLoader
return withCollate(author_rimeslines_dataset.AuthorRIMESLinesDataset,author_rimeslines_dataset.collate,batch_size,valid_batch_size,shuffle,shuffleValid,numDataWorkers,split,data_dir,config)
File "/home/ravi/Downloads/handwriting_line_generation-master/data_loader/data_loaders.py", line 64, in withCollate
trainData = setObj(dirPath=data_dir, split='train', config=config['data_loader'])
File "/home/ravi/Downloads/handwriting_line_generation-master/datasets/author_rimeslines_dataset.py", line 145, in init
self.authors = parseXML(xml)
File "/home/ravi/Downloads/handwriting_line_generation-master/utils/parseRIMESlines.py", line 13, in getLineBoundaries
tree = ET.parse(xmlPath)
File "/home/ravi/anaconda3/envs/hgn/lib/python3.9/xml/etree/ElementTree.py", line 1229, in parse
tree.parse(source, parser)
File "/home/ravi/anaconda3/envs/hgn/lib/python3.9/xml/etree/ElementTree.py", line 569, in parse
source = open(source, "rb")
FileNotFoundError: [Errno 2] No such file or directory: '../data/RIMES/lines_training_2011.xml'

Hi,
Great work, thanks a lot for your efforts. I was trying to generate images similar to datasets other than IAM/RIMES (for instance Bentham or Washington datasets or maybe some custom datasets). From the top of my head, I could think of two ways,

1. I extract styles for these datasets (or some samples of images from them) and use the pretrained models (maybe the IAM one) provided to generate those images.
2. Retrain the entire model from scratch on these new databases and use that model to generate new images.

Which one would you suggest will work better if at all?
Thanks again.

Hi. How would I run this on one person's handwriting? I have images of hundreds of pages of the handwriting of this person. But I don't comprehend what to do. (Sorry!) For example, how do I make a dataset? Then I need to train it on that dataset, right? Then how do I get it to write out the text I want it to write in that handwriting style?

Also, if the original writing is in Latin, do I need to add a Latin word dataset (text file)?

Thanks so much!

This is a great job
When I first started encoder training. In line 279 of "base_trainer. py", there is a problem with "val_ result=self. _validd_Epoch". This function cannot be found. My current solution is to annotate valid modules. Is there a better way to solve this problem?
Thank you.

Hey I'm trying to get sort of a minimum working example of your project running so I can better understand what's going on by running generate.py on the pre-trained snapshots which you provided.

python generate.py -c ~/Downloads/IAM_weights/IAMslant_noMask_charSpecSingleAppend_GANMedMT_autoAEMoPrcp2tightNewCTCUseGen_balB_hCF0.75_sMG/checkpoint-iteration175000.pth -d ~/Downloads -s ~/Downloads/IAM_weights/IAMslant_noMask_charSpecSingleAppend_GANMedMT_autoAEMoPrcp2tightNewCTCUseGen_balB_hCF0.75_sMG/test_styles_175000.pkl

The script errors out as follows:

File "/home/redpanda/Downloads/handwriting_line_generation-w1.0/generate.py", line 10, in <module> from trainer import * File "/home/redpanda/Downloads/handwriting_line_generation-w1.0/trainer/__init__.py", line 10, in <module> from .hw_with_style_trainer import HWWithStyleTrainer File "/home/redpanda/Downloads/handwriting_line_generation-w1.0/trainer/hw_with_style_trainer.py", line 10, in <module> from data_loader import getDataLoader File "/home/redpanda/Downloads/handwriting_line_generation-w1.0/data_loader/__init__.py", line 1, in <module> from .data_loaders import * File "/home/redpanda/Downloads/handwriting_line_generation-w1.0/data_loader/data_loaders.py", line 21, in <module> from datasets import mmd_hw_dataset ImportError: cannot import name 'mmd_hw_dataset' from 'datasets' (unknown location)
Is mmd_hw_dataset missing, or has the name of the file to import changed? I'm a bit stumped as to what's going on here. Thanks!

Good morning sir . I want to make an handwritten line recognition with the RIMES datasets. It is take too long to get the dataset from http://www.a2ialab.com/doku.php?id=rimes_database:start. Do you have an idea or an alternative please or can you provide me this dataset( I see you use it for your GAN )

Hi,
Thanks for creating a clean branch of the code, it has been a great help in understanding the paper. I was wondering if it is possible to train it with multiple GPUS in Pytorch. I tried to do it with, DDP and the code is running but I think the gradients in both GPUs are not in synch (GPU-0 is on iteration-3990 while GPU-1 is on iteration-3975). On further exploration, I found that DDP with retain_graph=True doesn't work in Pytorch. I was wondering if we can recreate the curriculum logic suitable with Pytorch DDP.

While training the GAN it is asking for english_text.txt file. Please provide some information on how to download that file and if we need to create that one than how to create?

It is a great paper, thank you for sharing the code.

I am curious, will the model work on word-level data sets? Or will the model work in a language where the strokes are connected together?

gen_text_ file has text as well as two numerical values as their output. What do these values indicate?
Are they mean and standard deviation of something?

it is cool for your job for hw generation,
But,your code is full with if .....else.....

It seems do a lots of job for experiments
but...i feel difficult to understand the code .
can you publish a clean code for useful experimental settings
( feel sad for seeing code with if ........else ......)
thanks

What determines the size of the generated output when generating images using generate.py?

Is all output the size to which all input images are resized (ie, height of 64)? Or is there a way to have the output generated have the same size/resolution as the input images?

Hi!
This source code is so amazing.
I tested it and want to apply it to another language like Vietnamese. However, the problem I encounter here is recogloss = 0. Is there a problem if the char to index is too much? I use it for Vietnamese and the number of images is almost the same as for IAM. My length of char_to_index is 229 and when I replace the IAM char_to_index encoder the code runs normally

Great paper! I'm trying to run 'generate.py'. I am using your pretrained snapshots (IAM_hwr_cnnonly...), and setting the two image paths using the f option, along with the text.

I've checked, and the images and text seem to be loaded correctly. But I'm getting this error with the style extractor:

handwriting_line_generation\model\hw_with_style.py", line 782, in extract_style
style = self.style_extractor(collapsed_image, collapsed_label)
TypeError: 'NoneType' object is not callable

Thanks for the amazing work.

Thanks for the awesome work!

Just curious to know if trained model published here is able to generate alpha numerical or just numerical images as these are trained on IAM and RIMEs which does not have much digits or number.

This looks like really cool work! My issue isn't with the code/functionality as it is about get a hold of one of the datasets, but I was hoping you could help. I assume I'd need the IAM or DIMES dataset to replicate the results in the paper, but I think the source of the IAM dataset is no longer active. I signed up but didn't get the promised activation link, and the website homepage says "The activities of the Research Group on Computer Vision and Artificial Intelligence terminated with the retirement of its head, Prof. Horst Bunke, by July 31, 2011."

This is curious to me since a lot of (recent) work in handwriting synthesis uses the IAM dataset and a cursory search doesn't reveal any other way to download the dataset. Do you have any insights?

Thanks!

Was re-directed to this repo from both Paperswithcode and arxiv.org. Can't find the code anywhere.

Hi. I'm trying to use the "from-to" mode described in the README.md's "Generating images using generate.py." I figured the argument would be -f path/to/image1 path/to/image2 path/to/text.

The output is

usage: generate.py
      [-h]
      [-c CHECKPOINT]
      [-d SAVEDIR]
      [-g GPU]
      [-T]
      [-f CONFIG]
      [-a ADDTOCONFIG]
      [-r RUN]
      [-s STYLE_LOC]
generate.py: error: unrecognized arguments: path/to/image2 path/to/text

The list of potential arguments (above) seemed odd to me, since there is no "from-to" or "random" mentioned. So then I thought maybe I get prompted later in some way for these modes. So I ran it without the -f.

This returned

loaded iteration 60000
Traceback (most recent call last):
  File "generate.py", line 963, in <module>
    main(args.checkpoint, args.savedir, gpu=args.gpu,  config=args.config, addToConfig=addtoconfig, test =args.test,arguments=arguments, style_loc=args.style_loc)
  File "generate.py", line 252, in main
    num_char = config['model']['num_class']
KeyError: 'num_class'

I poked around in the code, especially looking for instances of num_char and num_class, but couldn't come up with anything enlightening. Any help would be much appreciated.

Dear Concern,
It's great project. But It is hard for me for exploring the exact format of your dataset. Please give a brief review on format of Dataset and need ways/utilities for other language's offline handwritten line generation.
Thanks.

Hello,
Your work is fantastic. May I ask some questions?

About training:

Excuse me, did you use Lines-Text-Level data or Word-Level data during training with the IAM dataset?

About FID and GS assessment:

Excuse me, can the code for calculating FID and GS be shared?

Good day!
Thank you for your dedicated work.
I want to ask about the image recognition module. How many iterations did you train on the branch before moving on to train the other?

Hi,
Thank you for sharing us the great work! I really appreciate it.

I have a question when it comes to training the encoder.
I got 'nan' for the losses while training the encoder with a custom dataset. Training went well when I used IAM dataset though.
Do you have any idea why I got 'nan' for losses when I used a custom dataset?
One thing I can think of is that since every image from IAM dataset doesn't include white space margins in front and rear of a sentence, the training went well, but the dataset I used includes some white space margins, so I got 'nan' for the losses.
Do you think that's the reason why I got 'nan' for the losses? Or do you have any other ideas?

I will be grateful for any help you can provide.

Hi sir, this is an excellent contribution for me. I am trying to train it in my dataset (it is for Vietnamese). First, I am finetuning on HWR model. I think I am having some problems with CTC loss, cause it is nan for some epoch. I am trying to fix it, but I can not find where you create the saved folder as well as where you save the model.
Thanks for your effort.

hi great work there .I try to train handwriting recognition model with used command
HWR: python train.py -c configs/cf_IAM_hwr_cnnOnly_batchnorm_aug.json
but i got this error

do I miss something?

Really appreciate the great work you have done.

I'm trying to train the generator with IAM dataset and I'm getting the following error.

Traceback (most recent call last):
  File "train.py", line 134, in <module>
    main(config, args.resume)
  File "train.py", line 63, in main
    trainer = trainerClass(model, loss, metrics,
  File "handwriting_line_generation/trainer/hw_with_style_trainer.py", line 115, in __init__
    self.text_data = TextData(config['trainer']['text_data'],config['data_loader']['char_file'],text_data_batch_size,max_len=text_data_max_len,words=text_words,characterBalance=characterBalance) if 'text_data' in config['trainer'] else None
  File "handwriting_line_generation/datasets/text_data.py", line 15, in __init__
    with open(textfile) as f:
FileNotFoundError: [Errno 2] No such file or directory: '../data/english_text.txt'

Thank you for your help.

Hi, thanks for the excellent work! I have generate few handwriting images by using generate.py in from-to module, which is great! However, when I try to use random module, I get an error, which is NameError: name 'getDataLoader' is not defined. So I try to delete # before line 13: from data_loader import getDataLoader, and get a new error FileNotFoundError: [Errno 2] No such file or directory: '../data/IAM/xmls\m03-110.xml'. But I do not find any xml files in repo.

Hello again,

While training the generator with IAM daaset, I am getting the following error.

Begin training
Traceback (most recent call last):
  File "train.py", line 134, in <module>
    main(config, args.resume)
  File "train.py", line 79, in main
    trainer.train()
  File "/handwriting_line_generation/base/base_trainer.py", line 219, in train
    result = self._train_iteration(self.iteration)
  File "/handwriting_line_generation/trainer/hw_with_style_trainer.py", line 367, in _train_iteration
    losses = self.run_gen(instance,lesson)
  File "/handwriting_line_generation/trainer/hw_with_style_trainer.py", line 1069, in run_gen
    recon,style = self.model.autoencode(image,label,mask,a_batch_size,center_line=center_line,stop_grad_extractor=autoSGStyle)
  File "/handwriting_line_generation/model/hw_with_style.py", line 729, in autoencode
    style = self.extract_style(image,label,a_batch_size)
  File "/handwriting_line_generation/model/hw_with_style.py", line 781, in extract_style
    style = self.style_extractor(collapsed_image, collapsed_label)
  File "/anaconda3/envs/DocumentAnalysis/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/handwriting_line_generation/model/char_style.py", line 122, in forward
    x = self.down(x)
  File "/anaconda3/envs/DocumentAnalysis/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/anaconda3/envs/DocumentAnalysis/lib/python3.8/site-packages/torch/nn/modules/container.py", line 139, in forward
    input = module(input)
  File "/anaconda3/envs/DocumentAnalysis/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/handwriting_line_generation/model/MUNIT_networks.py", line 1124, in forward
    x = self.conv(self.pad(x))
  File "/anaconda3/envs/DocumentAnalysis/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/anaconda3/envs/DocumentAnalysis/lib/python3.8/site-packages/torch/nn/modules/padding.py", line 275, in forward
    return F.pad(input, self.padding, 'replicate')
  File "/anaconda3/envs/DocumentAnalysis/lib/python3.8/site-packages/torch/nn/functional.py", line 4172, in _pad
    return torch._C._nn.replication_pad2d(input, pad)
RuntimeError: "replication_pad2d_cuda" not implemented for 'Long'

Before this error I got the first error in #11 , which I solved as per the suggestion in the post.

Kindly help me with this error.
Thank you for your help.

Thank you for your great work.

1. I am trying to train HWR on RIMES dataset by this command python train.py -c configs/cf_RIMESLines_hwr_cnnOnly_batchnorm_aug.json

But the error happens.

Begin training
Traceback (most recent call last):
  File "train.py", line 132, in <module>
    main(config, args.resume)
  File "train.py", line 78, in main
    trainer.train()
  File "handwriting_line_generation/base/base_trainer.py", line 219, in train
    result = self._train_iteration(self.iteration)
  File "handwriting_line_generation/trainer/hw_with_style_trainer.py", line 378, in _train_iteration
    pred, recon, losses = self.run(instance)
  File "handwriting_line_generation/trainer/hw_with_style_trainer.py", line 736, in run
    pred = self.model.hwr(image, style)
  File "lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "handwriting_line_generation/model/cnn_only_hwr.py", line 132, in forward
    conv = self.cnn(input)
  File "lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "lib/python3.8/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "lib/python3.8/site-packages/torch/nn/modules/conv.py", line 423, in forward
    return self._conv_forward(input, self.weight)
  File "lib/python3.8/site-packages/torch/nn/modules/conv.py", line 419, in _conv_forward
    return F.conv2d(input, weight, self.bias, self.stride,
RuntimeError: Input type (torch.cuda.LongTensor) and weight type (torch.cuda.FloatTensor) should be the same

Then I add input = input.type(torch.cuda.FloatTensor) to this function in cnn_only_hwr.py

def forward(self, input, style=None):
        if self.pad is not None:
            input=self.pad(input)
        # conv features
        input = input.type(torch.cuda.FloatTensor)
        conv = self.cnn(input)
        b, c, h, w = conv.size()
        # assert h == 1, "the height of conv must be 1"
        conv = conv.view(b, -1, w)
        conv = self.cnn1d(conv)
        output = conv.permute(2, 0, 1)  # [w, b, c]

        return output

Then this error happens:

Begin training
{'recogLoss': nan}
Traceback (most recent call last):
  File "train.py", line 132, in <module>
    main(config, args.resume)
  File "train.py", line 78, in main
    trainer.train()
  File "/home/ocr/truchlp/handwriting_line_generation/base/base_trainer.py", line 219, in train
    result = self._train_iteration(self.iteration)
  File "/home/ocr/truchlp/handwriting_line_generation/trainer/hw_with_style_trainer.py", line 399, in _train_iteration
    assert(loss==0 or (not torch.isnan(loss) and not torch.isinf(loss)))
AssertionError

2. I am also trying to train GAN on RIMES dataset by this command python train.py -c configs/cf_RIMESLinesslant_noMask_charSpecSingleAppend_GANMedMT_autoAEMoPrcp2tightNewCTCUseGen_balB_hCF0.75_sMG.json
   But it says:

Set valid_data_loader to None
Traceback (most recent call last):
  File "train.py", line 132, in <module>
    main(config, args.resume)
  File "train.py", line 62, in main
    trainer = trainerClass(model, loss, metrics,
  File "handwriting_line_generation/trainer/hw_with_style_trainer.py", line 115, in __init__
    self.text_data = TextData(config['trainer']['text_data'],config['data_loader']['char_file'],text_data_batch_size,max_len=text_data_max_len,words=text_words,characterBalance=characterBalance) if 'text_data' in config['trainer'] else None
  File "handwriting_line_generation/datasets/text_data.py", line 15, in __init__
    with open(textfile) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'data/french_news.txt'

Thank you for helping me.

Hey There!

Thank you for your great work, it is incredible.

I was trying to create my own handwriting style vectors by the gen_style.py file, but the results were not as good as the test_styles you have provided in the releases. Actually, it generated handwriting, but the style didn't make sense (it wasn't similar to my handwriting style).

To create the custom style vectors, first I used my own OCR tool to crop the lines from the image I input, then I had to intervene and to change the author_hw_dataset.py a bit, in order to get my cropped lines and manipulate them.

For the generation process, I chose the RANDOM action, but the random sampling was made from my single handwriting style (represented as several styles, each compatible with each line cropped from my own image example).

The commands I used:

1. python3 get_styles.py -c "$checkpoint_path" -d "/content/new" -g 0 -T -f "$config_path"
2. !python3 generate.py -c "$checkpoint_path" -d "$test_results_path" -g 0 -T -f "$config_path" -s "/content/new/test_styles_175000.pkl"

"/content/new/test_styles_175000.pkl" is the path of my own line-style vectors, created by command number 1.

Is there anything that could help me to generate handwriting that is more similar to mine?

Again, thank you very much, it is awesome work!

indexes/random interp/vae random/strech/author display/math/turk gen/from-to/umap-images/Random styles/help/quit? True
author? 299
batch? 1
text? 299
author: 299
Traceback (most recent call last):
File "generate.py", line 904, in
main(args.checkpoint, args.savedir, gpu=args.gpu, config=args.config, addToConfig=addtoconfig, test =args.test,arguments=arguments, style_loc=args.style_loc)
File "generate.py", line 706, in main
image1 = instance1['image'].to(gpu)
UnboundLocalError: local variable 'instance1' referenced before assignment

Can you please help me out with this ? What would be the user input in
indexes/random interp/vae random/strech/author display/math/turk gen/from-to/umap-images/Random styles/help/quit?
author?
batch?
text?

I ran the pretrain model on RIMES dataset by this command.

python generate.py -c ../RIME_weights/RIMESLinesslant_noMask_charSpecSingleAppend_GANMedMT_autoAEMoPrcp2tightNewCTCUseGen_balB_hCF0.75_sMG/checkpoint-iteration175000.pth -d output -f configs/cf_RIMESLinesslant_noMask_charSpecSingleAppend_GANMedMT_autoAEMoPrcp2tightNewCTCUseGen_balB_hCF0.75_sMG.json

Next, I tried all the generate options. Sometimes I got this error:

File "/media/huynhtruc0309/DATA/vtcc/generate_vietnamese_htr/handwriting_line_generation/model/hw_with_style.py", line 845, in insert_spaces
    spaced[i,b,cls]=1
IndexError: index 80 is out of bounds for dimension 2 with size 78

Sometimes I got the generated handwriting but they look very weird, noisy and not text.

I don't know where I got wrong :(

Hello again,

I have completed generating the styles.pkl using the generate.py.

Save styles using get_styles.py Then umap_styles.py styles.pkl [image dir]

Could you please tell the format and type of the files present in the image_dir and from where I can get these files. An ordered.txt is required.
Thank you for your help.

PS. import os is missing in umap_styles.py