Hi guys: Everything before training goes well. However, when i got in epoches, pro

Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

pickle problem in train about sqlova HOT 2 CLOSED

naver commented on July 25, 2024

pickle problem in train

from sqlova.

Comments (2)

whwang299 commented on July 25, 2024

Hi @farsmile

The problem seems to be caused by lambda function in torch.utils.data.DataLoader

sqlova/sqlova/utils/utils_wikisql.py

Lines 91 to 98 in b7ce9ad

 def get_loader_wikisql(data_train, data_dev, bS, shuffle_train=True, shuffle_dev=False): 

 train_loader = torch.utils.data.DataLoader( 

 batch_size=bS, 

 dataset=data_train, 

 shuffle=shuffle_train, 

 num_workers=4, 

 collate_fn=lambda x: x # now dictionary values are not merged! 

 )

which is, according to this link, the problem between pytorch and Windows.
(Sorry I couldn't test this by myself as I don't have Window machine with GPU).

Using a custom data_loader function may solve this problem.

Thanks.

Wonseok

from sqlova.

farsmile commented on July 25, 2024

Hi Wonseok:
it works now and your link is helpful!

Thanks.

from sqlova.

Related Issues (20)

For real industrial application, what strategy to locate the exact table? HOT 1
How to fine tune sequence-to-SQL？
How do I use predict.py for custom data? HOT 2
weird error occurs when run predict.py
Testing models with predict.py does not give me any results file HOT 1
Need of results_train.jsonl
Standford coreNLP
Training on custom data HOT 4
Replicating dev results on BERT base
Training doesn't run + how to predict with custom query? HOT 1
Keras implementation of Column Attention
Error with annotate_ws.py HOT 7
Isuue in utlis.wikisql.py
TypeError: 'Document' object is not iterable
There is insufficient memory during training, and the get_wemb_bert function call in the train function applies for a lot of memory
dbengine HOT 5
How to use RoBERTa instead of BERT ?
How can I evaluate the pre-trained model on WikiSQL? HOT 1
Question about the human eval result, thank you. HOT 2
No module named 'matplotlib' after I download matplotlib

pickle problem in train about sqlova HOT 2 CLOSED

Comments (2)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	def get_loader_wikisql(data_train, data_dev, bS, shuffle_train=True, shuffle_dev=False):
	train_loader = torch.utils.data.DataLoader(
	batch_size=bS,
	dataset=data_train,
	shuffle=shuffle_train,
	num_workers=4,
	collate_fn=lambda x: x # now dictionary values are not merged!
	)