Giter VIP home page Giter VIP logo

Comments (5)

shengsli avatar shengsli commented on May 25, 2024

Hi @ClementPinard

  • I think I maybe find what the problem is. Since I am not familiar with the code, I assumed the dataset_dir is one of the dataset subfolders like 2011_09_26_drive_0001_sync. I change the dataset_dir to ..\SfmLearner-Pytorch\raw_data_downloader. And now it starts retrieving frames.

  • But, I have a new issue now that NameError: name 'data_loader' is not defined. I include part of the error message below:

C:\TensorFlow\SfmLearner-Pytorch>python data/prepare_train_data.py C:\TensorFlow\SfmLearner-Pytorch\raw_data_downloader --dataset-format kitti --dump-root C:\TensorFlow\SfmLearner-Pytorch\formatted_data --width 416 --height 128 --num-threads 4
Retrieving frames
  0%|                                                                     | 0/24 [00:00<?, ?it/s]multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "C:\Users\sheng\AppData\Local\Programs\Python\Python36\lib\site-packages\joblib\_parallel_backends.py", line 350, in __call__
    return self.func(*args, **kwargs)
  File "C:\Users\sheng\AppData\Local\Programs\Python\Python36\lib\site-packages\joblib\parallel.py", line 131, in __call__
    return [func(*args, **kwargs) for func, args, kwargs in self.items]
  File "C:\Users\sheng\AppData\Local\Programs\Python\Python36\lib\site-packages\joblib\parallel.py", line 131, in <listcomp>
    return [func(*args, **kwargs) for func, args, kwargs in self.items]
  File "C:\TensorFlow\SfmLearner-Pytorch\data\prepare_train_data.py", line 27, in dump_example
    scene_list = data_loader.collect_scenes(scene)
NameError: name 'data_loader' is not defined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\sheng\AppData\Local\Programs\Python\Python36\lib\multiprocessing\pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "C:\Users\sheng\AppData\Local\Programs\Python\Python36\lib\site-packages\joblib\_parallel_backends.py", line 359, in __call__
    raise TransportableException(text, e_type)
joblib.my_exceptions.TransportableException: TransportableException
___________________________________________________________________________
NameError                                          Sun Mar  4 19:22:09 2018
PID: 2692Python 3.6.2: C:\Users\sheng\AppData\Local\Programs\Python\Python36\python.exe
...........................................................................
C:\Users\sheng\AppData\Local\Programs\Python\Python36\lib\site-packages\joblib\parallel.py in __call__(self=<joblib.parallel.BatchedCalls object>)
    126     def __init__(self, iterator_slice):
    127         self.items = list(iterator_slice)
    128         self._size = len(self.items)
    129
    130     def __call__(self):
--> 131         return [func(*args, **kwargs) for func, args, kwargs in self.items]
        self.items = [(<function dump_example>, (Path('C:\\TensorFlow\\SfmLearner-Pytorch\\raw_da...nloader\\2011_09_26\\2011_09_26_drive_0001_sync'),), {})]
    132
    133     def __len__(self):
    134         return self._size
    135

...........................................................................
C:\Users\sheng\AppData\Local\Programs\Python\Python36\lib\site-packages\joblib\parallel.py in <listcomp>(.0=<list_iterator object>)
    126     def __init__(self, iterator_slice):
    127         self.items = list(iterator_slice)
    128         self._size = len(self.items)
    129
    130     def __call__(self):
--> 131         return [func(*args, **kwargs) for func, args, kwargs in self.items]
        func = <function dump_example>
        args = (Path('C:\\TensorFlow\\SfmLearner-Pytorch\\raw_da...nloader\\2011_09_26\\2011_09_26_drive_0001_sync'),)
        kwargs = {}
    132
    133     def __len__(self):
    134         return self._size
    135

...........................................................................
C:\TensorFlow\SfmLearner-Pytorch\data\prepare_train_data.py in dump_example(scene=Path('C:\\TensorFlow\\SfmLearner-Pytorch\\raw_da...nloader\\2011_09_26\\2011_09_26_drive_0001_sync'))
     22
     23 args = parser.parse_args()
     24
     25
     26 def dump_example(scene):
---> 27     scene_list = data_loader.collect_scenes(scene)
        scene_list = undefined
        scene = Path('C:\\TensorFlow\\SfmLearner-Pytorch\\raw_da...nloader\\2011_09_26\\2011_09_26_drive_0001_sync')
     28     for scene_data in scene_list:
     29         dump_dir = args.dump_root/scene_data['rel_path']
     30         dump_dir.makedirs_p()
     31         intrinsics = scene_data['intrinsics']

NameError: name 'data_loader' is not defined
___________________________________________________________________________

Thanks for taking your time.

from sfmlearner-pytorch.

ClementPinard avatar ClementPinard commented on May 25, 2024

Hello, thanks for trying this code for Windows !

I did some research on your problem (which does not appear on Linux) here is some doc about it : http://rhodesmill.org/brandon/2010/python-multiprocessing-linux-windows/

Basically global variables are not shared with multi threaded function Windows while they are in Linux.

I'll get a fix tomorrow. In the mean time you can add the data loader as an argument to the function dump_example or replace the multi thread command with a simple for loop.

Cheers,

Clément

from sfmlearner-pytorch.

ClementPinard avatar ClementPinard commented on May 25, 2024

before I push the changes, as I am not on windows, I need you to test it :
can you change this line to def dump_example(scene, data_loader):

and this line to Parallel(n_jobs=args.num_threads)(delayed(dump_example)(scene, data_loader) for scene in tqdm(data_loader.scenes)) ?

It should pass the data_loader to the function, and let it access it on windows (not needed on linux)

thanks

from sfmlearner-pytorch.

shengsli avatar shengsli commented on May 25, 2024

Hi @ClementPinard

Thank you for your prompt reply.

  • The command python3 data/prepare_train_data.py /path/to/raw/kitti/dataset/ --dataset-format 'kitti' --dump-root /path/to/resulting/formatted/data/ --width 416 --height 128 --num-threads 4 [--static-frames /path/to/static_frames.txt] [--with-gt] runs well now. It format the data with no problem.

  • Now I get stuck at PyTorch. It seems that PyTorch has no support for windows currently. I will keep searching for it. If there is really no PyTorch for windows, I will have a go with the TensorFlow version of this project.

  • The link about python's multiprocessing really helps me understand what is going on. Thank you very much.

from sfmlearner-pytorch.

ClementPinard avatar ClementPinard commented on May 25, 2024

Unfortunately no official support for windows pytorch yet, but I heard about unofficial builds, although it's still experimental
you might want to check this out for example :https://github.com/peterjc123/pytorch-scripts

Thanks anyway for your input ! Glad to know how to make my code windows ready ;)

from sfmlearner-pytorch.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.