Giter VIP home page Giter VIP logo

Comments (14)

xxr3376 avatar xxr3376 commented on May 18, 2024

您可以看到最开始的报错：devkit directory /mnt/CEPH_GALACTICA/dataset/imagenet_new/ILSVRC2012_devkit_t12 does not exists

说明 data loader 并未找到硬盘上的 imagenet 数据集。

from megengine.

windreamer avatar windreamer commented on May 18, 2024

Hi @YuejiYang

您的遇到的错误我们分析了一下，也确实一时难以定位和确认，不知道您是否方便尝试下列方法来帮我们更好的了解问题

  1. 您是否能尝试更简单的例子，例如 https://megengine.org.cn/doc/latest/basic/network_build.html#network-build 给出的简单网络，来确认您的 docker 环境已经可以正确运行 MegEngine

  2. 您是否可以检查一下您已经正确下载并放置了 ImageNet 数据集供 MegEngine 的DataLoader 使用？

希望能得到您更多的反馈

from megengine.

YuejiYang avatar YuejiYang commented on May 18, 2024

您可以看到最开始的报错：devkit directory /mnt/CEPH_GALACTICA/dataset/imagenet_new/ILSVRC2012_devkit_t12 does not exists

说明 data loader 并未找到硬盘上的 imagenet 数据集。

有train的数据和validation的数据

from megengine.

YuejiYang avatar YuejiYang commented on May 18, 2024

Hi @YuejiYang

您的遇到的错误我们分析了一下，也确实一时难以定位和确认，不知道您是否方便尝试下列方法来帮我们更好的了解问题

  1. 您是否能尝试更简单的例子，例如 https://megengine.org.cn/doc/latest/basic/network_build.html#network-build 给出的简单网络，来确认您的 docker 环境已经可以正确运行 MegEngine
  2. 您是否可以检查一下您已经正确下载并放置了 ImageNet 数据集供 MegEngine 的DataLoader 使用？

希望能得到您更多的反馈

1 可以跑简单的例子
2. 数据集是对的，因为数据集不对的话，应该不是这种错误。 我只用train 和validation的数据。 我不用docker 直接安装是可以跑的。 但是我希望在docker里build from source。 结果发现docker里pip install的也无法使用。 build from souce 可以成功，我自己build的版本也是同样的问题。

from megengine.

xxr3376 avatar xxr3376 commented on May 18, 2024

@zhouyizhuang-megvii 可以看看是不是 imagenet dataset 的问题，看看有没有简单一点的验证方法。

from megengine.

zhouyizhuang-megvii avatar zhouyizhuang-megvii commented on May 18, 2024

@zhouyizhuang-megvii 可以看看是不是 imagenet dataset 的问题，看看有没有简单一点的验证方法。

看上去不像是ImageNet的问题，devkit缺失应该没影响。如果pytorch用起来没问题那么megengine也没问题。看上去更像是pyarrow的问题，FYI @wjfwzzc，看看眼熟不？

from megengine.

zky001 avatar zky001 commented on May 18, 2024

像是pyarrow的问题

from megengine.

wjfwzzc avatar wjfwzzc commented on May 18, 2024

@zhouyizhuang-megvii 可以看看是不是 imagenet dataset 的问题，看看有没有简单一点的验证方法。

看上去不像是ImageNet的问题，devkit缺失应该没影响。如果pytorch用起来没问题那么megengine也没问题。看上去更像是pyarrow的问题，FYI @wjfwzzc，看看眼熟不？

没见过类似报错。
确实没有测试过pyarrow在docker上的行为。先num_workers=0绕过pyarrow看一下能不能跑？

from megengine.

xxr3376 avatar xxr3376 commented on May 18, 2024

突然意识到，会不是 docker shm-size 设置过小，导致 pyarrow 出错？ @wjfwzzc

@YuejiYang 你可以尝试在 docker run 的时候将 shm-size 设置的大一些（比如 5GB），来尝试规避这个问题。

from megengine.

xxr3376 avatar xxr3376 commented on May 18, 2024

closed due to inactive.

from megengine.

lijiansong avatar lijiansong commented on May 18, 2024

pip3 install megengine==1.4.0rc1 -f https://megengine.org.cn/whl/mge.html -i https://pypi.douban.com/simple/
也是报错的！

Looking in indexes: https://pypi.douban.com/simple/
Looking in links: https://megengine.org.cn/whl/mge.html
Collecting megengine==1.4.0rc1
  Using cached https://data.megengine.org.cn/pypi/MegEngine-1.4.0rc1-cp36-cp36m-manylinux2014_x86_64.whl (873.1 MB)
Requirement already satisfied: tqdm in ./anaconda2/envs/mega/lib/python3.6/site-packages (from megengine==1.4.0rc1) (4.60.0)
Requirement already satisfied: redispy in ./anaconda2/envs/mega/lib/python3.6/site-packages (from megengine==1.4.0rc1) (3.0.0)
Requirement already satisfied: deprecated in ./anaconda2/envs/mega/lib/python3.6/site-packages (from megengine==1.4.0rc1) (1.2.12)
Requirement already satisfied: numpy>=1.18 in ./anaconda2/envs/mega/lib/python3.6/site-packages (from megengine==1.4.0rc1) (1.19.5)
Requirement already satisfied: requests in ./anaconda2/envs/mega/lib/python3.6/site-packages (from megengine==1.4.0rc1) (2.25.1)
Requirement already satisfied: pyarrow in ./anaconda2/envs/mega/lib/python3.6/site-packages (from megengine==1.4.0rc1) (3.0.0)
Requirement already satisfied: opencv-python in ./anaconda2/envs/mega/lib/python3.6/site-packages (from megengine==1.4.0rc1) (4.5.1.48)
Requirement already satisfied: tabulate in ./anaconda2/envs/mega/lib/python3.6/site-packages (from megengine==1.4.0rc1) (0.8.9)
Requirement already satisfied: wrapt<2,>=1.10 in ./anaconda2/envs/mega/lib/python3.6/site-packages (from deprecated->megengine==1.4.0rc1) (1.12.1)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./anaconda2/envs/mega/lib/python3.6/site-packages (from requests->megengine==1.4.0rc1) (1.26.4)
Requirement already satisfied: chardet<5,>=3.0.2 in ./anaconda2/envs/mega/lib/python3.6/site-packages (from requests->megengine==1.4.0rc1) (4.0.0)
Requirement already satisfied: idna<3,>=2.5 in ./anaconda2/envs/mega/lib/python3.6/site-packages (from requests->megengine==1.4.0rc1) (2.10)
Requirement already satisfied: certifi>=2017.4.17 in ./anaconda2/envs/mega/lib/python3.6/site-packages (from requests->megengine==1.4.0rc1) (2020.12.5)
Installing collected packages: megengine
ERROR: Exception:
Traceback (most recent call last):
  File "/home/foo/work-space/anaconda2/envs/mega/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 189, in _main
    status = self.run(options, args)
  File "/home/foo/work-space/anaconda2/envs/mega/lib/python3.6/site-packages/pip/_internal/cli/req_command.py", line 178, in wrapper
    return func(self, options, args)
  File "/home/foo/work-space/anaconda2/envs/mega/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 400, in run
    pycompile=options.compile,
  File "/home/foo/work-space/anaconda2/envs/mega/lib/python3.6/site-packages/pip/_internal/req/__init__.py", line 88, in install_given_reqs
    pycompile=pycompile,
  File "/home/foo/work-space/anaconda2/envs/mega/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 796, in install
    requested=self.user_supplied,
  File "/home/foo/work-space/anaconda2/envs/mega/lib/python3.6/site-packages/pip/_internal/operations/install/wheel.py", line 827, in install_wheel
    requested=requested,
  File "/home/foo/work-space/anaconda2/envs/mega/lib/python3.6/site-packages/pip/_internal/operations/install/wheel.py", line 662, in _install_wheel
    file.save()
  File "/home/foo/work-space/anaconda2/envs/mega/lib/python3.6/site-packages/pip/_internal/operations/install/wheel.py", line 429, in save
    shutil.copyfileobj(f, dest)
  File "/home/foo/work-space/anaconda2/envs/mega/lib/python3.6/shutil.py", line 79, in copyfileobj
    buf = fsrc.read(length)
  File "/home/foo/work-space/anaconda2/envs/mega/lib/python3.6/zipfile.py", line 872, in read
    data = self._read1(n)
  File "/home/foo/work-space/anaconda2/envs/mega/lib/python3.6/zipfile.py", line 948, in _read1
    data = self._decompressor.decompress(data, n)
zlib.error: Error -3 while decompressing data: invalid literal/length code

Python 3.6.13
pip 21.0.1 from /home/foo/work-space/anaconda2/envs/mega/lib/python3.6/site-packages/pip (python 3.6)

from megengine.

xxr3376 avatar xxr3376 commented on May 18, 2024

@lijiansong 你的问题我已经复现，但是通过命令 pip3 install megengine==1.4.0rc1 -f https://megengine.org.cn/whl/mge.html 是正常的。

请你先用我们的 CDN 命令下载吧，我们正在调查是 PyPI 上的源坏了还是豆瓣上同步出错了

from megengine.

xxr3376 avatar xxr3376 commented on May 18, 2024

@lijiansong 类似的，pip3 install megengine==1.4.0rc1 也是正常的，我认为是豆瓣的 pypi 源在同步过程中出现了问题，可能跟这个包是刚刚上传的，豆瓣还没完全 rsync 完有关，建议先用其他源安装。

from megengine.

lijiansong avatar lijiansong commented on May 18, 2024

@xxr3376 感谢回复!
pip3 install megengine==1.4.0rc1 --no-cache-dir可以work!

from megengine.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.