Giter VIP home page Giter VIP logo

Comments (4)

DongHwanJang avatar DongHwanJang commented on July 20, 2024 2

확실히

if not self.opt.no_vgg_loss:
            #G_losses['VGG'] = self.criterionVGG(fake_RGB, target_RGB) * self.opt.lambda_vgg
            pass

로 comment out 하니까 nan이 안나옴. VGG loss가 확실한 원인인데 이유를 모르겠음

짐작하기를 LAB2RGB 연산에서 뭔가 미분하면서 터지는 부분이 있는게 아닌가 싶음

from spade_colorization.

DongHwanJang avatar DongHwanJang commented on July 20, 2024 1

https://discuss.pytorch.org/t/function-powbackward0-returned-nan-values-in-its-0th-output/60882
이거를 참고하여 해결

from spade_colorization.

ThisIsIsaac avatar ThisIsIsaac commented on July 20, 2024

아 이거 known error 구나, loss 들도 전부 0 아니면 Nan 으로 뜸:

(epoch: 1, iters: 8, time: 1.210) GAN: -0.005 GAN_Feat: 3.050 VGG: 0.476 reconstruction: 0.005 contextual: 4.767 D_Fake: 0.000 D_real: 0.995 
(epoch: 1, iters: 16, time: 5.390) GAN: nan GAN_Feat: nan VGG: nan contextual: nan D_Fake: 0.000 D_real: 0.000 
/home/minds/.virtualenvs/spade/lib/python3.6/site-packages/skimage/color/colorconv.py:979: RuntimeWarning: invalid value encountered in less
  if np.any(z < 0):
/home/minds/.virtualenvs/spade/lib/python3.6/site-packages/skimage/color/colorconv.py:987: RuntimeWarning: invalid value encountered in greater
  mask = out > 0.2068966
/home/minds/.virtualenvs/spade/lib/python3.6/site-packages/skimage/color/colorconv.py:630: RuntimeWarning: invalid value encountered in greater
  mask = arr > 0.0031308
(epoch: 1, iters: 24, time: 5.014) GAN: nan GAN_Feat: nan VGG: nan reconstruction: nan contextual: nan D_Fake: 0.000 D_real: 0.000 
(epoch: 1, iters: 32, time: 4.808) GAN: nan GAN_Feat: nan VGG: nan contextual: nan D_Fake: 0.000 D_real: 0.000 
(epoch: 1, iters: 40, time: 5.101) GAN: nan GAN_Feat: nan VGG: nan reconstruction: nan contextual: nan D_Fake: 0.000 D_real: 0.000 
(epoch: 1, iters: 48, time: 5.055) GAN: nan GAN_Feat: nan VGG: nan contextual: nan D_Fake: 0.000 D_real: 0.000
  • is_reconstructing 키든 안키든 터짐.
  • batchSize 상관 없이 2번째 Iteration 부터 터짐

from spade_colorization.

DongHwanJang avatar DongHwanJang commented on July 20, 2024 
        with autograd.detect_anomaly():
            g_loss.backward()

이렇게 옵션을 지정해주면 backward시 gradient가 nan으로 터지는 부분을 잡아줄 수 있다. (검색해도 바로 안나와서 좀 놀랐다... 몇번 타고 들어가야 나옴)

  File "/home/minds/.virtualenvs/spade/lib/python3.6/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Function 'PowBackward0' returned nan values in its 0th output.

다음과 같이 터지는 걸로 보아 power 쪽에 문제가 있고 이에 따라 power 연산 부분을 수정예정

from spade_colorization.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.