Giter VIP home page Giter VIP logo

Comments (4)

king-menin avatar king-menin commented on July 24, 2024

try to decrease fp16_opt_level

from ru-gpts.

fen0s avatar fen0s commented on July 24, 2024

try to decrease fp16_opt_level

Yeah, the problem here is that with O2 it still doesn't want to run training and goes OOM, even at lesser block sizes. Weird thing here is that gradient checkpointing is present in your model implementation, but somehow it doesn't optimize model enough for greater order models to be ran on less VRAM hardware, which is really weird... Still trying out different things to make it work.

from ru-gpts.

Dmitriuso avatar Dmitriuso commented on July 24, 2024

Hey guys,
@fen0s I'm trying to do the same thing - fine-tune GPT2 Large model in Colab, but I run out of RAM as well as you do. I tried to resolve this issue with a loop while True, but it doesn't seem to work either. Did you manage to find something to make it work? 🤞

from ru-gpts.

king-menin avatar king-menin commented on July 24, 2024

try use deepspeed version of script

from ru-gpts.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.