JavaScript heap out of memory about twitch-clip-downloader HOT 3 CLOSED

Hmm, looking at the logs it seems like it's happening at around 200k clips, which is lower than what I could get from previous versions of this tool (afaik I managed to get 300k at some point with no issues).

I'm not sure if there are obvious memory hogs I could fix, but I'll play around so see what I can get done.

In the mean time, since you have plenty more RAM available, I suggest bumping the heap size while I work this out.

EDIT1: I've been attempting to reproduce this problem in my machine with Hasan's channel and realized that the period splitting logic is going crazy splitting a whole month into <3h periods. Since this is a recursive call I imagine alot of variables are being held in memory until the code finds a period small enough to get <500 clips.

EDIT2: Just noticed a giant mistake in the caching system. Instead of each period file storing clips found ONLY in that period, it saved every clip that was found at that moment. This caused the /cache directory to grow beyond 10GB. Because of this CPU usage was VERY high, memory usage was higher than usual (the main problem was elsewhere), and my SSD to be suffocated to death. A fix in being tested right now.

from twitch-clip-downloader.

Can you please check if the most recent version fixes your problem?

from twitch-clip-downloader.

Hey sorry for the delay. It works perfectly. Was able to find 433,304 clips and save to the meta file.

Thank you very much

from twitch-clip-downloader.