Scheduler on Heroku OOMs (critical) about metaforecast HOT 8 CLOSED

Seems fine now (cronjob broke today on algolia step due to misconfiguration, sorry; but I see in logs that you've already fixed that).

from metaforecast.

Increasing dyno size is ok as a hotfix, yes.

from metaforecast.

After some investigating:

• polymarket-fetch uses around 260MB if run by itself
• there might be also be memory leaks from the previous steps, I'm not sure
• Heroku limits are 512MB on Hobby/Free and on Standard 1X; that sounds quite limiting to me, considering that we might want to fetch huge JSON files for the backend
• Heroku's Standard 2X with 1GB RAM is $50/month :( on DO 1GB droplet is just $6/month (and on Hetzner we can get 4GB RAM VPS for €5.83)

I'm running updates by hand on my machine for now (everything from 9=polymarket).

from metaforecast.

polymarket-fetch uses around 260MB if run by itself

This number is hard to bring down for now due to how platform fetchers are currently implemented (load all data in memory and then insert everything to the DB).

So I enabled pro dynos on heroku and changed the main cron job to use Standard-2X, that should be enough. It shouldn't be as bad as $50/month just for that, since Heroku charges only for a fraction of time the dyno runs, so might be around $10/month or so.

from metaforecast.

Received, seems ok for now.

from metaforecast.

I reworked the polymarket fetcher. It should be significantly less memory intensive now. In particular, we were fetching all markets (including resolved ones), which wasn't necessary. I've reverted the change and we are now using free dynos, but feel free to restore them.

from metaforecast.

Oh, great. Btw, are https://strapi-matic.poly.market/markets query parameters documented anywhere, or are you just reverse-engineering urls from https://polymarket.com/? I couldn't find anything useful on google.

from metaforecast.

Both. There is a devs channel on the polymarket Discord on which this is kinda-but-not-really documented. In this case, I did reverse engineer the parameters.

from metaforecast.