Giter VIP home page Giter VIP logo

Comments (6)

Linch1 avatar Linch1 commented on June 6, 2024

Ehi @GuolongCyber, when you say

and attempts to restart are ineffective.

you mean restarting mongodb or the scraper with the restarter.js script ?

All the optimization emprovements (like cache system or bulk writes to db) that came to my mind are already there,
initally this bot could consume 2-3 times more cpu than it does now,
at the moment i don't know how to emprove it efficency unfortunately.

Thanks for the report πŸ˜ƒ ,
If you have any sudjestion let me know πŸ‘

from wechartweb3.

GuolongCyber avatar GuolongCyber commented on June 6, 2024

I mean restart mongodb
I've been running for about 3 days and started to fill up the CPU. Do you have a similar situation?

from wechartweb3.

Linch1 avatar Linch1 commented on June 6, 2024

Not really, maybe i'm running a server with more resources, I'm using this one

https://www.namecheap.com/hosting/dedicated-servers/configure?server=dual_xeon_gold_5218_128gb_ddr4_4_x_1tb_ssd_nvme&period=p1m&redirectUri=configure-server&os=centos&cpanel=none&interworx=none&softaculous=0&whmcs=none&ips=0&management=user

on my end mongodb consume 60% cpu on average.
At the end the scraper is not doing one of the less heavy works, scraping every swap and analizing it's parameters for each block it's a bit resource consuming

from wechartweb3.

GuolongCyber avatar GuolongCyber commented on June 6, 2024

We use the same server configuration. I put the scraper on the selfnode server!😍

I just used the default startup mode:
bin/mongod --port=27017 --dbpath=mongodb/data --logpath=mongodb/log/mongodb.log --fork

Have you made any adjustments😁?
such as: Ulimit\Virtual Memory\Transparent HugePages\Filesystem and Options

Thanks buddy!

from wechartweb3.

GuolongCyber avatar GuolongCyber commented on June 6, 2024

i fixed! I didn't create indexes 😭

Now the running speed is super fast!! πŸš€

Thanks @Linch1 πŸ™Œ

from wechartweb3.

Linch1 avatar Linch1 commented on June 6, 2024

Ohh that's right, forgot to mention that, happy to know you figured that out πŸ˜„

from wechartweb3.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.