After running.
sudo pm2 startup
and
sudo pm2 start npm --name "charting-bot-restarter" -- run restarter

I can check there are error with the charting-bog. Here the logs and capture. I dont know how solve it.

1|charting-bot  |     at require (internal/modules/cjs/helpers.js:74:18)
1|charting-bot  |     at Object.<anonymous> (/root/WeChartWeb3-master/workers/lib/block.listner.js:1:33)
1|charting-bot  |     at Module._compile (internal/modules/cjs/loader.js:999:30)
1|charting-bot  |     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
1|charting-bot  |     at Module.load (internal/modules/cjs/loader.js:863:32)
1|charting-bot  |     at Function.Module._load (internal/modules/cjs/loader.js:708:14) {
1|charting-bot  |   code: 'MODULE_NOT_FOUND',
1|charting-bot  |   requireStack: [
1|charting-bot  |     '/root/WeChartWeb3-master/workers/lib/web3.js',
1|charting-bot  |     '/root/WeChartWeb3-master/workers/lib/block.listner.js',
1|charting-bot  |     '/root/WeChartWeb3-master/workers/lib/scrape.block.stream.js',
1|charting-bot  |     '/root/WeChartWeb3-master/workers/updater/master.js'
1|charting-bot  |   ]
1|charting-bot  | }

Thanks!

Hi @Linch1 ,

Thanks for the script.

Just a question, how to run price scrapper for specific token or pair?

Thanks

Hello, can you help with the installation, for a fee. I didn't. [email protected]

Hey mate, Hope you're well.

Is it easy to limit the number of requests in a batch call for instance, less than 500.

For example, analyseBlock-> getBlockSyncEvents->getReceiptsBatch(transactions) is returning error due to the batch size being greater than 500.

Transaction hashes are 1200 and the node provider is limiting to 500 in a batch call.

Can we update getReceiptsBatch(hashes) function in this file ./workers/lib/scrape.block.past.js to split the hashes array into multiple as per the set size like 500 in each array.

Any thoughts please?

I scrap price doge coin,
Looks like the calculation: if (mainToken.contract == token0) dependantTokenPrice = reserve0 / reserve1; // here decimals
else dependantTokenPrice = reserve1 / reserve0; Not true in this case

When I trying run, error arises. P.S. I install full-node provider https://github.com/self-node-official and it run successfully, also I do all Setup steps, ws and http URL added
node --version v12.22.12
npm --version 6.14.16
Can you help me?

npm run scraper

[email protected] scraper /root/WeChartWeb3
node ./workers/updater/master.js

WARNING: NODE_ENV value of 'staging' did not match any deployment config file names.
WARNING: See https://github.com/node-config/node-config/wiki/Strict-Mode
WARNING: No configurations found in configuration directory:/root/WeChartWeb3/config
WARNING: To disable this warning set SUPPRESS_NO_CONFIG_WARNING in the environment.
/root/WeChartWeb3/workers/lib/web3.js:6
web3_wss.setProvider(new Web3.providers.WebsocketProvider( scraperConfig[process.env.CHAIN_ID].ws_provider ));
^

TypeError: Cannot read property 'ws_provider' of undefined
at Object. (/root/WeChartWeb3/workers/lib/web3.js:6:96)
at Module._compile (internal/modules/cjs/loader.js:999:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
at Module.load (internal/modules/cjs/loader.js:863:32)
at Function.Module._load (internal/modules/cjs/loader.js:708:14)
at Module.require (internal/modules/cjs/loader.js:887:19)
at require (internal/modules/cjs/helpers.js:74:18)
at Object. (/root/WeChartWeb3/workers/lib/block.listner.js:1:33)
at Module._compile (internal/modules/cjs/loader.js:999:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
at Module.load (internal/modules/cjs/loader.js:863:32)
at Function.Module._load (internal/modules/cjs/loader.js:708:14)
at Module.require (internal/modules/cjs/loader.js:887:19)
at require (internal/modules/cjs/helpers.js:74:18)
at Object. (/root/WeChartWeb3/workers/lib/scrape.block.stream.js:17:23)
at Module._compile (internal/modules/cjs/loader.js:999:30)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] scraper: node ./workers/updater/master.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] scraper script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2022-12-14T10_45_38_579Z-debug.log

Hello!

Can you help me to understand a couple of things?

1. What mean this? And how you found them?
   let swapSign = "0xd78ad95fa46c994b6551d0da85fc275fe613ce37657fb8d5e3d130840159d822"; let syncSign = "0x1c411e9a96e071241c2f21f7726b17ae89e3cab4c78be50e062b03a9fffbbad1"; let transferSign = "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef";

2. How to parse more swaps? I want to add Biswap, 1inch and others?

3. And the last one :) How to add more networks to this scraper?

Thank you very much for your job!

first, to all thanks for your great app I have tried to run this but I face this error.

javedali@Javeds-MacBook-Pro WeChartWeb3-master % npm run scraper

[email protected] scraper /Users/javedali/Documents/WeChartWeb3-master
node ./workers/updater/master.js

[LAST SCRAPED]
MongoDB connected
MongoDB connected
[NO TOPICS] false
(node:35002) UnhandledPromiseRejectionWarning: Error: cannot convert null value to array
at toByteArray (/Users/javedali/Documents/WeChartWeb3-master/workers/lib/bloomfilter.js:41:15)
at keccak256 (/Users/javedali/Documents/WeChartWeb3-master/workers/lib/bloomfilter.js:10:35)
at setInBloom (/Users/javedali/Documents/WeChartWeb3-master/workers/lib/bloomfilter.js:132:18)
at BlockInfo.addLog (/Users/javedali/Documents/WeChartWeb3-master/workers/lib/block.listner.js:27:27)
at BlockListener._processLog (/Users/javedali/Documents/WeChartWeb3-master/workers/lib/block.listner.js:108:19)
at Timeout._onTimeout (/Users/javedali/Documents/WeChartWeb3-master/node_modules/web3-core-subscriptions/lib/subscription.js:257:23)
at listOnTimeout (internal/timers.js:557:17)
at processTimers (internal/timers.js:500:7)
(Use node --trace-warnings ... to show where the warning was created)
(node:35002) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:35002) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:35002) UnhandledPromiseRejectionWarning: Error: Returned error: invalid argument 0: hex string without 0x prefix
at Object.ErrorResponse (/Users/javedali/Documents/WeChartWeb3-master/node_modules/web3-core-helpers/lib/errors.js:28:19)
at /Users/javedali/Documents/WeChartWeb3-master/node_modules/web3-core-requestmanager/lib/index.js:302:36
at XMLHttpRequest.request.onreadystatechange (/Users/javedali/Documents/WeChartWeb3-master/node_modules/web3-providers-http/lib/index.js:98:13)
at XMLHttpRequestEventTarget.dispatchEvent (/Users/javedali/Documents/WeChartWeb3-master/node_modules/xhr2-cookies/dist/xml-http-request-event-target.js:34:22)
at XMLHttpRequest._setReadyState (/Users/javedali/Documents/WeChartWeb3-master/node_modules/xhr2-cookies/dist/xml-http-request.js:208:14)
at XMLHttpRequest._onHttpResponseEnd (/Users/javedali/Documents/WeChartWeb3-master/node_modules/xhr2-cookies/dist/xml-http-request.js:318:14)
at IncomingMessage. (/Users/javedali/Documents/WeChartWeb3-master/node_modules/xhr2-cookies/dist/xml-http-request.js:289:61)
at IncomingMessage.emit (events.js:387:35)
at endReadableNT (internal/streams/readable.js:1317:12)
at processTicksAndRejections (internal/process/task_queues.js:82:21)
(node:35002) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
MAIN_PRICE: 248.33250196119187
MAIN_PRICE: 248.33250196119187

What is process.env.SERVER_VOTING url @Linch1 .
please provide url information @Linch1

Thanks,

https://github.com/Linch1/Web3TokenCharting/blob/master/readme.md?plain=1#L19

Missing the example .env file stated in the readme.md can you provide this?

I've have the script running for a few hours now and nothing is present in the database.
The collections are created but they're empty.

I see this error here and there in the logs:

[ERR UPDATING MAIN PRICE] Error: Returned values aren't valid, did it run Out of Gas? You might also see this error if you are not using the correct ABI for the contract you are retrieving data from, requesting data from a block number that does not exist, or querying a node which is not fully synced

Any ideas what might be happening?
Thanks.

Hello @Linch1

I observed that some transactions would not be recorded, But those transactions did happen. I don't know what the issue is? Maybe this transactions is through bridges or unpopular swap platforms?

some unrecorded transactions
https://bscscan.com/tx/0xaed9c91373cc56872ad45b23452f4385a975a59a0089fe16ffd95516f1ae8767
https://bscscan.com/tx/0x461955bbe430d62a888b32a0568834ab0de0dd47903961d02a356df5ac5af494
https://bscscan.com/tx/0xaf0b9fa92e11e7984d8872dd8407f6f200dfd2ac8a360e27d3882c424d1cb54f
https://bscscan.com/tx/0x9c622a8045af144761e633f049f8b1fa4bc2be3b9c304c910a859529a4ebd7a0

attention to this token(Wombat), issue can easily recur.
0xAD6742A35fB341A9Cc6ad674738Dd8da98b94Fb1
https://dexscreener.com/bsc/0xAD6742A35fB341A9Cc6ad674738Dd8da98b94Fb1

If i add cake LIKE [EnumChainId.BSC]: {
BTC: {
address: UtilsAddresses.toCheckSum("0x7130d2a12b9bcbfae4f2634d864a1ee1ce3ead9c"),
decimals: 18
},//
// DOT: {
// address: UtilsAddresses.toCheckSum("0x7083609fce4d1d8dc0c979aab8c869ea2c873402"),
// decimals: 18
// },//

    WBNB: {
        address: UtilsAddresses.toCheckSum("0xbb4cdb9cbd36b01bd1cbaebf2de08d9173bc095c"),
        decimals: 18
    },
    BUSD: {
        address: UtilsAddresses.toCheckSum("0xe9e7cea3dedca5984780bafc599bd69add087d56"),
        decimals: 18
    },//
    USDC: {
        address: UtilsAddresses.toCheckSum("0x8ac76a51cc950d9822d68b83fe1ad97b32cd580d"),
        decimals: 18
    },//
    USDT: {
        address: UtilsAddresses.toCheckSum("0x55d398326f99059ff775485246999027b3197955"),
        decimals: 18
    },//
    CAKE: {
        address: UtilsAddresses.toCheckSum("0x0e09fabb73bd3ade0a17ecc321fd13a19e81ce82"),
        decimals: 18
    }
}

}

EnumMainTokens[EnumChainId.BSC].STABLECOINS = [ "BUSD", "USDC", "USDT", "CAKE" ].map( e => EnumMainTokens[EnumChainId.BSC][e].address ) ===> dependantToken: "0x0E09FaBB73Bd3Ade0a17ECC321fD13a19e81cE82" scrap always don't have data

Hi, Linch1! Thanks for your great work.
At present, the run of the scraper is normal but noticed that Mongodb's CPU usage is often too high, and attempts to restart are ineffective. It seems to me that this is a question about performance optimization?

Do you have any solutions and suggestions?

thanks!

Hey Linch1,

Thanks for sharing your code, I am building an own version and basically its almost done. I have a question about the pancakeswap functions, as I checked your code it seems that your code is only interested in 6 out of the 9 swap functions of the router contract. And I checked bsc and see the missing 3 functioms are used, but I do not fully understand if they are needed. Maybe we can get in touch if you have a few minutes spare time?

Hello! can you explain why we need tokenhistories collection?

Hello, on which part i need to change, if i want to start the scrape from the specific block ?

What happens if i change number of queue from 100 to 1000 or 10 , is it fast ? If i want change speed scrap, What should i do ??

Can i pair only token/btc, token/usdt, token/doge, token/eth, token/usdc ? i wan't same poocoin.app =)

When running price scraper from the start block, the below error occurs.

Looking at the transaction hash, it contains ERC-721 transfer or mint.

Had error: 0x65269b3f27fd7485049887e0165d1880006cf85ca8706eb03fc011864525c584 Error: data out-of-bounds (length=0, offset=32, code=BUFFER_OVERRUN, version=abi/5.5.0)

Thanks

I have this error on my production server when run npm run scraper, but on local all good. Can you help?

Hello - First, thank you for your contribution. It's nice to see something that actually runs.

I've run this in the following environments: Windows10, Windows WSL Ubuntu, and a dedicated Ubuntu Server. Windows and Dedicated Ubuntu runs will with the exception of the issue I'm reporting. WSL Ubuntu has some issues with mongo connections.

The problem I'm reporting occurs both in Windows10 and in the dedicated Ubuntu server environment.

When running the scraper, the slave (I assume) will occasionally throw an error and the slave will die, but the main thread continues to run and just keeps displaying "Main Price xxxxxx.xxxxx" over and over. No other messages are displayed.

When the slave dies, a stack trace is displayed, but as I said, the main thread knows nothing about it. I would think the main thread needs to either monitor the slave with some kind of heartbeat and restart it if it dies, or if the slave dies the whole thing dies. I'd much rather it just restart the slave automagically. I've included the stack trace for your review :

Distributor ID: Ubuntu
Description: Ubuntu 20.04.4 LTS
Release: 20.04
Codename: focal
Node Version 12.16.2
Full Node On Chainstack : Will provide credentials for testing if needed.

[SCRAPED BLOCK][OLD][CB] 20613920 0
(node:1459182) UnhandledPromiseRejectionWarning: Error: Invalid JSON RPC response: ""
at Object.InvalidResponse (/export/home/exchange/wsock_server/WeChartWeb3/node_modules/web3-core-helpers/lib/errors.js:43:16)
at XMLHttpRequest.request.onreadystatechange (/export/home/exchange/wsock_server/WeChartWeb3/node_modules/web3-providers-http/lib/index.js:95:32)
at XMLHttpRequestEventTarget.dispatchEvent (/export/home/exchange/wsock_server/WeChartWeb3/node_modules/xhr2-cookies/dist/xml-http-request-event-target.js:34:22)
at XMLHttpRequest._setReadyState (/export/home/exchange/wsock_server/WeChartWeb3/node_modules/xhr2-cookies/dist/xml-http-request.js:208:14)
at XMLHttpRequest._onHttpRequestError (/export/home/exchange/wsock_server/WeChartWeb3/node_modules/xhr2-cookies/dist/xml-http-request.js:349:14)
at ClientRequest. (/export/home/exchange/wsock_server/WeChartWeb3/node_modules/xhr2-cookies/dist/xml-http-request.js:252:61)
at ClientRequest.emit (events.js:310:20)
at TLSSocket.socketOnEnd (_http_client.js:460:9)
at TLSSocket.emit (events.js:322:22)
at endReadableNT (_stream_readable.js:1187:12)
at processTicksAndRejections (internal/process/task_queues.js:84:21)
(node:1459182) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:1459182) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Thank you again for your hard work and contribution

Hey, is there any ETA for adding Ethereum node scraping support?