grayedfox / blockscrape Goto Github PK

Promised feature inside of Coming Soon, pretty self explanatory

• update requirements to say that local chain is optional (although strongly recommended)
• update coming soon section(s)
• update supported blockchains section and specify which ones are supported remotely via blockcyher
• move "remote-able" into current features

Initial goal was to allow for easy exendability in terms of supporting diferebt blockchains and although it's certainly possible it's a bit more complex than my initial estimates.

• write a guide which explains the steps for adding a new blockchain (ideally: check their docs, map paths to local api commands like getTransactionByHash, then add default params where required)
• encourage users to make requests if they're happy to wait a little and don't want to touch the code

The current logic of the program writes a block, and all of its transactions, in a single write and retains the scraped order when writing to the exportedData.csv file.

While this is good for later data analysis (and saves us having to sort hundreds of millions of lines later on) the write mechanism will wait until a worker sends it the next descending block before it writes, which can result in the following bug:

• master process writes block 1000 and all transactions to file
• worker 1 finishes block 1002, sending required data to master
• worker 2, 3, and 4 are still busy with blocks 1001, 1003, 1004
• master stores block 1002 data in memory, tells worker 1 to go to next available block
• worker 1 starts scraping block 1005
• worker 3 and 4 finish, master also adds their blocks to memory (block 1003 and 1004), since it hasn't received block 1001 yet, and start scraping next blocks
• worker 2 dies/freezes/poos for some reason, so block 1001 is never sent to master
• remaining workers continue to scrape next available blocks but master never restarts worker 1 and (currently) will never actually write any more data, since worker 1 is never restarted and remaining workers will continue to scrape blocks in order of blockHeight

This can result in the scraper doing all of it's good work but never exporting new data since it refuses to write blocks out of order.

Possible fixes:

• allow blocks to be written and skip a block sometimes if the blocksToWrite array is getting too full (more of a workaround)
• make workers smarter so that they will only scrape the next available block if a previous block is not available to work on (this case should only happen when a worker errors or dies)
• above solution would require proper error handling so that workers save failed blocks somewhere

Ideally string responses that are JSON formatted should be parsed inside of the API and not multiple times inside the scraper file