eosio / demux-js Goto Github PK

View Code? Open in Web Editor NEW

305.0 305.0 78.0 3.23 MB

💫 Deterministic event-sourced state and side effect handling for blockchain applications

License: MIT License

Shell 3.47% TypeScript 96.53%

demux-js's People

Contributors

Stargazers

Watchers

demux-js's Issues

TS7016: Could not find declaration for module 'bunyan'

The definition for AbstractActionHandler refers to the bunyan logger which prevents Typescript from compiling if you are not using that logger.

The workaround is to install the typings for it via @types/bunyan, regardless if you are using Bunyan or not. However, this shouldn't be the case as that logger isn't really needed anywhere in the typings for the handler.

can't run examples

Tried running

$ ./run-example.sh
module.js:557
    throw err;
    ^

Error: Cannot find module '/root/demux-js/examples/index.js'
    at Function.Module._resolveFilename (module.js:555:15)
    at Function.Module._load (module.js:482:25)
    at Function.Module.runMain (module.js:701:10)
    at startup (bootstrap_node.js:194:16)
    at bootstrap_node.js:618:3

Installation

Hello,

I'm getting during the installation the following error:

# Using yarn
yarn add demux

# Using npm
npm install demux --save

33 error Linux 4.15.0-33-generic
34 error argv "/usr/bin/node" "/usr/bin/npm" "install" "demux" "--save"
35 error node v8.10.0
36 error npm  v3.5.2
37 error code ENOSELF
38 error Refusing to install demux as a dependency of itself
39 error If you need help, you may report this error at:
39 error     <https://github.com/npm/npm/issues>
40 verbose exit [ 1, true ]

block handler

Hi.

Are you going to make block handler any time soon to watch for whole blocks?

Does demux check action's status?

For example, hard failed attack

How to watch block generation process?

In the example eos-transfers, is watch the transfer data. Now I want to watch block generation process. How should I do in the code?

bnet support for sync data

Implement bnet protocol from nodeos in order to sync with blockchain.

Right now using http rpc endpoints will take too much time if you get out of sync.

As far as I know bnet is websockets protocol.

MySQL support

We (Everipedia) are forking this repo to add MySQL support and plan on submitting a pull request. Before starting, we want to make sure someone at Block.one isn't already working on this. If they are, we'll stop our coding and wait on them to prevent conflicts.

Send email on action

Hello,
It would be nice to have an email sent on transfer to a selected account(s).
If someone adds an example for this effect handler, it would be appreciated.

How can I use demux in my test network?

I change nodeosEndpoint into my test network endpoint,

const actionReader = new NodeosActionReader(
  "http://192.168.1.107:8888", 
  0,
)

but terminal didn't print anything when I complete a transition.

Add "action" and "account" fields for BunyanLogger

It would be helpful to include an action and account field to the demux logs. This issue is simply a reminder to look into where/how we can accomplish this.

Error: currentBlockData must not be null

When am pushing the data into eosio,am getting error:currentBlockData must not be null

(node:2986) DeprecationWarning: current URL string parser is deprecated, and will be removed in a future version. To use the new parser, pass option { useNewUrlParser: true } to MongoClient.connect.
Example app listening on port 4000!
Mongoose default connection open to mongodb://127.0.0.1/TweetDatabase
(node:2986) UnhandledPromiseRejectionWarning: Error: currentBlockData must not be null.
    at NodeosActionReader.<anonymous> (/home/swapna/Documents/decenttwt/backend/node_modules/demux/dist/AbstractActionReader.js:76:23)
    at Generator.next (<anonymous>)
    at fulfilled (/home/swapna/Documents/decenttwt/backend/node_modules/demux/dist/AbstractActionReader.js:4:58)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)
(node:2986) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:2986) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

The same state is printed more than once.

I run examples/eos-transfers/ and got the results like below:

State updated:
 {
  "volumeBySymbol": {
    "EOS": 200
  },
  "totalTransfers": 1,
  "indexState": {
    "blockNumber": 9824478,
    "blockHash": "0095e8de6fb6c6cd022963ab5d34f1ede76d5015e9b4c5e2046df621ceccc294"
  }
}
State updated:
 {
  "volumeBySymbol": {
    "EOS": 2037.2462999999998
  },
  "totalTransfers": 4,
  "indexState": {
    "blockNumber": 9824585,
    "blockHash": "0095e949ff4b903ae495895510e4a03cbc406cf4af3c6a6ff73f056d65067417"
  }
}
State updated:
 {
  "volumeBySymbol": {
    "EOS": 2037.2462999999998
  },
  "totalTransfers": 4,
  "indexState": {
    "blockNumber": 9824585,
    "blockHash": "0095e949ff4b903ae495895510e4a03cbc406cf4af3c6a6ff73f056d65067417"
  }
}
State updated:
 {
  "volumeBySymbol": {
    "EOS": 2037.2462999999998
  },
  "totalTransfers": 4,
  "indexState": {
    "blockNumber": 9824585,
    "blockHash": "0095e949ff4b903ae495895510e4a03cbc406cf4af3c6a6ff73f056d65067417"
  }
}
State updated:
 {
  "volumeBySymbol": {
    "EOS": 2037.2472999999998
  },
  "totalTransfers": 5,
  "indexState": {
    "blockNumber": 9824595,
    "blockHash": "0095e953419c71e69787aa35dfd91998357c82ddfb70b6886a2553091dd9d8ca"
  }
}

This is because that handleActions() in src/demux/handlers/AbstractActionHandler.ts first processes the actions in runUpdaters() and then processes them in runEffects(). So if a block contains more than one monitored actions, the updater gets called a few times and so does the state, before the effect is called.

Is this an expected behavior (so that the user of demux-js should be careful about their usage) or an issue of demux-js?

How can I get a stream of smart contract table changes?

For example I want to get real time changes of this table, glad to accept every help

https://bloks.io/account/betdiceadmin?loadContract=true&tab=Tables&account=betdiceadmin&scope=betdiceadmin&limit=100&table=user

What is the significance of- "12345678, // First actions relevant to this dapp happen at this block"?

Can this be the current block on the mainnet?

Error retrieving last irreversible block

I keep running into this error after setting up using the example code in the repo. I've tried 3 different api endpoints. But all return same error.

Some tests are failing.

Tests in the MassiveActionHandler are failing when run locally or through travis-ci.

 PASS  src/demux/handlers/AbstractActionHandler.test.ts
 PASS  src/demux/readers/eos/NodeosActionReader.test.ts
 FAIL  src/demux/handlers/postgres/MassiveActionHandler.test.ts (15.831s)
  ● MassiveActionHandler › populates database correctly

    error: syntax error at or near "is_replay"

      at Connection.Object.<anonymous>.Connection.parseE (node_modules/pg/lib/connection.js:553:11)
      at Connection.Object.<anonymous>.Connection.parseMessage (node_modules/pg/lib/connection.js:378:19)
      at Socket.<anonymous> (node_modules/pg/lib/connection.js:119:22)

  ● MassiveActionHandler › populates database correctly

    error: relation "public.task" does not exist

      at Connection.Object.<anonymous>.Connection.parseE (node_modules/pg/lib/connection.js:553:11)
      at Connection.Object.<anonymous>.Connection.parseMessage (node_modules/pg/lib/connection.js:378:19)
      at Socket.<anonymous> (node_modules/pg/lib/connection.js:119:22)

  ● MassiveActionHandler › populates database correctly

    TypeError: Cannot read property 'findOne' of undefined

      42 |
      43 |   protected async loadIndexState(): Promise<IndexState> {
    > 44 |     const { blockNumber, blockHash } = await this.massiveInstance._index_state.findOne({ id: 0 })
      45 |     if (blockNumber && blockHash) {
      46 |       return { blockNumber, blockHash }
      47 |     }

      at MassiveActionHandler.<anonymous> (src/demux/handlers/postgres/MassiveActionHandler.ts:44:80)
      at src/demux/handlers/postgres/MassiveActionHandler.ts:24:71
      at Object.<anonymous>.__awaiter (src/demux/handlers/postgres/MassiveActionHandler.ts:4:12)
      at MassiveActionHandler.loadIndexState (src/demux/handlers/postgres/MassiveActionHandler.ts:69:16)
      at MassiveActionHandler.<anonymous> (src/demux/handlers/AbstractActionHandler.ts:33:97)
      at src/demux/handlers/AbstractActionHandler.ts:24:71
      at Object.<anonymous>.__awaiter (src/demux/handlers/AbstractActionHandler.ts:4:12)
      at MassiveActionHandler.handleBlock (src/demux/handlers/AbstractActionHandler.ts:51:16)
      at Object.<anonymous> (src/demux/handlers/postgres/MassiveActionHandler.test.ts:52:25)
      at fulfilled (src/demux/handlers/postgres/MassiveActionHandler.test.ts:9:32)

Test Suites: 1 failed, 3 passed, 4 total
Tests:       1 failed, 9 passed, 10 total```

Significant state about forks is stored in ephemeral blockHistory

Rollbacks can only be initiated when the action reader sees them happening, with its occurrence stored in AbstractActionReader#blockHistory. If the indexState is on a forked block at startup, then the action reader will not have any way of knowing this and send a block whose blockHash does not match, and it is not trivial to determine how far the Handler needs to roll back.

something about actionType?

Hi.
i need to intercept all the actions from actionType.
Official example writtend with eosio.token.
but i want to change to other contract.

something like
contract : eosiochaince
action : transfer

so i think => actionType : eosiochaince::transfer

but it is not working. what's wrong with this..?
thank you.

Demux Unable to catch up to head block [help]

I have been using demux for a few months now. It appears that with the number of transactions picking up, the demux becomes slower than usual. The demux instance that i ran becomes unable to catch up with the speed of blocks being generated every seconds.

I was wondering if there are anyone facing this issue and if there are any suggestions on how to improve the situation.

I am using the following demux js eos state history :
https://github.com/simplex-software/demux-js-eos-state-history

Nodeos server is an in-house instance (fully synched).

Thanks.

The eos-transfers example crashes when started in replay mode and start at block -1

When the eos-transfers example is run via yarn example eos-transfers, it works. However, is you start modifying values to see demux's difference behaviours, it crashes.

Changing startAtBlock from 50000000 to -1:

const actionReader = new NodeosActionReader({
  startAtBlock: -1,
  onlyIrreversible: false,
  nodeosEndpoint: "https://api.eosnewyork.io"
})

And changing watch to be in replay mode:

actionWatcher.watch(true)

Will crash on the first block with this error:

[2019-07-11T16:16:40.446Z] ERROR: demux/32914: Invalid array length
    RangeError: Invalid array length
        at ObjectActionHandler.rollbackTo (/Users/olivier/projects/demux-js-dfuse/node_modules/demux/examples/eos-transfers/ObjectActionHandler.js:68:26)
        at ObjectActionHandler.<anonymous> (/Users/olivier/projects/demux-js-dfuse/node_modules/demux/examples/eos-transfers/node_modules/demux/dist/AbstractActionHandler.js:187:28)
        at Generator.next (<anonymous>)
        at /Users/olivier/projects/demux-js-dfuse/node_modules/demux/examples/eos-transfers/node_modules/demux/dist/AbstractActionHandler.js:7:71
        at new Promise (<anonymous>)
        at __awaiter (/Users/olivier/projects/demux-js-dfuse/node_modules/demux/examples/eos-transfers/node_modules/demux/dist/AbstractActionHandler.js:3:12)
        at ObjectActionHandler.handleRollback (/Users/olivier/projects/demux-js-dfuse/node_modules/demux/examples/eos-transfers/node_modules/demux/dist/AbstractActionHandler.js:182:16)
        at ObjectActionHandler.<anonymous> (/Users/olivier/projects/demux-js-dfuse/node_modules/demux/examples/eos-transfers/node_modules/demux/dist/AbstractActionHandler.js:54:24)
        at Generator.next (<anonymous>)
        at fulfilled (/Users/olivier/projects/demux-js-dfuse/node_modules/demux/examples/eos-transfers/node_modules/demux/dist/AbstractActionHandler.js:4:58)

It is caused by ObjectActionHandler's rollbackTo trying to create an array with negative length, since latestBlockNumber is 0 (by default), and blockNumber will be something like 68149703.

I'm trying to understand why is a rollback attempted here, maybe you can shed some light on this.

BaseActionWatcher should require an IActionReader instead of an AbstractActionReader

I am in the process of writing an AbstractActionReader to connect to the dfuse.io GraphQL API rather than directly to a Nodeos via demux-eos.

I'm running into a few issues. Since dfuse navigates forks for us, and the way rollbacks are handled is different, I don't need most of the private methods in AbstractActionReader. The blockHistory is unnecessary to solve this problem. So ideally, I'd prefer to use implements AbstractActionReader rather than extends AbstractActionReader.

However BaseActionWatcher requires an AbstractActionReader, which is a bit problematic in Typescript. In TS, you need to have all private methods and fields defined to match, and even then the constructors won't match so it won't compile.

A solution would be to have BaseActionReader require something that implements IActionReader rather than an instance of AbstractActionReader.

Let me know if this makes sense, and I can proceed to the implementation right away.

How to call the data in the smart contract

With using the DEMUX, if the data is stored in mangoDB, how can the smart contract call the data which on the mongodb

Replace nodeos endpoint

http://mainnet.eoscalgary.io is down
Use the working https://api.eosnewyork.io

running example results in a fork detection.

yarn example eos-transfers

...
State updated:
 {
  "volumeBySymbol": {
    "EOS": 590.2256
  },
  "totalTransfers": 14,
  "indexState": {
    "blockNumber": 8655579,
    "blockHash": "008412db36228bc3cce31eeede203a8ba7962f8c8d133de8438182464ab988b1"
  }
}
!! Fork detected !!
✓ BLOCK 8655587 MATCH:
  expected: 008412e2859f7530e9f130837b65f28ce0881a209013afae86a91579fd90aab4
  received: 008412e2859f7530e9f130837b65f28ce0881a209013afae86a91579fd90aab4
Rewinding 1 blocks to block (8655587)...
(node:2230) UnhandledPromiseRejectionWarning: TypeError: this.rollbackTo is not a function
    at ObjectActionHandler.<anonymous> (/root/demux-js/dist/demux/handlers/AbstractActionHandler.js:39:28)
    at Generator.next (<anonymous>)
    at /root/demux-js/dist/demux/handlers/AbstractActionHandler.js:7:71
    at new Promise (<anonymous>)
    at __awaiter (/root/demux-js/dist/demux/handlers/AbstractActionHandler.js:3:12)
    at ObjectActionHandler.handleBlock (/root/demux-js/dist/demux/handlers/AbstractActionHandler.js:37:16)
    at BaseActionWatcher.<anonymous> (/root/demux-js/dist/demux/watchers/BaseActionWatcher.js:45:75)
    at Generator.next (<anonymous>)
    at fulfilled (/root/demux-js/dist/demux/watchers/BaseActionWatcher.js:4:58)
    at <anonymous>
(node:2230) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 3)
(node:2230) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Done in 109.09s.

how to stop watching?

I use demux as a part of my project. I dont find the method to stop watching

any plan to implement action type wildcard match or something like this?

like use *::method to match contractA::method and contractB::method
and user * to match all action type

any plan about it?

The program stops automatically and no error is reported

And after I restart the program,it works fine.But after a few hours the problem arose again.

const rawBlock = yield this.httpRequest('post', {
            url: `${this.nodeosEndpoint}/v1/chain/get_block`,
            json: {block_num_or_id: blockNumber},
          })

no timeout ??

demux version:

"demux": "^3.0.0",
"demux-eos": "2.0.0",

Multiple Contracts

If I had multiple contracts,how can demux handle ??
For Example:
EOSIO_CONTRACT_ACCOUNT=blog

BaseActionWatcher.watch should provide a way to handle error

Currently method BaseActionWatcher.watch catches errors in an internal try/catch block and silently swallow the errors. It stops watching on any error without providing any chance to retry. It returns undefined in all cases :O, and all state properties are private so there's no way to handle errors from derived classes.

Catch inline actions

Right now if an action is inside another actions this system wont catch it, data stored could be incomplete

Implement `rollbackExhausted`

Right now, AbstractActionReader.rollbackExhausted throws an error. Instead, we should default to sending a rolling back message for the last irreversible block. This covers the case where a new Demux process starts and its corresponding datastore has indexed blocks that are not currently part of the chain.

If for some reason we run out of history, and we request to roll back to a block that is in the future (according the the index state of the datastore), then the ActionHandler should then, in return, request its correct next block.

there is no demux-eos version 1.0.1

With the most recent commit 92698a9 it looks like it is now referencing demux-eos 1.0.1 however this package is not available in npm. Currently the latest version in npm is 0.0.1-hotfix

https://www.npmjs.com/package/demux-eos

Crash

demux-js/src/demux/readers/eos/NodeosActionReader.ts

Lines 28 to 37 in 0f1717f

 public async getHeadBlockNumber(): Promise<number> { 

 const blockInfo = await this.httpRequest("get", { 

 url: `${this.nodeosEndpoint}/v1/chain/get_info`, 

 json: true, 

 }) 

 if (this.onlyIrreversible) { 

 return blockInfo.last_irreversible_block_num 

 } 

 return blockInfo.head_block_num 

 }

This code will sometimes silently fail with 500 - {"code":500,"message":"Internal Service Error","error":{"code":3100002,"name":"unknown_block_exception","what":"Unknown block","details":[{"message":"Could not find block: 10895151","file":"chain_plugin.cpp","line_number":1276,"method":"get_block"}]}}'

which causes a hard crash at other points in the code

how to query millions of rows from the smart contract multindex table ?

i am building a dapp which stores millions of rows in the multindex table, how can I use demux to efficinetly query all the millions of rows quickly to feed to my analytics layer ? it would be great if you can point me to an example where demux can be used to query smart contract multindex rows ?

Replaying actions

Is there a simple way to replay the actions of a contract that demux is watching. For example you lose state and want to reiterate all the prior actions.

TypeError when running eos-transfers example

When running the unmodified eos-transfers example, after a while I get the following error:

{"name":"demux","hostname":"Ubuntu-1804-bionic-64-minimal","pid":30211,"level":30,"msg":"Rolling back 1 blocks to block 7986487...","time":"2019-01-10T05:07:43.811Z","v":0}
(node:30211) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'indexState' of undefined
    at ObjectActionHandler.loadIndexState (/home/demo/demux-js/examples/eos-transfers/ObjectActionHandler.js:56:18)
    at ObjectActionHandler.<anonymous> (/home/demo/demux-js/examples/eos-transfers/node_modules/demux/dist/AbstractActionHandler.js:165:79)
    at Generator.next (<anonymous>)
    at /home/demo/demux-js/examples/eos-transfers/node_modules/demux/dist/AbstractActionHandler.js:7:71
    at new Promise (<anonymous>)
    at __awaiter (/home/demo/demux-js/examples/eos-transfers/node_modules/demux/dist/AbstractActionHandler.js:3:12)
    at ObjectActionHandler.refreshIndexState (/home/demo/demux-js/examples/eos-transfers/node_modules/demux/dist/AbstractActionHandler.js:164:16)
    at ObjectActionHandler.<anonymous> (/home/demo/demux-js/examples/eos-transfers/node_modules/demux/dist/AbstractActionHandler.js:50:28)
    at Generator.next (<anonymous>)
    at fulfilled (/home/demo/demux-js/examples/eos-transfers/node_modules/demux/dist/AbstractActionHandler.js:4:58)
(node:30211) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 4)
(node:30211) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

I suspect it has something to do with fork handling because it usually comes immediately after a fork has been handled, but not always. Sometimes it handles a dozen forks fine and then suddenly, the error pops up.

The Jungle testnet seems to have conditions that provoke this error more often than then mainnet.

can‘t specify multiple nodes (NodeosActionReader)

Not able to match "eosio::bidname" actions.

I'm running the example code included in this repo except I have modified the effects array in attempt to match bidding on premium names.

const effects = [
  {
    actionType: "eosio::bidname",
    run: console.log,
  },
]

I've been starting the NodeosActionReader so that it replays blocks where eosio::bidname actions definitely occur but I've been unable to successfully match them. Otherwise, my project works fine for other actions like eosio.token::transfer. Also logging the blockinfo passed through to my effects handler when matching other action types does show that bidname actions are included in the block actions array.

Perhaps I am misunderstanding something about how to use this project? I'd really appreciate any advice you can give.

TypeError: this.updaters is not iterable

at ActionHandler. (/home/moon/demux/node_modules/demux/dist/AbstractActionHandler.js:74:44)
at Generator.next ()
at /home/moon/demux/node_modules/demux/dist/AbstractActionHandler.js:7:71
at new Promise ()
at __awaiter (/home/moon/demux/node_modules/demux/dist/AbstractActionHandler.js:3:12)
at ActionHandler.runUpdaters (/home/moon/demux/node_modules/demux/dist/AbstractActionHandler.js:71:16)
at ActionHandler. (/home/moon/demux/node_modules/demux/dist/AbstractActionHandler.js:103:24)
at Generator.next ()
at /home/moon/demux/node_modules/demux/dist/AbstractActionHandler.js:7:71
at new Promise ()
at __awaiter (/home/moon/demux/node_modules/demux/dist/AbstractActionHandler.js:3:12)
at ActionHandler.handleActions (/home/moon/demux/node_modules/demux/dist/AbstractActionHandler.js:101:16)
at ActionHandler. (/home/moon/demux/node_modules/demux/dist/AbstractActionHandler.js:61:28)
at Generator.next ()
at /home/moon/demux/node_modules/demux/dist/AbstractActionHandler.js:7:71
at new Promise ()

two txs in one block,but only one tx found even twice

28,313,995

4a62af888aa36d3ea5710ad3dbe1a2629a8bd16943d4038f03f66a78b374fe72
b8fa5e46700846d699bc4959c534e7db8c1a45b8dc6fa1899d6fd2e1ee4b8a8e

But only “b8fa5e46700846d699bc4959c534e7db8c1a45b8dc6fa1899d6fd2e1ee4b8a8e” was found and returned twice.
"4a62af888aa36d3ea5710ad3dbe1a2629a8bd16943d4038f03f66a78b374fe72" was ignored.

help

Some mini-fork scenarios are not handled

We've been using demux for some of our blockchain apps and we recently discovered that on occasion, demux processing will just suddenly stop. After investigation we found out that this seems due to certain mini fork scenarios that demux is not able to correct. The situation causes some versions (current develop, and 2.0.0) to run an infinite loop and never make progress on new blocks, and in another version (3.1.3), the scenario causes demux to crash. Checkout this test case which recreates the mini-forks, and the results from running on develop, and 3.1.3. Copy the test case into src/ and run the tests. You can also switch branches and select the branch in the test case. From this test case, it looks like when the index state is at the fork block, then demux doesn't really correct the fork.
Maybe the test Action* implementations have some error? Or if there is something about the demux lib itself you recommend fixing I'm happy to try to tackle it.
Thank you in advance for any feedback.

MiniFork.test.ts:

// Do an `rm -rf node_modules && npm install` followed by `npm run test`
// You will find that miniforks do not work in all scenarios on 2.0.0, 3.1.3, and the current develop

// git checkout one of these branches and uncomment the appropriate string to try it out
const branch: string = 'develop'
// const branch: string = '2.0.0'
// const branch: string = '3.1.3'

import { BaseActionWatcher } from "./BaseActionWatcher"
import { AbstractActionHandler } from "./AbstractActionHandler"
import blockchains from "./testHelpers/blockchains"
import { TestActionReader } from "./testHelpers/TestActionReader"
import { Block, IndexState, Updater, Effect } from "./interfaces"

class TestActionWatcher extends BaseActionWatcher {
  public async _checkForBlocks(isReplay: boolean = false) {
    await this.checkForBlocks(isReplay)
  }
}

class TestActionReaderAsync extends TestActionReader {
    public async getBlock(blockNumber: number): Promise<Block> {
      // Briefly give up execution control. This allows jest to check whether tests have timed out
      await new Promise(resolve => setTimeout(resolve, 0))
      return super.getBlock(blockNumber);
    }
}

class TestActionHandler extends AbstractActionHandler {
  public state: any = {
    indexDb: [],
  }

  // tslint:disable-next-line
  public async handleWithState(handle: (state: any) => void) {
    await handle(this.state)
  }

  // for the develop branch
  protected async setup(): Promise<void> {}

  public async rollbackTo(blockNumber: number) {
    const index = this.state.indexDb.findIndex((state: any) => state.blockNumber === blockNumber)
    this.state.indexDb.splice(index + 1);
  }

  protected async loadIndexState(): Promise<IndexState> {
    return this.state.indexDb[this.state.indexDb.length - 1];
  }

  protected async updateIndexState(state: any, block: Block, isReplay: boolean, handlerVersionName: string) {
    const { blockNumber, blockHash } = block.blockInfo
    state.indexDb.push({ blockNumber, blockHash, isReplay, handlerVersionName });
  }
}

let initialChain = blockchains.blockchain
let forkedChain = blockchains.forked

function newFork(initialChain: Block[], forkedChain: Block[], forkBlock: number): Array<any> {
  let actionReader: TestActionReaderAsync
  let actionHandler: TestActionHandler
  let actionWatcher: TestActionWatcher

  actionReader = new TestActionReaderAsync()

  initialChain = JSON.parse(JSON.stringify(initialChain))
  forkedChain = JSON.parse(JSON.stringify(forkedChain))
  actionReader.blockchain = forkedChain
  const updaters: Updater[] = []
  const effects: Effect[] = []

  if(branch === '2.0.0') {
    // @ts-ignore
    actionHandler = new TestActionHandler(updaters, effects)
  } else {
    // @ts-ignore
    actionHandler = new TestActionHandler([{ versionName: "v1", updaters, effects }])
  }

  if(branch === 'develop') {
    // @ts-ignore
    actionReader.isInitialized = true
  }

  actionHandler.state.indexDb = initialChain.slice(0, forkBlock).map(({blockInfo}) => {
    if(branch === 'develop') {
      return {blockNumber: blockInfo.blockNumber, blockHash: blockInfo.blockHash, isReplay: false, handlerVersionName: "v1"}
    } else {
      return {blockNumber: blockInfo.blockNumber, blockHash: blockInfo.blockHash, isReplay: false, handlerVersion: "v1"}
    }
  })

  actionWatcher = new TestActionWatcher(actionReader, actionHandler, 500)
  return [actionReader, actionHandler, actionWatcher]
}


describe("Mini Fork", () => {
  for(let block of initialChain) {
    test(`processes mini fork ${block.blockInfo.blockNumber}`, async () => {
      let [actionReader, actionHandler, actionWatcher] = newFork(initialChain, forkedChain, block.blockInfo.blockNumber)
      await actionWatcher._checkForBlocks()
      expect(actionHandler.state.indexDb.length).toEqual(5)
      expect(actionReader.currentBlockNumber).toBe(5)
      expect(actionReader.headBlockNumber).toBe(5)
    }, 3000)
  }
})

Against 3.1.3:

{"name":"demux","hostname":"raven","pid":4248,"level":30,"msg":"  MISMATCH:","time":"2019-02-05T21:48:22.536Z","v":0}
{"name":"demux","hostname":"raven","pid":4248,"level":30,"msg":"    ✓ NEW Block 3 previous: 0000000000000000000000000000000000000000000000000000000000000001","time":"2019-02-05T21:48:22.536Z","v":0}
{"name":"demux","hostname":"raven","pid":4248,"level":30,"msg":"    ✕ OLD Block 2 id:       0000000000000000000000000000000000000000000000000000000000000000","time":"2019-02-05T21:48:22.536Z","v":0}
{"name":"demux","hostname":"raven","pid":4248,"level":30,"msg":"Refetching Block 1...","time":"2019-02-05T21:48:22.536Z","v":0}
 FAIL  src/MiniFork.test.ts"raven","pid":4248,"level":30,"msg":"!! FORK DETECTED !!","time":"2019-02-05T21:48:22.554Z","v":0}
  ● Mini Fork › processes mini fork 3

    TypeError: Cannot destructure property `blockInfo` of 'undefined' or 'null'.

      191 |       if (this.currentBlockData !== null) {
      192 |         const { blockInfo: currentBlockInfo } = this.currentBlockData
    > 193 |         const { blockInfo: previousBlockInfo } = previousBlockData
      194 |         if (currentBlockInfo.previousBlockHash === previousBlockInfo.blockHash) {
      195 |           this.logForkResolved(currentBlockInfo, previousBlockInfo)
      196 |           break
      
      at TestActionReaderAsync.<anonymous> (src/AbstractActionReader.ts:193:50)
      at fulfilled (src/AbstractActionReader.ts:4:58)

Test Suites: 1 failed, 3 passed, 4 total
Tests:       1 failed, 22 passed, 23 total

Against develop:

  ● Mini Fork › processes mini fork 3

    Timeout - Async callback was not invoked within the 3000ms timeout specified by jest.setTimeout.

      100 | describe("Mini Fork", () => {
      101 |   for(let block of initialChain) {
    > 102 |     test(`processes mini fork ${block.blockInfo.blockNumber}`, async () => {
          |     ^
      103 |       let [actionReader, actionHandler, actionWatcher] = newFork(initialChain, forkedChain, block.blockInfo.blockNumber)
      104 |       await actionWatcher._checkForBlocks()
      105 |       expect(actionHandler.state.indexDb.length).toEqual(5)

      at Spec (node_modules/jest-jasmine2/build/jasmine/Spec.js:85:20)
      at Suite.Object.<anonymous>.describe (src/MiniFork.test.ts:102:5)

  ● Mini Fork › processes mini fork 4

    Timeout - Async callback was not invoked within the 3000ms timeout specified by jest.setTimeout.

      100 | describe("Mini Fork", () => {
      101 |   for(let block of initialChain) {
    > 102 |     test(`processes mini fork ${block.blockInfo.blockNumber}`, async () => {
          |     ^
      103 |       let [actionReader, actionHandler, actionWatcher] = newFork(initialChain, forkedChain, block.blockInfo.blockNumber)
      104 |       await actionWatcher._checkForBlocks()
      105 |       expect(actionHandler.state.indexDb.length).toEqual(5)

      at Spec (node_modules/jest-jasmine2/build/jasmine/Spec.js:85:20)
      at Suite.Object.<anonymous>.describe (src/MiniFork.test.ts:102:5)

{"name":"demux","hostname":"raven","pid":5871,"level":30,"msg":"API server listening on port 56544.","time":"2019-02-05T21:57:48.343Z","v":0}
{"name":"demux","hostname":"raven","pid":5871,"level":30,"msg":"API server closing down. (NOTE: This does not shut down Demux itself!)","time":"2019-02-05T21:57:48.360Z","v":0}
{"name":"demux","hostname":"raven","pid":5871,"level":30,"msg":"API server listening on port 56544.","time":"2019-02-05T21:57:48.362Z","v":0}
{"name":"demux","hostname":"raven","pid":5871,"level":30,"msg":"Starting indexing.","time":"2019-02-05T21:57:48.364Z","v":0}
{"name":"demux","hostname":"raven","pid":5871,"level":30,"msg":"API server closing down. (NOTE: This does not shut down Demux itself!)","time":"2019-02-05T21:57:48.374Z","v":0}
{"name":"demux","hostname":"raven","pid":5871,"level":30,"msg":"API server listening on port 56544.","time":"2019-02-05T21:57:48.376Z","v":0}
{"name":"demux","hostname":"raven","pid":5871,"level":30,"msg":"Starting indexing.","time":"2019-02-05T21:57:48.378Z","v":0}
{"name":"demux","hostname":"raven","pid":5871,"level":30,"msg":"Pausing indexing.","time":"2019-02-05T21:57:48.382Z","v":0}
 PASS  src/ExpressActionWatcher.test.ts
{"name":"demux","hostname":"raven","pid":5871,"level":30,"msg":"Indexing paused.","time":"2019-02-05T21:57:48.880Z","v":0}
{"name":"demux","hostname":"raven","pid":5871,"level":30,"msg":"API server closing down. (NOTE: This does not shut down Demux itself!)","time":"2019-02-05T21:57:48.919Z","v":0}
{"name":"demux","hostname":"raven","pid":5871,"level":30,"msg":"!! FORK DETECTED !!","time":"2019-02-05T21:57:48.977Z","v":0}
{"name":"demux","hostname":"raven","pid":5871,"level":30,"msg":"  MISMATCH:","time":"2019-02-05T21:57:48.977Z","v":0}
{"name":"demux","hostname":"raven","pid":5871,"level":30,"msg":"    ✓ NEW Block 5 previous: wrench","time":"2019-02-05T21:57:48.977Z","v":0}
{"name":"demux","hostname":"raven","pid":5871,"level":30,"msg":"    ✕ OLD Block 4 id:       0000000000000000000000000000000000000000000000000000000000000003","time":"2019-02-05T21:57:48.977Z","v":0}
{"name":"demux","hostname":"raven","pid":5871,"level":30,"msg":"Refetching Block 4...","time":"2019-02-05T21:57:48.977Z","v":0}
{"name":"demux","hostname":"raven","pid":5871,"level":30,"msg":"  MISMATCH:","time":"2019-02-05T21:57:48.977Z","v":0}
{"name":"demux","hostname":"raven","pid":5871,"level":30,"msg":"    ✓ NEW Block 4 previous: foo","time":"2019-02-05T21:57:48.977Z","v":0}
{"name":"demux","hostname":"raven","pid":5871,"level":30,"msg":"    ✕ OLD Block 3 id:       0000000000000000000000000000000000000000000000000000000000000002","time":"2019-02-05T21:57:48.977Z","v":0}
{"name":"demux","hostname":"raven","pid":5871,"level":30,"msg":"Refetching Block 3...","time":"2019-02-05T21:57:48.977Z","v":0}
{"name":"demux","hostname":"raven","pid":5871,"level":30,"msg":"  MATCH:","time":"2019-02-05T21:57:48.977Z","v":0}
{"name":"demux","hostname":"raven","pid":5871,"level":30,"msg":"    ✓ NEW Block 3 previous: 0000000000000000000000000000000000000000000000000000000000000001","time":"2019-02-05T21:57:48.977Z","v":0}
{"name":"demux","hostname":"raven","pid":5871,"level":30,"msg":"    ✓ OLD Block 2 id:       0000000000000000000000000000000000000000000000000000000000000001","time":"2019-02-05T21:57:48.977Z","v":0}
{"name":"demux","hostname":"raven","pid":5871,"level":30,"msg":"!! FORK RESOLVED !!","time":"2019-02-05T21:57:48.977Z","v":0}
 PASS  src/AbstractActionReader.test.ts
 PASS  src/BaseActionWatcher.test.ts
 PASS  src/AbstractActionHandler.test.ts

Test Suites: 1 failed, 4 passed, 5 total
Tests:       2 failed, 32 passed, 34 total

copy of issue in EOS #5568- Best practice question relating to Demux

Summarizing the Problem

What is the best practice when it comes to tracking RAM changes on the ledger? Consider the following situation: we have a smart contract with a series of actions that are manipulating various tables on the on-chain RAM. When a smart contract action manipulates multiple entries on the RAM tables, how do we get that information on the ledger?

Example: A user can have an account, items to trade, and can send and receive trade offers. Consider the following action:

deleteAccount

data: {

id: 7

}

The smart contract takes that action and performs the following logic:

get account with an ID of 7
get all items that belong to that account
get all offers that belong to each of those items
delete offers
delete items
delete account
The ledger, to our understanding, just sees the information passed into the account, which is:

deleteAccount

data: {

id: 7

}

Those watching the blocks (such as applications running demux to track the state of this particular dApp) would be unaware of the other entries that were manipulated on RAM (offers and items).

We are looking for a way to expose any change done to RAM on the ledger so that anybody or anything watching the blocks can derive a state locally if they replay the chain.

We have come up with a few approaches, and were wondering if there’s a best practice and/or a more appropriate way of doing this than presented below.

Approach one

Make an assumption of what will be affected by the smart contract action by looking at your local state and passing that information in the action. The local state would be derived from a demux implementation that’s watching all blocks from the chain and mapping relevant data to our local database.

So by following the example above, before we pass the deleteAccount action to the smart contract, we would first look to our database and gather all of the other entities that will be affected and include them in the payload to the smart contract.

deleteAccount

data: {

id: 7,

items: [10, 13, 44, 233, 321],

offers: [32, 108, 2332, 2433]

}

Then the logic would be as follows:

get account with an ID of 7
get all items that belong to that account
verify items retrieved equal 10, 13, 44, 233, 321
get all offers that belong to each of those items
verify the offers retrieved equal 32, 108, 2332, 2433
delete offers
delete items
delete account
If either of the verification steps fail, the action would be kicked back as invalid. But if all succeeds, then data that flows through to the ledger would fully instruct a demux implementation of how to update a local database’s state.

Approach two

The seconds approach would follow the same philosophy of packaging all information in action’s payload that will be affected by the smart contract, but instead of retrieving the information from the local database, it retrieves that information from the chain's database through an API node.

So the payload below would reach out the API node and request all items that belong to account 7. Simultaneously it would request all offers that belong to each item retrieved. Then it would include that freshly-retrieved information in the payload and pass it to the smart contract, which would follow the same logic of verifying the state is in-sync before processing.

deleteAccount

data: {

id: 7,

items: [10, 13, 44, 233, 321],

offers: [32, 108, 2332, 2433]

}

Questions

The only difference between these two approaches is the way they each get the ancillary data before passing it into the smart contract. Our questions are:

Is this design philosophy the most effective way to make the changes to RAM transparent on the ledger?
Which approach would be quicker while keeping in mind the necessity for accuracy: using our local database or using an API node?
Is there no way to include additional data with an action as it hits the ledger, such as response{...} or result{...}?

NotInitializedError is thrown by MongoActionReader on startup

In AbstractActionReader.getNextBlock(), this.getLastIrreversibleBlockNumber() is called before the initialized check, which causes a NotInitializedError to be immediately thrown in the MongoActionReader by this.throwIfNotInitialized()

UnhandledPromiseRejectionWarning: StatusCodeError: 500

Hi,

I pulled the code and ran the example. Worked fine the first time and failed afterwards due to unhandled promise.

➜  demux-js git:(develop) yarn example eos-transfers
yarn run v1.1.0
$ ./run-example.sh "eos-transfers"
(node:6496) UnhandledPromiseRejectionWarning: StatusCodeError: 500 - {"code":500,"message":"Internal Service Error","error":{"code":3100002,"name":"unknown_block_exception","what":"Unknown block","details":[{"message":"Could not find block: 8656566","file":"chain_plugin.cpp","line_number":1276,"method":"get_block"}]}}
    at new StatusCodeError (/Users/harshitgupta/Developer/demux-js/node_modules/request-promise-core/lib/errors.js:32:15)
    at Request.plumbing.callback (/Users/harshitgupta/Developer/demux-js/node_modules/request-promise-core/lib/plumbing.js:104:33)
    at Request.RP$callback [as _callback] (/Users/harshitgupta/Developer/demux-js/node_modules/request-promise-core/lib/plumbing.js:46:31)
    at Request.self.callback (/Users/harshitgupta/Developer/demux-js/node_modules/request/request.js:185:22)
    at emitTwo (events.js:126:13)
    at Request.emit (events.js:214:7)
    at Request.<anonymous> (/Users/harshitgupta/Developer/demux-js/node_modules/request/request.js:1157:10)
    at emitOne (events.js:116:13)
    at Request.emit (events.js:211:7)
    at IncomingMessage.<anonymous> (/Users/harshitgupta/Developer/demux-js/node_modules/request/request.js:1079:12)
(node:6496) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:6496) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
✨  Done in 13.65s.

How can I get the data from the table which in the EOS smart contract?

I am trying to develop my DAPP by using demux. Now I need to get the data from my contract. How should demux be implemented?

	public async getHeadBlockNumber(): Promise<number> {
	const blockInfo = await this.httpRequest("get", {
	url: `${this.nodeosEndpoint}/v1/chain/get_info`,
	json: true,
	})
	if (this.onlyIrreversible) {
	return blockInfo.last_irreversible_block_num
	}
	return blockInfo.head_block_num
	}

eosio / demux-js Goto Github PK

demux-js's People

Contributors

Stargazers

Watchers

Forkers

demux-js's Issues

Recommend Projects

Recommend Topics

Recommend Org