equibit / wallet-api Goto Github PK

View Code? Open in Web Editor NEW

0.0 5.0 1.0 1.05 MB

License: MIT License

HTML 2.16% JavaScript 97.79% Makefile 0.02% Dockerfile 0.03%

wallet-api's Introduction

wallet-api

Equibit Wallet API

About

This project uses Feathers. An open source web framework for building modern real-time applications.

Getting Started

Getting up and running is as easy as 1, 2, 3.

Make sure you have NodeJS and npm installed.
Install your dependencies
```
cd path/to/wallet-api; npm install
```
For local development remember to get local-development-local.json or correct bitcoin settings from one of the maintainers.
For local development you also need to have a running MongoDB instance on your local machine; you can run docker-compose up if you're working with Docker.
Start your app
```
npm start
```

Testing

Simply run npm test and all your tests in the test/ directory will be run.

Scaffolding

Feathers has a powerful command line interface. Here are a few things it can do:

$ npm install -g feathers-cli             # Install Feathers CLI

$ feathers generate service               # Generate a new Service
$ feathers generate hook                  # Generate a new Hook
$ feathers generate model                 # Generate a new Model
$ feathers help                           # Show all commands

Developing

To run a local Bitcoin node in the regtest mode:

Download BitcoinCore from https://bitcoin.org/en/download
Run the core with params:

$ /Applications/Bitcoin-Qt.app/Contents/MacOS/Bitcoin-Qt \
  -logtimestamps \
  -server \
  -port=8338 \
  -debug \
  -regtest \
  -rpcuser=user \
  -rpcpassword=password \
  -rpcport=18332 \
  -rpcallowip=127.0.0.1

Help

For more information on all the things you can do with Feathers visit docs.feathersjs.com.

Changelog

0.1.0

Initial release

License

Licensed under the MIT license.

wallet-api's People

Contributors

Watchers

Forkers

ilyavf

wallet-api's Issues

Populate transactions on login

It's possible that the user will have received additional transactions while they were not logged in, so we need to crawl the portfolio addresses to verify this and record any not-yet-recorded transactions and addresses.

scan the gap of 20 #23
Addresses will need to be scanned, first. #20
We can use the listUnspent RPC call to scan the known addresses with unspent money.
We need to compare the list of transactions recorded in the wallet-api with the ones from listunspent. Any listunspent transactions that aren't in the wallet-api database will need to be recorded. The browser will create a transaction with the address, but without a toAddress, also marking it as type: "in" (Only transactions marked as "out" will need to have the toAddress field populated.

The downside with using /listunspent to verify addresses is that any money that was both received and spent outside of our wallet-ui system will not be accounted for. The only way that we can currently see to account for this money would be to utilize the user's xPub and scan the blockchain, which will be an resource-expensive process.

/companies endpoint security profile

@ilyavf what are the security requirements for the /companies endpoint? Which users should be able to access/create/remove records, etc.?

Update Shares Issued - Secondary Market

When some shares were sold to an investor, and he re-sells it back to issuer, they are sent to the original issuance address (not portfolio). So, these shares becomes “un-issued”.
and we need to decrease the number

this isn't finished on the front end but we can make a transactions hook to handle the case

Prevent portfolio-addresses from creating dups

Currently I can see a lot of dups in my addressesMeta in UI, and I checked that DB contains duplicate records (the same index/isChange/type and different _id).

Add before create hook to prevent creating dups. Return existing record instead.
Related UI issue is captured here: Equibit/wallet-ui#732

I am not sure how UI created the following dups but this should be prevented on API anyway.

Blockchain info service

We need a service that will provide blockchain information in real-time and update transactions when they are added to a block.

TODO:

Add a new service /blockchain-info.
Query blockchain for info: block height, ~~transaction ids that are included in a newly mined block.~~ Interval of querying: 10 min.
Update Transaction table with block height value when a transaction is recorded in the blockchain.

endpoint `/getbalance?query=address1|address2|xpub1|xpub2`

Given xPub(s) and/or address(es) return balance per address and total. Similar to blockchain info:

https://blockchain.info/multiaddr?active=xpub67xpozcx8pe95XVuZLHXZeG6XWXHpGq6Qv5cmNfi7cS5mtjJ2tgypeQbBs2UAR6KECeeMVKZBPLrtJunSDMstweyLXhRgPxdp14sk9tJPW9

{
  "wallet": {      <<< summary
    "n_tx": 0,
    "n_tx_filtered": 0,
    "total_received": 0,
    "total_sent": 0,
    "final_balance": 0
  },
  "addresses": [
    {
      "address": "xpub67xpozcx8pe95XVuZLHXZeG6XWXHpGq6Qv5cmNfi7cS5mtjJ2tgypeQbBs2UAR6KECeeMVKZBPLrtJunSDMstweyLXhRgPxdp14sk9tJPW9",
      "n_tx": 0,
      "total_received": 0,
      "total_sent": 0,
      "final_balance": 0,
      "gap_limit": 20,
      "change_index": 0,
      "account_index": 0
    }
  ],
  "txs": []
}

Cannot login with tmp password

After a new signup received a tmp password and tried to login.

POST http://localhost:3030/authentication

{
  "email": "[email protected]",
  "challenge": "6c28316b6cc6a0993c3db66011afb8636563439bfcbff8c831376c8331bd3c2bf5149b94510055e63e262efc7298da36cc194799b4b80aa42c74ccfa1a596d51",
  "signature": "3380499f3377bcdea3eb4509211022679104d76eca4a7f595d14c972f8c6bd62429ac8a3f581c75da8d36ff49f61063943883283e6099488377f594859c96265",
  "strategy": "challenge"
}

Response:
401 Unauthorized

{
  "name": "NotAuthenticated",
  "message": "Key must be a buffer",
  "code": 401,
  "className": "not-authenticated",
  "errors": {}
}

`/subscribe` endpoint for realtime address notifications

For logged-in socket users, the /subscribe endpoint will allow the UI to receive notifications for

We can do DELETE /subscribe addresses to unsubscribe if I can't do it on the server.

Orders service

First draft of settings, needs verification!

what needs to be protected is in the ❌

These would be values set by the server/hooks if no other role is marked as allowed to set it

what needs to be allowed is the ✅

currently most things on most services, to any authenticated person, is a green checkmark

For #63

	Unauthenticated			Authenticated Non-Owner			Authenticated Owner
Full Record	Can Delete ❌			Can Delete ❌			Can Delete ❌
Column	Create	Get / Find	Patch / Update	Create	Get / Find	Patch / Update	Create	Get / Find	Patch / Update
userId	❌	❌	❌	❌	❌	❌	❌	✅	❌
type	❌	✅	❌	❌	✅	❌	✅	✅	❌
assetType	❌	✅	❌	❌	✅	❌	✅	✅	❌
btcAddress	❌	✅	❌	❌	✅	❌	❌	✅	❌
eqbAddress	❌	✅	❌	❌	✅	❌	❌	✅	❌
timelock	❌	✅	❌	❌	✅	❌	✅	✅	❌
portfolioId	❌	✅	❌	❌	✅	❌	❌	✅	❌
quantity	❌	✅	❌	❌	✅	❌	✅	✅	❌
price	❌	✅	❌	❌	✅	❌	✅	✅	❌
isFillOrKill	❌	✅	❌	❌	✅	❌	✅	✅	❌
goodFor	❌	✅	❌	❌	✅	❌	✅	✅	❌
status	❌	✅	❌	❌	✅	❌	❌	✅	✅
issuanceId	❌	✅	❌	❌	✅	❌	✅	✅	❌
issuanceAddress	❌	✅	❌	❌	✅	❌	✅	✅	❌
issuanceName	❌	✅	❌	❌	✅	❌	❌	✅	❌
issuanceType	❌	✅	❌	❌	✅	❌	❌	✅	❌
companyName	❌	✅	❌	❌	✅	❌	❌	✅	❌
createdAt	❌	✅	❌	❌	✅	❌	❌	✅	❌
updatedAt	❌	✅	❌	❌	✅	❌	❌	✅	❌

Crawl the Gap of 20

2 tasks:

Receive an xPub from the client.
Crawl addresses (BTC and Equibit)

updateAddressMeta()


// listUnspentOutputs() // grouped by address
{
  type: 'BTC',
  summary: {
    "total": 2.35
  },
  addressesByIndex: {
    1: "mpS2RuNkAEALvMhksCa6fPpLVb5yCPanLu"
  },
  addresses: {
    "mpS2RuNkAEALvMhksCa6fPpLVb5yCPanLu": {
      "address": "mpS2RuNkAEALvMhksCa6fPpLVb5yCPanLu",
      "index": 1,
      "isChange": false,
      "amount": 2.35,
      "txouts": [
        {
          "txid": "2c3c1de1357c6155bfffab5ce6bbd1807d5db30db09ebc96bbb9a55031626e1a",
          "vout": 0,
          "address": "mpS2RuNkAEALvMhksCa6fPpLVb5yCPanLu",
          "account": "",
          "scriptPubKey": "76a91461ca8116d03694952a3ad252d53c695da7d95f6188ac",
          "amount": 1,
          "confirmations": 105,
          "spendable": false,
          "solvable": false
        },
        {
          "txid": "8455e6ac64fcf4142bc73d3f0236067319c9483bd15de89ce7c3aa2d0371db60",
          "vout": 1,
          "address": "mpS2RuNkAEALvMhksCa6fPpLVb5yCPanLu",
          "account": "",
          "scriptPubKey": "76a91461ca8116d03694952a3ad252d53c695da7d95f6188ac",
          "amount": 1.35,
          "confirmations": 105,
          "spendable": false,
          "solvable": false
        }
      ]
    }
  }
}


// To generate xpub (for testing).  The browser will do this, normally.
const mnemonic = bip39.generateMnemonic();
const seed = bip39.mnemonicToSeed(mnemonic, '') // 2nd param is empty string
const hdnode = bitcoin.HDNode.fromSeedBuffer(seed, bitcoin.networks.testnet)
// Bitcoin xPub
const btc = hdnode.derivePath("m/44'/0'/0'").neutered().toBase58() // type string
// Equibit xPub
const eqb = hdnode.derivePath("m/44'/72'/0'").neutered().toBase58() // type string



// portfolioBalance :: portfolio xpub -> portfolio index -> listunspent + updateAddresses
/portfolio-balance/<portfolioIndex>?btc=111&eqb=222
const portfolioBalance.get = function (params) {
  const portfolioIndex = hook.id
  // Validate that we have a portfolioIndex
  // `m / purpose' / coin_type' / account' ` => generate `/ change`
  // Generate the change xpub
  //   https://github.com/bitcoinjs/bitcoinjs-lib/blob/master/src/hdnode.js
  const hdNode = bitcoin.HDNode.fromBase58(xPubBtc, bitcoin.networks.testnet)
  const xpubBtcExternal = hdNode.derive(0).toBase58() // string
  const xpubBtcChange = hdNode.derive(1).toBase58() // string

  const hdNodeEqb = bitcoin.HDNode.fromBase58(xPubEqb, bitcoin.networks.testnet)
  const xpubEqbExternal = hdNodeEqb.derive(0).toBase58() // string
  const xpubEqbChange = hdNodeEqb.derive(1).toBase58() // string

  Promise.all(
    app.service('crawl-xpub').create({type: 'btc', xpub: xpubBtcExternal)
    app.service('crawl-xpub').create({type: 'btc', xpub: xpubBtcChange)
    app.service('crawl-xpub').create({type: 'eqb', xpub: xpubEqbExternal)
    app.service('crawl-xpub').create({type: 'eqb', xpub: xpubEqbChange)
  ).then(responses => {
    // Aggregate into from two BTC crawls into one object.  Merge addresses into one object.
    // and add up the summary amounts.

    // Update addressMeta records

    // Send response
  })

  updateAddressMeta()
})


// xpubCrawl :: xpub -> blockchainServerType -> listunspent
// xpub format:
//   `m / purpose' / coin_type' / account' / change ` => generate 20 `address_index`
xpubCrawl (xpub, blockchainServerType) {
  // Step 1: generate 20 addresses

  // Step 2: use `listunspent` with the 20 addresses & see which ones have funds, repeat with remaining addresses until we have 20 empty.
  listUnspent()

  // Step 3: format the data as noted, above, and return it in the response.
}

...

Collect transactions related to the xPub addresses.
- Add to our Transactions table.
- A Transaction should have a type of:
  - from my address (SENT)
  - to my address (from outside) (RECEIVED)
  - from my address to my address (TRANSFERRED)
2). Collect a list of unspent outputs (aka /listunspent).

№ 1 - Collect Transactions

/listtransactions gives a list of transactions of our wallet (watch-only addresses that were imported for users of our wallet)
- user can generate an address and receive funds outside of our wallet. To capture this kind of addresses we need to /importaddress before calling /listtransactions.

GET /portfolios?btc[0]=111&eqb[0]=222&btc[1]=333&eqb[1]=444

m / purpose' / coin_type' / account' / change / address_index

Option 1:

use user's Portfolio.addresses indexes to derive known (registered in database) addresses
plus generate 20 more (the Gap)
applies to both external and change addresses

Option 2:

just generate the Gap of 20 (external and change)

Example:

having xpubs of Portfolio: m / purpose' / 0' / index=0' and m / purpose' / 73' / 0'
having addresses:

{
  btc: [{ 
    index: 0, 
    isChange: false
  }, {
    index: 0,
    isChange: true 
  }], 
  eqb: [{ 
    index: 0, 
    isChange: false
  }, {
    index: 0,
    isChange: true 
  }]
}

Missing API key for bitcoin-average service

API key is missing for the service. Needs to be added to ENV vars on QA.

  "bitcoinAverage": {
    "key": "BITCOIN_AVERAGE_KEY",
    "secret": "BITCOIN_AVERAGE_SECRET"
  }

Network message:

UI error in console:

UPDATE:
ticker with a particular symbol is the only command we’re using right now, and that’s an anonymous endpoint on bitcoin-average.
To fix the issue:

check if the API key is available, and if its NOT then remove headers from the axios request to bitcoin-average and use it anonymously

On bad email user still gets created.

Error on Signup:

"You tried to send to a recipient that has been marked as inactive.
Found inactive addresses: [email protected].
Inactive recipients are ones that have generated a hard bounce or a spam complaint. "

UI showed an error, but user still got created in DB.

Real time updates

Figure out a strategy for real time updates.

E.g. currently everybody are getting notified about a new Order. Event orders created. Even the creator of the offer receives it.

Then transactions are broadcasted to everybody (should only be limited to transaction owner).

Offers are also broadcasted to everybody. Should probably be limited to the user who owns the issuance.

Here is a list of all services and their current real-time events status. Let's check the boxes once we've reviewed and implemented.

Protect services patch when no id is passed

E.g. when UI patches a portfolio-address and provides no id, then all items are patched:

Address Duplication problem

creating an issue to track progress
The fix is already done:
#87

am nearly finished with the migration so we can clean up the portfolio-addresses before applying the fix

Offers Services

WIP, offers service

what needs to be protected is in the ❌

These would be values set by the server/hooks if no other role is marked as allowed to set it

what needs to be allowed is the ✅

currently most things on most services, to any authenticated person, is a green checkmark

we'll probably need an icon for custom rule too

like status is technically only patchable by the Owner if the patch value is 'CANCELLED' and current status is not 'CLOSED'

For #63

	Unauthenticated			Authenticated Non-Owner			Authenticated Owner
Full Record	Can Delete ❌			Can Delete ❌			Can Delete ❌
Column	Create	Get Find	Patch Update	Create	Get Find	Patch Update	Create	Get Find	Patch Update
userId	❌	❌	❌	❌	❌	❌	❌	✅	❌
orderId	❌	✅	❌	❌	✅	❌	✅	✅	❌
type	❌	✅	❌	❌	✅	❌	✅	✅	❌
assetType	❌	✅	❌	❌	✅	❌	✅	✅	❌
quantity	❌	✅	❌	❌	✅	❌	✅	✅	❌
price	❌	✅	❌	❌	✅	❌	❌	✅	❌
status	❌	✅	❌	❌	✅	❌	❌	✅	✅
isAccepted	❌	✅	❌	❌	✅	✅	❌	✅	❌
issuanceId	❌	✅	❌	❌	✅	❌	❌	✅	❌
issuanceAddress	❌	✅	❌	❌	✅	❌	❌	✅	❌
companyName	❌	✅	❌	❌	✅	❌	❌	✅	❌
issuanceName	❌	✅	❌	❌	✅	❌	❌	✅	❌
issuanceType	❌	✅	❌	❌	✅	❌	❌	✅	❌
secretEncrypted	❌	✅	❌	❌	✅	❌	❌	✅	❌
secret	❌	✅	❌	❌	✅	❌	❌	✅	✅
hashlock	❌	✅	❌	❌	✅	❌	❌	✅	❌
timelock	❌	✅	❌	❌	✅	❌	❌	✅	❌
timelock2	❌	✅	❌	❌	✅	❌	❌	✅	❌
timelockExpiresBlockheight	❌	✅	❌	❌	✅	❌	❌	✅	❌
timelock2ExpiresBlockheight	❌	✅	❌	❌	✅	❌	❌	✅	❌
timelockExpiredAt	❌	✅	❌	❌	✅	❌	❌	✅	❌
timelock2ExpiredAt	❌	✅	❌	❌	✅	❌	❌	✅	❌
htlcStep	❌	✅	❌	❌	✅	❌	❌	✅	❌
htlcTxId1	❌	✅	❌	❌	✅	❌	❌	✅	❌
htlcTxId2	❌	✅	❌	❌	✅	❌	❌	✅	❌
htlcTxId3	❌	✅	❌	❌	✅	❌	❌	✅	❌
htlcTxId4	❌	✅	❌	❌	✅	❌	❌	✅	❌
btcAddress	❌	✅	❌	❌	✅	❌	✅	✅	❌
eqbAddress	❌	✅	❌	❌	✅	❌	✅	✅	❌
description	❌	✅	❌	❌	✅	❌	✅	✅	✅
createdAt	❌	✅	❌	❌	✅	❌	❌	✅	❌
updatedAt	❌	✅	❌	❌	✅	❌	❌	✅	❌

Update /user service with portfolio summary data

When user is logged in and opens My Portfolio menu (page-portfolio) he should see the following summary data about his portfolios:

Implementing Portfolios - Receive funds

UI receives portfolios on login
Portfolio will have an addresses key that stores an array of the portfolio addresses

// portfolio.addresses
[
  {index: 0, type: "eqb", used: true},
  {index: 0, type: "btc", used: false}
]

UI user clicks receive funds.
UI pulls the last unused address index and displays it to the user.
- If an address is not available, a new one is created from the highest index.
- UI makes the import-address call: #14

A. A transaction comes through Wallet Server.

B. A transaction comes through blockchain.

poll for new listunspent items => addresses -> user?
User logins
User gets all his generated addresses
Sends them to server for listunspent

Security audit

Make a full security audit to point all security and privacy issues.

E.g.:

Not being able to link users to transactions having DB access.
...

Related: PR #82

/transaction-info service

To get transaction info by transaction id:

gettransaction( txid ) to get hex
decoderawtransaction( hex ) to get full transaction info (the same structure as how transaction was created).

GetTransaction:

http://localhost:3030/proxycore?method=gettransaction&params[]=0201cfdb3f51fe23839f4fdff617e0a52ac570a857a5a7c8b88171114b6e87f1&params[]=true

{
  "result": {
    ...,
    "hex": "01000000018cc11051fe827653720140dad355fa7264706d91bf2dd72bdc00bab26941f99f000000006a4730440220512c181b46a9aebdaaa33ae66abbb2f07faea52b1466e3f753ba35ef7af5b38402206fd196a392111f66be6f987ed873e0fa4e22b5ad7d754122bfed864b558ad11a0121032359f23a3324eb22071297ba76c3268cc02aeb95659b7dd70549d96c45e399aaffffffff0280f0fa02000000001976a914c17fb88fe5d903538ecbfc80c2862247e0d9f96088acf8760927010000001976a914bf4a5bf55834290caebb9aac899f4861c5c5523688ac00000000"
  }
}

DecodeRawTransaction:

http://localhost:3030/proxycore?method=decoderawtransaction&params[]=01000000018cc11051fe827653720140dad355fa7264706d91bf2dd72bdc00bab26941f99f000000006a4730440220512c181b46a9aebdaaa33ae66abbb2f07faea52b1466e3f753ba35ef7af5b38402206fd196a392111f66be6f987ed873e0fa4e22b5ad7d754122bfed864b558ad11a0121032359f23a3324eb22071297ba76c3268cc02aeb95659b7dd70549d96c45e399aaffffffff0280f0fa02000000001976a914c17fb88fe5d903538ecbfc80c2862247e0d9f96088acf8760927010000001976a914bf4a5bf55834290caebb9aac899f4861c5c5523688ac00000000

{
  "result": {
    "txid": "0201cfdb3f51fe23839f4fdff617e0a52ac570a857a5a7c8b88171114b6e87f1",
    ...
    "vin": [
      {
        "txid": "9ff94169b2ba00dc2bd72dbf916d706472fa55d3da400172537682fe5110c18c",
        "vout": 0,
        ...
      }
    ],
    "vout": [
      {
        "value": 0.5,
        "n": 0,
        "scriptPubKey": {
          ...,
          "addresses": [
            "myA5iHSDcpEHK2uSUx4P74HMvbEgBtYKj4"
          ]
        }
      },
      {
        "value": 49.49899,
        "n": 1,
        "scriptPubKey": {
          "addresses": [
            "mxxQSGAai9NCV5An9fnXxp3L3CEsNc8dmh"
          ]
        }
      }
    ]
  }
}

Refactor the transactions model for clearer types/addressing

As currently modeled, the Transactions model in Mongo has some limitations affecting our ability to deliver a robust feature set for all known use cases.

the need for independent description fields between the sending party and receiving party led to making duplicate transactions, one for each.
insufficient transaction type enumerations caused a collision between cancelling trades and cancelling authorizations
Labeling all HTLC transactions as SELL or BUY is confusing.
External mutability is a data security risk.

The following changes to the transactions model are recommended to alleviate these issues and allow swift development of outstanding feature requests:

Make transactions externally immutable; a server process will still be able to set confirmation block height.
Remove description as a field from the transactions model, and move the descriptions for a transaction to a separate transaction-notes service, which will allow each participant in a transaction their own notes on the issue (this will eventually be mutable via signed patch requests, but for now, immutable is acceptable)
Remove the address field and change find queries on the transactions table to use fromAddress or toAddress.
Change the enumeration for transaction types to:
- TRANSFER -- regular, non-trade transfers to a user's own wallet or another wallet
- AUTH -- authorization of shares (Blank EQB in, marked EQB out)
- CANCEL -- deauthorization of shares (marked EQB in, blank EQB out)
- TRADE -- a happy path step in HTLC swaps -- works alongside the htlcStep field: step 1&2 lock assets; step 3&4 collect
- REFUND -- a fallback path in HTLC swaps -- step 4 recovers assets locked in TRADE step 1, step 3 recovers assets locked in TRADE step 2
deduplicate transactions (remove all "receiver" transactions)

New transactions created by a connected socket should automatically subscribe to that socket.

Transaction Hooks to update orders and offers

We do this from the front end currently:

the backend should take care of this instead.

Steps we need to complete:

detect this type of transaction (Transactions are final actions that hit the blockchain and can be many different things) on the backend (so we can trigger the hook only when appropriate)
update the related order the same way the front end does it now
update the related offer in the same way front does now
remove the save() calls from the front

NOTE:
In the future we would remove the front end code that updates the order and offer locally too because when we update order and offers on the back, it should broadcast those updates to the user via their respective filters automatically.

User Accounts - API

This is a ZenHub epic for tracking the moving parts of development of features related to User Accounts.

/portfolio crawling

/portfolio?

keys[]=0.eqb.tpubDCdJjskBsAyLUmiAS66VS4HXJx2e7MYATG9bjECVJVpK55hnFnaNCLY98CHygFu64L2pWAUrPeWja5LmeGPUfhnTqjuTmymVLw8NZUvfsm1
&
keys[]=1.btc.tpubDCdJjskBsAyLUmiAS66VS4HXJx2e7MYATG9bjECVJVpK55hnFnaNCLY98CHygFu64L2pWAUrPeWja5LmeGPUfhnTqjuTmymVLw8NZUvfsm1

Add Offer patch validation for updating order related info

For HTLC2 and HTLC3 actions of a trade that are performed by the user who does not own the Offer.

But this user needs to update the following fields of the Offer:

htlcTxId2
htlcTxId4
orderTimeLock (may be this should be just stored on Transaction?)
htlcStep

Todo:

The Offer update/patch hook should check if a user who tries to patch an offer owns the corresponding order (offer.orderId).

Remove tempPassword from user after password change.

Remove the tempPassword from the user after they successfully change password. Otherwise, the temporary password could be used to change the password a second time.

Integrate database migrations into the build process

We have had several instances during development where the mongo store had to be modified by hand due to buggy behavior or not meeting assumed invariants, and it can prevent proper operation of the API application up to and including failure to launch. While we've so far been hand-rolling solutions as problems arise, it is important to keep an easily accessible, easily replayable history of our migrations.

I recommend:

choosing a mongo migration engine for Node (there are several out there, pick whatever seems easiest)
storing migrations in a separate directory in the API source repo.
updating migrations as part of the deploy process for QA/UAT/prod
ensuring all devs and DBAs for the project can write Mongo automation scripts.

Add Helmet

https://github.com/helmetjs/helmet

mostly for this: https://helmetjs.github.io/docs/xss-filter/

Need a Forgot password endpoint

Need an endpoint for forgot password.

Currently in UI there is a static method on user model that just resolves.

Needs to also record

User agent data
IP Address

`/import-address` endpoint

Create portfolio address

Server creates a portfolio record & returns the next index - based on portfolio records in the wallet-api database. Return the new portfolio record to the UI.
UI derives portfolio keys using portfolio index m /44' /0' /portfolioIndex'
UI derives address key from portfolio keys m /44' /0' /portfolioIndex' /0 /addressIndex and generates a bitcoin address.
UI sends the address to the /import-address endpoint
Server the /import-address service makes a hash of the address and checks if the address hash is already stored.
- If it was stored, it returns a success response without making an RPC call.
- If it was not stored, it makes the RPC call to importAddress before returning the success response
UI User clicks done.

Upgrade to Feathers 3

https://www.npmjs.com/package/feathers

Migration doesn't look tooo bad:
https://docs.feathersjs.com/migrating.html

but it'll take some time

Cleanup tests to not clear records in DB

When I run tests locally my db gets updated: real records that i use for development are gettgin removed, and some test records are created and stay there. I though I fixed it but i must have missed some tests.

Service /transactions

We want to store a transaction when:

user sends funds/equity;
user receives funds/equity.

Cases to cover:

Amount is received.
Amount is sent from one address (txout).
Amount is sent from multiple addresses, when none of the existing addresses has enough amount in it.

Browser builds a transaction hex to be sent to the Core and can add any meta data to be stored in DB.

Data to store:

blockchain address (to identify who the transaction belongs to):
- for sending: the address the funds were sent from;
- for receiving: the address the funds were received for;
- this address will identify a Portfolio in browser.
type: btc or eqb;
for EQB:
- company name / id;
- issuance name / id.
transaction ID;
amount;
description (?);
timestamp.

HTLC: update offer's htlcStep when creating receiver transaction

For HTLC we have 4 transactions: 2 made by a seller and 2 made by a buyer.

Whenever one party sends a transaction (e.g. for an order) we need to update the corresponding party (e.g. offer) with the new htlcStep value.

Move `addressesMeta` into its own collection from `portfolios`

This will allow to patch individual items of addressesMeta (e.g. isUsed), or adding a new item. Otherwise browser will have to send the whole array of addressesMeta (hundreds and thousands of rows) with a single change.

A new collection portfolio-addresses should be linked to portfolios by portfolio's _id.

The structure of the model should look like this:
{index: 0, type: 'EQB', isChange: false, isUsed: true}

_id: ObjectID
portfolioId: ObjectID
index: Number
type: enum(['EQB', 'BTC'])
isChange: Boolean
isUsed: Boolean, default: false

The logical key of an item is portfolioId + type + isChange + index.

Recycle addressMaps

addressMap objects should always be upserted by address to prevent duplicate addresses with differing identifiers.

Create simple Import Multi service

for use with Equibit/wallet-ui#693

3 reasons:

We want to steer away from using or exposing proxycore service on the front so may as well start here.
Proxycore uses url params (GET) which is limiting, not as secure as POST, and then turns it into POST for the actual call to core anyway.
The call into core for this is fairly bulky but it's a lot of repeat data in the request. Front end will just need to send a list of base58 addresses, this service will decorate them for the core call and do what's needed.

Listunspent service is similar - uses core call under the hood but is easier to use than proxycore for the same service.

Create /watchlist service

User should be able to maintain a list of companies he wants to watch for.

Audit and updating of old services

A lot of the old services need a careful audit and are in an unprotected state.

The most important questions:

For EACH field on each service model: Who can update/patch them?
Who, if anybody, can delete the record?
Who can get/find the record? (a lot of these seems like it's okay for anyone to view but I am not sure if I'm right about that)

For example, nobody should be able to change address of a transaction. (anybody can though)
And right now anyone can query all offers and update them all to 'CANCELLED' status without any trouble.

Making these changes should include tests too.

I don't know all the ins and outs of the app yet so I'd just need to have a call with Ilya for one service at a time, ask all of the 3 questions above, then protect 'em.

All the individual issues are visible below because they reference this one.
This is the list in any case: Security securing data and fields based on role and model/property

The other services that I didn't open issues for seem fine to me as-is. I am not a security expert though!

Authenticated user data

On a successful login UI needs the following info (received on the last step of the login process) apart from JWT:

user data including:
- keys (encoded private key / pair of keys - depending on what we decide to save from UI);
- usingTmpPassword {boolean};
- user.isNew {boolean}.

Bug: when HTLC 2 transaction was sent, the corresponding receiver transaction was not created.

transactions::create

{
address: "mnG96nXVHxzS9mf19FwpivCYGeNuK6PXy1"
addressTxid: "fce15801bdb7b86c9c309b04563f09d51514b15c557b5111769085f0947b1db3"
addressVout: 0
amount: 500
companyName: "Ilya company"
companySlug: "ilya-company"
currencyType: "EQB"
description: "Selling securities (HTLC #2)"
fee: 1000
fromAddress: "mnG96nXVHxzS9mf19FwpivCYGeNuK6PXy1"
hashlock: "9bdc9100691e4eae8a7674ac7d79bb7358666ac0ee28d912a117aaab0e003497"
hex: "0200000002b31d7b94f085907611517b555cb11415d5093f56049b309c6cb8b7bd0158e1fc000000006b483045022100cbfb6ac36e2d69f6cfa421cb4f152eae934eb52033ffcb4b55de0f4252de49b9022060c33f3f1e7cd6f15f073eb54537365716f4f5b3bcde7e17f4c48bce513be4850121030eac2c2144d73bc037b07fdfb8b400a6bb3219d82af8d94b7742365f7acbf16000000000b31d7b94f085907611517b555cb11415d5093f56049b309c6cb8b7bd0158e1fc010000006b483045022100d8139384689bb1df6ae3b6be4cc8277bbd94749bb5c7aee1d124b72a112da71102201e9304c85e8e0ed79f0590101fc809d36bf9368b5ac3cd13d865bd66f0f198470121021ef7f4bf77c3c2c0d6759f36524e6a1f4ffebd266031aa3a6c74af54cc69337b0000000003f4010000000000005963a8209bdc9100691e4eae8a7674ac7d79bb7358666ac0ee28d912a117aaab0e0034978876a9140d7a8829a21761713eab10e9bfbdd46baf8409f6675ab17576a914ac14222446e8e23432d4a7eacd170cd27254f7f46888ac00000000000000000000000000000000000000000000000000000000000000000000000000000cdff505000000001976a914ac14222446e8e23432d4a7eacd170cd27254f7f488ac0000000000000000000000000000000000000000000000000000000000000000000000000000ad164008000000001976a9144bf8a4a010f0facf609528e12a1c8392a460d6f988ac000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
htlcStep: 2
issuanceId: "5a39876765aa4c068e9ad79f"
issuanceName: "Ilya Series One"
issuanceType: "common_shares"
timelock: 10
toAddress: "mgkDsDGy6ynkgQFUEpxD9m53tbXADYVPDC"
txId: "27b663db4ad204d7edae5059b8ffba9f4a1d38c97402ea3ebd163ce10fe9bbeb"
type :"SELL"
}

Page User Preferences: 2nd factor email

Add a service for 2nd factor email when user wants to backup his recovery phrase:

send 2nd factor email;
verify the code;
verify the code again when we request the mnemonic.

Important: Right now we send the encrypted mnemonic to browser with user data. But looks like we need to send it only when user enters 2nd factor using this UI. For this: