Giter VIP home page Giter VIP logo

go-archiver's Introduction

The qubic archiver service

The archiver service's purpose is to store and make available data regardless of the current epoch.

High level description:

The archive system consists of two services:

  • qubic-archiver - the archiver processor and HTTP server that provides rpc endpoints to query the archiver
  • qubic-nodes - a service responsible with providing information regarding reliable nodes and the max tick of the network

IMPORTANT

Before starting the system, open the docker-compose.yml file and make sure that you have a reliable peer list setup for the qubic-nodes service. This can be configured using the QUBIC_NODES_QUBIC_PEER_LIST environment variable.

Other optional configuration parameters for qubic-archiver can be specified as env variable by adding them to docker compose:

  $QUBIC_ARCHIVER_SERVER_READ_TIMEOUT                        <duration>  (default: 5s)
  $QUBIC_ARCHIVER_SERVER_WRITE_TIMEOUT                       <duration>  (default: 5s)
  $QUBIC_ARCHIVER_SERVER_SHUTDOWN_TIMEOUT                    <duration>  (default: 5s)
  $QUBIC_ARCHIVER_SERVER_HTTP_HOST                           <string>    (default: 0.0.0.0:8000)
  $QUBIC_ARCHIVER_SERVER_GRPC_HOST                           <string>    (default: 0.0.0.0:8001)
  $QUBIC_ARCHIVER_SERVER_NODE_SYNC_THRESHOLD                 <int>       (default: 3)
  $QUBIC_ARCHIVER_SERVER_CHAIN_TICK_FETCH_URL                <string>    (default: http://127.0.0.1:8080/max-tick)
  
  $QUBIC_ARCHIVER_POOL_NODE_FETCHER_URL                      <string>    (default: http://127.0.0.1:8080/status)
  $QUBIC_ARCHIVER_POOL_NODE_FETCHER_TIMEOUT                  <duration>  (default: 2s)
  $QUBIC_ARCHIVER_POOL_INITIAL_CAP                           <int>       (default: 5)
  $QUBIC_ARCHIVER_POOL_MAX_IDLE                              <int>       (default: 20)
  $QUBIC_ARCHIVER_POOL_MAX_CAP                               <int>       (default: 30)
  $QUBIC_ARCHIVER_POOL_IDLE_TIMEOUT                          <duration>  (default: 15s)
  
  $QUBIC_ARCHIVER_QUBIC_NODE_PORT                            <string>    (default: 21841)
  $QUBIC_ARCHIVER_QUBIC_STORAGE_FOLDER                       <string>    (default: store)
  $QUBIC_ARCHIVER_QUBIC_PROCESS_TICK_TIMEOUT                 <duration>  (default: 5s)

Run with docker-compose:

$ docker-compose up -d

Available endpoints:

Instance information

/status

Provides information regarding the status of the archiver instance.

curl http://127.0.0.1:8001/status
{
  "lastProcessedTick":{
    "tickNumber":13683006,
    "epoch":107
  },
  "lastProcessedTicksPerEpoch":{
    "106":13548510,
    "107":13683006
  },
  "skippedTicks":[
    {
      "startTick":1,
      "endTick":13547629
    },
    {
      "startTick":13548511,
      "endTick":13679999
    }
  ],
  "processedTickIntervalsPerEpoch":[
    {
      "epoch":106,
      "intervals":[
        {
          "initialProcessedTick":13547630,
          "lastProcessedTick":13548510
        }
      ]
    },
    {
      "epoch":107,
      "intervals":[
        {
          "initialProcessedTick":13680000,
          "lastProcessedTick":13683006
        }
      ]
    }
  ]
}

/healthcheck

Mainly used by the load-balancer to decide if the instance should be added to the balancing rotation based on if it's up-to-date with the network or not.

curl http://127.0.0.1:8001/healthcheck
{
  "code":13,
  "message":"processor is behind node by 7684 ticks",
  "details":[]
}

/latestTick

Returns the number of the latest tick processed by the archiver instance.

curl http://127.0.0.1:8001/latestTick
{
  "latestTick":13690806
}

Tick related endpoints

/ticks/{tick_number}/tick-data

Returns the tick information for the given tick.

curl http://127.0.0.1:8001/ticks/13683397/tick-data
{
  "tickData":{
    "computorIndex":481,
    "epoch":107,
    "tickNumber":13683397,
    "timestamp":"1714593920000",
    "varStruct":"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
    "timeLock":"1BAUdVy8iM0d9LYFK2/WACnR+Fvn7cPY3sLgnt2W/Es=",
    "transactionIds":[
      "xgniuxigsnbeifvkithkcgnvxhmglgkppscwupescgwoqljxdecekhueutfn",
      "ilhefvlxgzpzmdfiixupcusbgyihxxqfomdvumwfngkuugagyutjudhfhobd",
      "jvpovfnhpiijehctdtsivngewabbzwxinjijwmeiihuebraaheokpzdayhsb"
    ],
    "contractFees":[],
    "signatureHex":"74b77a2f61b363bc0f2ebeda39b5255c8b5ffb767f571a32603ace8b4f47c23f531018bb4efac3f5a9c3f7896c08d04e4149319a666df25d0bed6c6cfbed0000"
  }
}

/ticks/{tick_number}/quorum-tick-data

Returns the quorum tick information for the given tick.

curl http://127.0.0.1:8001/ticks/13683397/quorum-tick-data
{
  "quorumTickData":{
    "quorumTickStructure":{
      "epoch":107,
      "tickNumber":13683397,
      "timestamp":"1714593914000",
      "prevResourceTestingDigestHex":"42359ba71f11311f",
      "prevSpectrumDigestHex":"f635766cbfe80a021e18d03a87fc8304e7f8cdd25e56f8147e2b97e21ea38161",
      "prevUniverseDigestHex":"52da6fe4a4da50a34582f11b53094ebdfa25087614a5dc6e5e15f0f418804102",
      "prevComputerDigestHex":"9ea25c5c9fa90e55d68cca1d3c78183d5b91bbac82c96de5400e267a16223bd2",
      "txDigestHex":"09ab681628b3e0263c444c2edbce51adb22c0e38e81cbfedd7c08abab2d6c67f"
    },
    "quorumDiffPerComputor":{
      "0":{
        "saltedResourceTestingDigestHex":"0ee4483f607b4865",
        "saltedSpectrumDigestHex":"41d5452637b88cba3b9b44c320ff958f26f7898864752c82ee42ebcacedb6e40",
        "saltedUniverseDigestHex":"751da9c5e2d9e9a74dae2e6552ffaad249ac0eae728a0c940d1cdb3d3e1c8fcd",
        "saltedComputerDigestHex":"755957976e92e0bf65e5537ec4398867f1afabfc777ab4f888e3ccecd6ff787a",
        "expectedNextTickTxDigestHex":"b0f4b3415740c927d351c528355ab13bd2d0640395b5c0eae380e7b93ca5872d",
        "signatureHex":"36ac100871d93c0719370fc75f034e72780f7c4fc21605fe98ec69229a7df2d1fdc1f65aa3bcc74c927bc9296400914bc8909677b2f1e26643764718fcc40100"
      },
      "1":{
        "saltedResourceTestingDigestHex":"d0bfef154b6bbc28",
        "saltedSpectrumDigestHex":"43d1c210e61c3481df7bc65a5456d089be0932ddabac0690ba5a4df918561ef0",
        "saltedUniverseDigestHex":"86bff2d98c8a04517cfe92d662f2185640a3af0a6e125b7e9353119f57b96e3a",
        "saltedComputerDigestHex":"46d7e8504caa635789a8c014e2270db92e8f996851637eca11ddf7f533e76b21",
        "expectedNextTickTxDigestHex":"b0f4b3415740c927d351c528355ab13bd2d0640395b5c0eae380e7b93ca5872d",
        "signatureHex":"c47b58fc51247bb4d223d922e589771956103280c82ccb0927da464bdd0861791d788109c0bdee63dc94553215fc21d5db994eb1e4c1ec268a8d7663fd8d2500"
      },
//      ...
      "675":{
        "saltedResourceTestingDigestHex":"fd730f13a8973bd3",
        "saltedSpectrumDigestHex":"ae20699498a6231169d936fd749a5f8ce5c7608ee8bb019824ff984c7e7f0321",
        "saltedUniverseDigestHex":"9cb63a800d5c8bce5933c82c322205db7a6be9d0389512193b33166efcb0ac61",
        "saltedComputerDigestHex":"816131300e1e13e7e9bf9fc5831653764ea9fb2a0bd7e01570db2e049a3f2caa",
        "expectedNextTickTxDigestHex":"b0f4b3415740c927d351c528355ab13bd2d0640395b5c0eae380e7b93ca5872d",
        "signatureHex":"c4f4ac87c1edf351e60bb545d15a457d7d803646f1c324d9c2429213c326ace736181b1135f718b97a52649a9c6abb43b3774cb0049bf2f959d5cfa8e08c0300"
      }
    }
  }
}

/ticks/{tick_number}/transactions

Returns the full list of transactions for the given tick.

Note that this will include ALL transactions, approved or not.

curl http://127.0.0.1:8001/ticks/13683397/transactions
{
  "transactions":[
    {
      "sourceId":"IAHIRNYARPTESFWWKHWBIICNUEYCIGFMXOGOOBNNBAKSEEGFDWUHPNDHJQUK",
      "destId":"AFZPUAIYVPNUYGJRQVLUKOPPVLHAZQTGLYAAUUNBXFTVTAMSBKQBLEIEPCVJ",
      "amount":"0",
      "tickNumber":13683397,
      "inputType":0,
      "inputSize":32,
      "inputHex":"716c692d637564616b008dd1814005e586d28cbf0687c5d380817ee3f5494304",
      "signatureHex":"814d9e9ccb01766b0885d0592e6c39054c9311d13971be51e3a798c9f3ec080c6f280e4b6519042ee4a9be0c8d9b23127c9ee788282e7578cc8d52dd351c1d00",
      "txId":"xgniuxigsnbeifvkithkcgnvxhmglgkppscwupescgwoqljxdecekhueutfn"
    },
    {
      "sourceId":"UHDZJOGELURMKBEQUSMNBMWAVPHDADXMZPCOLTFAUGQZVELHGUYRCPADDBEF",
      "destId":"AFZPUAIYVPNUYGJRQVLUKOPPVLHAZQTGLYAAUUNBXFTVTAMSBKQBLEIEPCVJ",
      "amount":"0",
      "tickNumber":13683397,
      "inputType":0,
      "inputSize":32,
      "inputHex":"716c692d637564616b0197b23003bc90c9b26688c3329109deba9654162b5c12",
      "signatureHex":"8b165c45532b2a2ef52da2c7bb78030ceacb4154c107297dc6f0f2ccb9da5dd2b602b8578ba391a916c1fefbcb31115b6345456fac868a188cfdeb5530b21100",
      "txId":"ilhefvlxgzpzmdfiixupcusbgyihxxqfomdvumwfngkuugagyutjudhfhobd"
    },
    {
      "sourceId":"MYVLWZGGVHALIEVZQXCEVQJXNEUAYULLEMBMFTNORGJGESCMQTPCVOHBBBTI",
      "destId":"AFZPUAIYVPNUYGJRQVLUKOPPVLHAZQTGLYAAUUNBXFTVTAMSBKQBLEIEPCVJ",
      "amount":"0",
      "tickNumber":13683397,
      "inputType":0,
      "inputSize":32,
      "inputHex":"716c692d637564616b01c8af176f1f3b2ec5b8e53af9215b86d139dbc835003d",
      "signatureHex":"7bea63bcf5fe31c670ec1feee93c943c5e8c4f12fc58f69cfe50b59f0a92dd3347b9ab2797b8184441a4be885ac4579a1000fdf429b0a6dd540bf0cff12a0300",
      "txId":"jvpovfnhpiijehctdtsivngewabbzwxinjijwmeiihuebraaheokpzdayhsb"
    }
  ]
}

/ticks/{tick_number}/transfer-transactions

Returns the list of transfer transactions for the given tick.

Note that there is a difference between transfer transactions and mining transactions.

curl http://127.0.0.1:8001/ticks/13686173/transfer-transactions
{
  "transactions": [
    {
      "sourceId": "FNXHQOKFGKMZUGWQHLTNOPMIGXQAQZWYUPAGXMGAKAASHCVNGUPHEMJHUSOK",
      "destId": "IZTNWDKXSFULQADTOLTMLUPHSCFCXLOJMQOUHPBSRGQZMMXZCJYQFTRDOGRE",
      "amount": "45832157",
      "tickNumber": 13686173,
      "inputType": 0,
      "inputSize": 0,
      "inputHex": "",
      "signatureHex": "8b897f20911d4df01c9faa56782760173f95e1da6f22bbbbae13a519904356b25db5eb736a27056d34f29d09b8fc847142703c0deacedb143b759acd42be0b00",
      "txId": "whhhorfprtqayfygoqwoyqbpnajdfwocnyunwwvjqgykcqpnmphrbozdxscj"
    }
  ]
}

/ticks/{tick_number}/approved-transactions

Returns the list of all approved transactions for the given tick.

curl http://127.0.0.1:8001/ticks/13686387/approved-transactions
{
  "approvedTransactions": [
    {
      "sourceId": "ARALPBGBRNORYBDFRWKQSLENOELBMFJWOFKBRQJNXDXTRZPYGGFKSADAXJON",
      "destId": "NLRQDYJUXUDLTEMGPZSBWAABQTIAYZCELAOZIAPBTGTRMGFPTTEBALRAYPPN",
      "amount": "24427392",
      "tickNumber": 13686387,
      "inputType": 0,
      "inputSize": 0,
      "inputHex": "",
      "signatureHex": "526370cd218a33d53ff17d1de4194158e713222ebee88e53d26c73872d1946fcaae42ec83be1a4b84dbf14bc5458ed462ab43ccc63c4446554341e782c541700",
      "txId": "ktwllcxqbvlrffrbweestshxqxbhpulqwdnvljssmcuzuefuzcwufedgmkya"
    }
  ]
}

/ticks/{tick_number}/chain-hash

Returns the hash of the given processed tick. This is mainly used to compare archiver instances and verify they process ticks the same.

curl http://127.0.0.1:8001/ticks/13686387/chain-tick
{
  "hexDigest": "cf8914a346c217036a71d92f4e81c123aeea7a2d187f4123932e98326360c101"
}

Transaction related endpoints

/transactions/{tx_id}

Returns the transaction information for the given transaction id.

curl http://127.0.0.1:8001/transactions/ktwllcxqbvlrffrbweestshxqxbhpulqwdnvljssmcuzuefuzcwufedgmkya
{
  "transaction": {
    "sourceId": "ARALPBGBRNORYBDFRWKQSLENOELBMFJWOFKBRQJNXDXTRZPYGGFKSADAXJON",
    "destId": "NLRQDYJUXUDLTEMGPZSBWAABQTIAYZCELAOZIAPBTGTRMGFPTTEBALRAYPPN",
    "amount": "24427392",
    "tickNumber": 13686387,
    "inputType": 0,
    "inputSize": 0,
    "inputHex": "",
    "signatureHex": "526370cd218a33d53ff17d1de4194158e713222ebee88e53d26c73872d1946fcaae42ec83be1a4b84dbf14bc5458ed462ab43ccc63c4446554341e782c541700",
    "txId": "ktwllcxqbvlrffrbweestshxqxbhpulqwdnvljssmcuzuefuzcwufedgmkya"
  }
}

/tx-status/{tx_id}

Returns the status of the given transaction.

curl http://127.0.0.1:8001/tx-status/ktwllcxqbvlrffrbweestshxqxbhpulqwdnvljssmcuzuefuzcwufedgmkya
{
  "transactionStatus": {
    "txId": "ktwllcxqbvlrffrbweestshxqxbhpulqwdnvljssmcuzuefuzcwufedgmkya",
    "moneyFlew": true
  }
}

/identities/{identity}/transfer-transactions

Returns the list of transfer transactions for the given identity.

curl http://127.0.0.1:8001/identities/ARALPBGBRNORYBDFRWKQSLENOELBMFJWOFKBRQJNXDXTRZPYGGFKSADAXJON/transfer-transactions?start_tick=13686000&end_tick=13686400
{
  "transferTransactionsPerTick": [
    {
      "tickNumber": 13686387,
      "identity": "ARALPBGBRNORYBDFRWKQSLENOELBMFJWOFKBRQJNXDXTRZPYGGFKSADAXJON",
      "transactions": [
        {
          "sourceId": "ARALPBGBRNORYBDFRWKQSLENOELBMFJWOFKBRQJNXDXTRZPYGGFKSADAXJON",
          "destId": "NLRQDYJUXUDLTEMGPZSBWAABQTIAYZCELAOZIAPBTGTRMGFPTTEBALRAYPPN",
          "amount": "24427392",
          "tickNumber": 13686387,
          "inputType": 0,
          "inputSize": 0,
          "inputHex": "",
          "signatureHex": "526370cd218a33d53ff17d1de4194158e713222ebee88e53d26c73872d1946fcaae42ec83be1a4b84dbf14bc5458ed462ab43ccc63c4446554341e782c541700",
          "txId": "ktwllcxqbvlrffrbweestshxqxbhpulqwdnvljssmcuzuefuzcwufedgmkya"
        }
      ]
    }
  ]
}

Epoch related endpoints

/epochs/{epoch}/computors

Returns the list of computors for the given epoch

curl http://127.0.0.1:8001/epochs/107/computors
{
  "computors": {
    "epoch": 107,
    "identities": [
      "OTSWYJAUPATSAHMJBEYVRCPZXJPAYQOEWWYEZKUTWELLTLKHCPPGWXBCXVVH",
      "AUHFHHNWLWPXEEXRCAKDWUCYRDRAZNRDNISHEDZYAFYJOUZZKYFMMWFAAFXM",
      "YQZPLDQLMCWDSDETFBBABRXGNRTCZSSXFYSGBPFCZFNSSQZDJVDQYUPDNYIM",
      "RGGNEEZYXQYTYFNFTLQYZKNNFMSCTBRSNZJIQGCXKAVVELCXQQQRMAKDDGOA",
      "JKUYHUUTSYESTATAZVGPQCFFCHHACXLEBXJWUBNBBAFGPSRLNFLNYRPGXEHO",
      "HDPWVFRQBPZREDVBVHYJLMLDACFDLPCLKGJOMFETCGEHMPVNTXVOVVHCNATJ",
      "SPECJVLASVEPUGKQVXOPURMJUWYBLXAFWHPVEUHBVGAGUIZDJULCOZNGEQIC",
//      ...
      "SIZEQJHYGNJDIDFNXMZTJKZXUVUCUIEKGOHRJMRNNDIRIOMTTZSXUMKBXUWA"
    ],
    "signatureHex": "77a9faf00728badab3452b64a9c75e3997128068c4bc3dab1f741264f4f95ece4b11425098debc3984f592aab7d5d827206627a3bdefa24c29797d5065650000"
  }
}

go-archiver's People

Contributors

0xluk avatar linckode avatar

Stargazers

couch42 avatar avatar

Watchers

avatar avatar 0xdev247 avatar

Forkers

nulls0x5 evoandro

go-archiver's Issues

healthcheck endpoint

Rationale:

In order to have a way to detect if an archive node is synced up, we need a healthcheck endpoint.

We should define a threshold equal to 2 (preferably configurable)
This should compare lastProcessingTick with currentTickInfo of a qubic node, and if (currentTickInfo - lastProcessingTick) > threshold endpoint should return a non 2xx http status code, otherwise 200.

Things to consider:

  • Getting currentTickInfo only from a node is not desirable. There is an edgecase that we want to cover: If a bad node is reporting a very high tick, the condition will always be false and the system will be put offline. To avoid that we want to query multiple nodes (from the pool) sort them and check if the biggest one is not very big compared to the 2nd biggest (edgecase described above). If indeed it's higher than the 2nd with more than 50 ticks for example, then real currentTickInfo is 2nd biggest.

Update Go projects dependencies

Our archiver-related projects seem to have some outdated dependencies, some of them being deprecated.
We must ensure that all of our dependencies are up to date and that they do not break the projects.

Daily archive snapshots

JoeTom: "can we export a daily snapshot of the archive for people who would like to run their own nodes? can we cryptographically mark a complete epoch somehow to unsure the archive is complete?"

luk: "I think we can add a flag(which anyone can enable or disable) to check at startup store hash of lastProcessedTick against a predefined(also configurable by env) store hash which we can publish along with latest snapshot"

Identity transactions endpoint

The issue

Archiver has the /identities/{identity}/transfer-transactions endpoint which only returns the transfer transactions of an identity.
There is a need to also return:

  1. the mining transactions for an identity
  2. the completion status
    We should preferably achieve these results by only calling one endpoint at a time, unlike how we do it for ticks.

Proposals for issue 1

1. New endpoint - Not recommended

We could add new endpoints for fetching the mining transactions.
As an example, we could have another endpoint called /identities/{identity}/mining-transactions.
This endpoint would like /transfer-transactons, with the difference that we return just the mining transactions this time.
Reasons this might not be a good idea:

  • API consumer needs to make multiple requests to get a full list of transactions.

2. Return all the data at once - Not recommended

We could rename the /transfer-transactions endpoint to just /transactions and return both mining and transfer transactions at the same time.
This probably isn't the best idea for the following reasons:

  • Doesn't allow the API consumer to choose the information he needs.
  • May be more bandwidth intensive.

3. Change endpoint to allow for choosing the returned data - Recommended

As with Proposal 2 we could rename the /transfer-transactions to /transactions, the difference being that we could accept an input in the request that determines the output we return.
The input could be either another request parameter, or an object in the body request. Personally I would go with the object.

Pros and cons for input types:

Request parameter:

Pros:

  • Can be easily added, as we also accept the min and max tick as request parameters.
  • Can use GET method.
    Cons:
  • Tedious to assemble the request.
  • Tedious to if we want to add more arguments in the future.
Json body object

Pros:

  • Way easier to assemble the request, allows for grouping other request arguments together.
  • Easy to implement.
  • Easy to extend in the future with more arguments.
    Cons:
  • The request type will need to be POST.

Request type

Our new will server the purpose of choosing what kind of output the user desires.
For the time being we could have three accepted values:

  • TRANSFER
  • MINE
  • FULL

Based on the input value we will only return the required information.

Proposal for issue 2

We could change the struct definition for the Transaction type to also include the approved status as a boolean.
We could look at how /approved-transactions works to differentiate between approved and not approved transactions.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.