kadena-io / chainweb-data Goto Github PK

View Code? Open in Web Editor NEW

14.0 17.0 8.0 1001 KB

Data ingestion for Chainweb.

License: BSD 3-Clause "New" or "Revised" License

Haskell 97.78% Nix 2.22%

chainweb kadena haskell

chainweb-data's People

Contributors

Stargazers

Watchers

Forkers

newmight2015 thednatechorg lbpage runonflux trendzetter alber70g devopsgoth mbwmbw1337

chainweb-data's Issues

Make txHandler properly handle reintroduced transactions

The runSelectReturningOne call erroneously fails when a transaction has been reintroduced.

Chainweb-data fill always causes ResponseTimeout on the node

Trying to fill up the chainweb-data using a local node. The node is configured with the parameters specified in node-config-for-chainweb-data.yaml. No error is shown in the chainweb-node log. Is there any other data I can provide to troubleshoot the issue?

chainweb-data-2.3.0$ chainweb-data fill --service-host=localhost --dbstring="host=localhost dbname=chainweb port=5432 user=chainweb password=x"
2023-10-29T18:39:33.289Z [Info] [] Using database: PGString "host=localhost dbname=chainweb port=5432 user=chainweb password=x"
2023-10-29T18:39:33.289Z [Info] [] Service API: http://localhost:1848
2023-10-29T18:39:33.312Z [Info] [] No migrations to run
2023-10-29T18:39:33.363Z [Info] [] The DB schema is compatible with the ORM definition.
2023-10-29T18:39:33.363Z [Info] [] DB Tables Initialized
2023-10-29T18:39:33.476Z [Info] [] Filling 78 gaps and 75489252 blocks
2023-10-29T18:40:03.477Z [Info] [] Progress: 6840/75489252 (0.01%), ~91 hours remaining at 228 current items per second (228 overall average).
chainweb-data: HttpExceptionRequest Request {
  host                 = "localhost"
  port                 = 1848
  secure               = False
  requestHeaders       = [("content-type","application/json")]
  path                 = "/chainweb/0.0/mainnet01/chain/9/payload/outputs/batch"
  queryString          = ""
  method               = "POST"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 ResponseTimeout

Use postgres date/time types for timestamp fields

Support the new WebAuthn keys/sigs

When making transactions using the new WebAuthn keysets those transactions end up in an error when Chainweb Data attempts to parse it. The error indicates that the sig cannot be parsed, which is due to the new interface necessary for WebAuthn based signatures.

For our kadena internal folks, you can navigate to the webauthn-wallet PoC. Follow the README.md instructions and simply register and then buy a cookie.

Here is the error message:

2023-11-23T11:30:03.517Z [Error] [] Couldn't fetch payload batch for chain: 14
2023-11-23T11:30:03.517Z [Error] [] ApiError {apiError_type = OtherError "Decoding error in payloadWithOutputsBatch: Error in $[0].transactions[0][0]: Error in $.sigs[0]: key \"sig\" not found\nHashes: ( \"dprjb8G8OyOiBgQF_uIhLOyRfXQ1KSqwvcwGwr2-60E\" )", apiError_status = Status {statusCode = 200, statusMessage = "OK"}, apiError_body = "[{\"transactions\":[[\"eyJoYXNoIjoiWG1mRFhfZk1IUFhTQ2k0alFQMnlWU01MOUZPLVBYUENLQ1oySTVyR3k2NCIsInNpZ3MiOlt7ImF1dGhlbnRpY2F0b3JEYXRhIjoibDROVFNkWkJjNGN4MDZSMVE0K0hrSFVra29DRFRpMDF2WklLazd6eUNsd2RBQUFBQUE9PSIsInNpZ25hdHVyZSI6Ik1FUUNJSEJsQ2Y1VmhUbjVrVitvQUhPRGJsU0Y1S0JZV3hleUpzbm8wZEVMSGJ3MEFpQVY1aFlQVHFiczYzZTQ1a2VaTVdPRngxd3J5K1JXRFd5R0dYMGJVZUZqK2c9PSIsImNsaWVudERhdGFKU09OIjoiZXlKMGVYQmxJam9pZDJWaVlYVjBhRzR1WjJWMElpd2lZMmhoYkd4bGJtZGxJam9pV0cxbVJGaGZaazFJVUZoVFEyazBhbEZRTW5sV1UwMU1PVVpQTFZCWVVFTkxRMW95U1RWeVIzazJOQ0lzSW05eWFXZHBiaUk2SW1oMGRIQnpPbHd2WEM5M1pXSmhkWFJvYmkxM1lXeHNaWFF0WkdWdGJ5NTJaWEpqWld3dVlYQndJaXdpWVc1a2NtOXBaRkJoWTJ0aFoyVk9ZVzFsSWpvaVkyOXRMbTFwWTNKdmMyOW1kQzVsYlcxNEluMD0ifV0sImNtZCI6IntcInBheWxvYWRcIjp7XCJleGVjXCI6e1wiY29kZVwiOlwiKG5fNTYwZWVmY2VlNGEwOTBhMjRmMTJkN2NmNjhjZDQ4ZjExZDhkMmJkOS53ZWJhdXRobi13YWxsZXQudHJhbnNmZXIgXFxcInc6WHlKX3BLcmxfbXZzZFlEM19UMEtrRGpiLUZEVWxxemdZeDNYRzcybXQwNDprZXlzLWFueVxcXCIgXFxcImNvb2tpZS1zaG9wXFxcIiA2LjU1KVwiLFwiZGF0YVwiOnt9fX0sXCJtZXRhXCI6e1wiZ2FzTGltaXRcIjoxMDAwLFwiZ2FzUHJpY2VcIjoxZS03LFwic2VuZGVyXCI6XCJjOm5qVjY2cEhCWm9kVmNaWU83enR3TXk2cllteVFRV3YxN29WdDN6RC0tbGdcIixcInR0bFwiOjYwMDAwLFwiY3JlYXRpb25UaW1lXCI6MTcwMDczNjk0MyxcImNoYWluSWRcIjpcIjE0XCJ9LFwibmV0d29ya0lkXCI6XCJmYXN0LWRldmVsb3BtZW50XCIsXCJzaWduZXJzXCI6W3tcInB1YktleVwiOlwiV0VCQVVUSE4tYTUwMTAyMDMyNjIwMDEyMTU4MjA0MGFjZGJhZjliODJlOTJlODliOTBlMDgyNzkwMTQ4ZDQ5ZjJmOWVmYTk0ZTkzMzdjNDFhMGI3NzI5ODEyMmZkMjI1ODIwNGQyMzMyZGVmOTllMGY5ODMxYTUxZWYyNDZkZjBhZmJmYWVhMTFiYWY0ZDM4MWE4Njc3YWQzMTQ5ZjI3YjIwOFwiLFwic2NoZW1lXCI6XCJXZWJBdXRoblwiLFwiY2xpc3RcIjpbe1wibmFtZVwiOlwibl81NjBlZWZjZWU0YTA5MGEyNGYxMmQ3Y2Y2OGNkNDhmMTFkOGQyYmQ5LndlYmF1dGhuLXdhbGxldC5UUkFOU0ZFUlwiLFwiYXJnc1wiOltcInc6WHlKX3BLcmxfbXZzZFlEM19UMEtrRGpiLUZEVWxxemdZeDNYRzcybXQwNDprZXlzLWFueVwiLFwiY29va2llLXNob3BcIiw2LjU1XX0se1wibmFtZVwiOlwibl81NjBlZWZjZWU0YTA5MGEyNGYxMmQ3Y2Y2OGNkNDhmMTFkOGQyYmQ5LndlYmF1dGhuLXdhbGxldC5HQVNfUEFZRVJcIixcImFyZ3NcIjpbXCJ3Olh5Sl9wS3JsX212c2RZRDNfVDBLa0RqYi1GRFVscXpnWXgzWEc3Mm10MDQ6a2V5cy1hbnlcIix7XCJpbnRcIjoxfSwxXX0se1wibmFtZVwiOlwiY29pbi5HQVNcIixcImFyZ3NcIjpbXX1dfV0sXCJub25jZVwiOlwia2pzOm5vbmNlOjE3MDA3MzY5NDM2ODlcIn0ifQ\",\"eyJnYXMiOjg0NywicmVzdWx0Ijp7InN0YXR1cyI6InN1Y2Nlc3MiLCJkYXRhIjoiV3JpdGUgc3VjY2VlZGVkIn0sInJlcUtleSI6IlhtZkRYX2ZNSFBYU0NpNGpRUDJ5VlNNTDlGTy1QWFBDS0NaMkk1ckd5NjQiLCJsb2dzIjoiZXlXQVlrSGNIZkdlOFZsUnFLLWtlVFZmdDNSZXZxallnUVZfb2tPNURBdyIsImV2ZW50cyI6W3sicGFyYW1zIjpbImM6bmpWNjZwSEJab2RWY1pZTzd6dHdNeTZyWW15UVFXdjE3b1Z0M3pELS1sZyIsIms6ZjkwZWY0NjkyN2Y1MDZjNzBiNmE1OGZkMzIyNDUwYTkzNjMxMWRjNmFjOTFmNGVjM2Q4ZWY5NDk2MDhkYmYxZiIsOC40N2UtNV0sIm5hbWUiOiJUUkFOU0ZFUiIsIm1vZHVsZSI6eyJuYW1lc3BhY2UiOm51bGwsIm5hbWUiOiJjb2luIn0sIm1vZHVsZUhhc2giOiJNMWdhYmFrcWtFaV8xTjhkUkt0NHo1bEV2MWt1Q19ueExUbnlEQ3VaSUswIn0seyJwYXJhbXMiOlsidzpYeUpfcEtybF9tdnNkWUQzX1QwS2tEamItRkRVbHF6Z1l4M1hHNzJtdDA0OmtleXMtYW55IiwiY29va2llLXNob3AiLDYuNTVdLCJuYW1lIjoiVFJBTlNGRVIiLCJtb2R1bGUiOnsibmFtZXNwYWNlIjoibl81NjBlZWZjZWU0YTA5MGEyNGYxMmQ3Y2Y2OGNkNDhmMTFkOGQyYmQ5IiwibmFtZSI6IndlYmF1dGhuLXdhbGxldCJ9LCJtb2R1bGVIYXNoIjoiNG5hQUtRZnNqQ3pEQ1hkT0JONjREUnBSbFczbU9PclJkYk9UUjFtdUdlRSJ9LHsicGFyYW1zIjpbImM6bmpWNjZwSEJab2RWY1pZTzd6dHdNeTZyWW15UVFXdjE3b1Z0M3pELS1sZyIsImNvb2tpZS1zaG9wIiw2LjU1XSwibmFtZSI6IlRSQU5TRkVSIiwibW9kdWxlIjp7Im5hbWVzcGFjZSI6bnVsbCwibmFtZSI6ImNvaW4ifSwibW9kdWxlSGFzaCI6Ik0xZ2FiYWtxa0VpXzFOOGRSS3Q0ejVsRXYxa3VDX254TFRueURDdVpJSzAifV0sIm1ldGFEYXRhIjpudWxsLCJjb250aW51YXRpb24iOm51bGwsInR4SWQiOjEwMDUxfQ\"]],\"minerData\":\"eyJhY2NvdW50IjoiazpmOTBlZjQ2OTI3ZjUwNmM3MGI2YTU4ZmQzMjI0NTBhOTM2MzExZGM2YWM5MWY0ZWMzZDhlZjk0OTYwOGRiZjFmIiwicHJlZGljYXRlIjoia2V5cy1hbGwiLCJwdWJsaWMta2V5cyI6WyJmOTBlZjQ2OTI3ZjUwNmM3MGI2YTU4ZmQzMjI0NTBhOTM2MzExZGM2YWM5MWY0ZWMzZDhlZjk0OTYwOGRiZjFmIl19\",\"transactionsHash\":\"KrlXpOlV881bgB2RgN2LBahS6eBB69wxDeKEximAWFU\",\"outputsHash\":\"3ugNDEkvinUW-6COYQrPMZoIEK8uFWVOJL-6O7WHg1A\",\"payloadHash\":\"qY_NhWVh4ZQV6fpV2pSVbC_yi637R48Qw6jT_Vd4vFc\",\"coinbase\":\"eyJnYXMiOjAsInJlc3VsdCI6eyJzdGF0dXMiOiJzdWNjZXNzIiwiZGF0YSI6IldyaXRlIHN1Y2NlZWRlZCJ9LCJyZXFLZXkiOiJJbGxuYTB4YU5HTlFlVll6U1ZwYU5UaE9iamg1UjJOMU9GZGFOVGxFTXpaVlN6Vlpia0ZtVFUxeFgwMGkiLCJsb2dzIjoiVXlHaWl1WnYyVUVDb1ZqZ2ZWWE1vcWlxNFhyNU1BY0VkcEltb3V0NUhjZyIsImV2ZW50cyI6W3sicGFyYW1zIjpbIiIsIms6ZjkwZWY0NjkyN2Y1MDZjNzBiNmE1OGZkMzIyNDUwYTkzNjMxMWRjNmFjOTFmNGVjM2Q4ZWY5NDk2MDhkYmYxZiIsMS4xNTIyNjE1XSwibmFtZSI6IlRSQU5TRkVSIiwibW9kdWxlIjp7Im5hbWVzcGFjZSI6bnVsbCwibmFtZSI6ImNvaW4ifSwibW9kdWxlSGFzaCI6Ik0xZ2FiYWtxa0VpXzFOOGRSS3Q0ejVsRXYxa3VDX254TFRueURDdVpJSzAifV0sIm1ldGFEYXRhIjpudWxsLCJjb250aW51YXRpb24iOm51bGwsInR4SWQiOjEwMDQ5fQ\"}]"}

Repeated endpoint description in README.md

In the file README.md, at the list of available endpoints the description (gets the details of a transaction with the given request key) for the transaction search by request key endpoint, /txs/tx?requestkey=<request-key>, is repeated in the events endpoint, /txs/events?search=foo&limit=20&offset=40, e.g.:

/txs/tx?requestkey=<request-key> gets the details of a transaction with the given request key
/txs/events?search=foo&limit=20&offset=40 gets the details of a transaction with the given request key

GitHub workflow that publishes binaries to S3 and GitHub artifacts

Make gaps more efficient by offloading to DB

We have a query here that calculates the gaps in Postgres. This should allow us to make gaps significantly more efficient because the DB doesn't have to send all the block heights and chain IDs back to chainweb-data. This should first be implemented as a separate command gaps2 so that the existing gaps functionality is preserved. Later on after the new command has been well tested we can think about getting rid of the old implementation.

Latest Master nix-build Seg Fault on Docker Image Build

Having this across over 40 different nodes on various flavours of Ubuntu OS... Never had an issue building the docker image before.

trace: WARNING: 9.6.3 is out of date, consider using 9.6.4.
these 3 derivations will be built:
  /nix/store/qm10dzfq8fb7wmwsbz5qr4k0grn15yl1-docker-layer-chainweb-data.drv
  /nix/store/pihjkyv052jhr1c6694w966sl8anbyyl-runtime-deps.drv
  /nix/store/n3482wsdrfcc1d6nxrpdg01s1snbdgr6-docker-image-chainweb-data.tar.gz.drv
building '/nix/store/qm10dzfq8fb7wmwsbz5qr4k0grn15yl1-docker-layer-chainweb-data.drv'...
Formatting './image/disk-image.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=1073741824 lazy_refcounts=off refcount_bits=16
Could not access KVM kernel module: Permission denied
qemu-kvm: failed to initialize kvm: Permission denied
qemu-kvm: falling back to tcg
cSeaBIOS (version rel-1.16.2-0-gea1b7a073390-prebuilt.qemu.org)


iPXE (http://ipxe.org) 00:03.0 CA00 PCI2.10 PnP PMM+1EFD0F00+1EF30F00 CA00
                                                                               


Booting from ROM...
Probing EDD (edd=off to disable)... ok
c./run-vm: line 5:    12 Segmentation fault      /nix/store/cfvrcbmb5bvvrz63f4c5dyvvijwfgg74-qemu-host-cpu-only-8.1.0/bin/qemu-kvm -cpu max -nographic -no-reboot -device virtio-rng-pci -virtfs local,path=/nix/store,security_model=none,mount_tag=store -virtfs local,path=/build/xchg,security_model=none,mount_tag=xchg -drive file=./image/disk-image.qcow2,if=virtio,cache=unsafe,werror=report -kernel /nix/store/ng20x62170qvivq1ym2gbcxq1s8vpl1d-linux-6.1.53/bzImage -initrd /nix/store/ijp66m0xyb3b9h998w3i7c9in9invw72-initrd/initrd -append "console=ttyS0 panic=1 command=/nix/store/0qgga5n54adny309857n7lwaz90hnbg1-vm-run-stage2 out=/nix/store/sb1lqgm3v38xnfn1l38z69z7r61f4l4r-docker-layer-chainweb-data mountDisk= loglevel=4" -m 512 -smp cpus=16
error: builder for '/nix/store/qm10dzfq8fb7wmwsbz5qr4k0grn15yl1-docker-layer-chainweb-data.drv' failed with exit code 139
error: 1 dependencies of derivation '/nix/store/n3482wsdrfcc1d6nxrpdg01s1snbdgr6-docker-image-chainweb-data.tar.gz.drv' failed to build

Transfers request taking unexpectedly long

@emmanueldenloye noticed that the following request takes a very long time to respond:

time curl -v "$CWDURL/txs/account/6d87fd6e5e47185cb421459d2888bddba7a6c0f2c4ae5246d5f38f993818bb89"
...
real	0m3,437s

Disabling -threaded dramatically reduces CPU consumption

I am using a Nix build (GHC 8.10.7) on Intel macOS 11.6.8 (Big Sur).

If I build and run chainweb-data normally, it often uses up to 700% CPU (7 full cores) continuously on my iMac Pro, even when nothing is being done. GHC profiling shows that the executable itself is not doing much work, even over a 24 hour period. Here is the output from that profiling run:

chainweb-data.prof.zip

This is for a multiple hour run, but you'll see that it reports actual time consumed as being only 2 seconds!

What I've discovered is that if I disable threading in chainweb-data, everything still works just fine, but all of the excess CPU utilization completely disappears! So this is the change I make to the cabal file:

executable chainweb-data
  import:         commons
  main-is:        Main.hs
  hs-source-dirs: exec
  -- ghc-options:    -threaded -rtsopts -with-rtsopts=-N
  build-depends:

I'm creating this issue to suggest that we make this change in chainweb-data if we don't actually need the multi-threaded behavior.

Block lineage tracking

The goal of this issue is to implement an efficient lineage tracking for chainweb-data. Currently, there's no efficient nor straightforward way to deduce lineage information about a block or any data that derives from a block. For example, if one wants to determine whether a block is an orphan or even worse, if one wants to jump from a continuation transaction back to its earlier steps that are in their blockchain history one would need to perform complex and expensive SQL queries.

Increasing CPU and memory consumption over time

Concern

When running chainweb-data application over a long period of time CPU and memory consumption increase. I am worried that eventually, it will result in the degradation of the system's performance.
May I please get the explanation of the observed behavior?

ModuleHash should be pretty-printed in block explorer output

Seems like it's just using a default Show instance at the moment:

Use postgres numeric type for target and weight

Script-based Migrations

We're currently using beam-automigrate + a number of custom migration commands at chainweb-data start up for managing our Postgres schema. This combination served us well so far, but going forward we're planning to use more and more advanced features of Postgres like partitioned tables, triggers etc. so it seems like we'd be better served by converting to the more conventional migration scheme of expressing the schema migrations as a sequence of SQL scripts to be executed at start up.

As part of this transition to the script-based migrations, we also need to consider the case of an existing chainweb-data deployment using a very old version of the chainweb-data binary. Here's how we're planning to perform this transition in a way that works for the existing chainweb-data deployments too:

We declare that the next release is a "transition version".
The transition version keeps migrating the database using beam-automigrate same as how been doing it so far.
The transition version also runs the MigrationInitialization command of postgresql-simple-migration seems like a good library to support this alternative way of managing schema migrations. Marking the deployment as having gone through the "transition migration". I.e. when this command is run, we know that the DB schema is in the known transition state.
We use pg_dump to take a snapshot of the Postgres DB schema as it appears at the transition version and manually go through the dump to make sure we don't have anything deployment-specific in there so that it can be run on any empty Postgres database. We call the resulting SQL dump init.sql.
Post-transition versions of chainweb-data performs the following logic:
- If blocks table doesn't exist) That means this is an entirely new deployment, so we run init.sql followed by the incremental migrations scripts.
- If blocks table exists) Then we check whether MigrationInitialization has already been performed. (I.e. we run the MigrationValidation MigrationInitialization MigrationCommand.)
  - If MigrationInitialization was run) This means this is a properly transitioned deployment, so we just apply any missing migration scripts.
  - If MigrationInitialization was not run) That means this is a pre-transition deployment, so we exit with a message instructing the user to run the transition release

Constraints on Added Tables/Columns (Outside of Chainweb-data rquired) throws error

While ORM code has been updated to allow for additional tables within the database, adding a primary key to the table or other constraints to tables outside of the chainweb-data ORM causes an error.

I.e. add a new table, add a primary key. ORM checker throws an error.

I have fixed this and will open PR.

BA.TableConstraintRemoved{} -> False BA.ColumnConstraintRemoved{} -> False

Sigs field missing from txdata

There is no sigs field in txdata despite PR #153

Cross posting as it may be related to the issues starting in block-explorer through PR kadena-io/block-explorer#79

Unable to build with Nix and Cabal

The problem

I cannot build the project on my local machine (Linux Mint).
It will be more community-friendly if the project contains a Dockerfile that would allow building the project outside of GitHub Actions.

I prepared two such Dockerfiles, one using NixOS and building with Nix (recommended for newcomers) and one repeating the steps from GitHub Workflows: https://github.com/kadena-io/chainweb-data/blob/master/.github/workflows/ci.yaml.
Both Dockers fail in a repeatable way, it would be great if the team fixes at least one of those for us, newcomers, as the reference.

Build with Nix

Dockerfile

Place it in the project's root directory:

FROM nixos/nix
VOLUME nix

# Disable the default settings
RUN cat /etc/nix/nix.conf
RUN mv /etc/nix/nix.conf /etc/nix/nix.conf.default

# Set Kadena's development-specific settings
COPY nix.conf /etc/nix/nix.conf
RUN cat /etc/nix/nix.conf

WORKDIR /chainweb-data
COPY . .

CMD nix-build

Additional files

Nix configuration file nix.conf:

build-users-group = nixbld
sandbox = false

substituters = https://nixcache.chainweb.com https://cache.nixos.org/
trusted-public-keys = nixcache.chainweb.com:FVN503ABX9F8x8K0ptnc99XEz5SaA4Sks6kNcZn2pBY= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=

Docker commands

docker build -t chainweb-data_nix .
docker run -v nix:/nix -it chainweb-data_nix

Result

Building executable 'chainweb-data' for chainweb-data-2.0.1..
[ 1 of 11] Compiling Chainweb.Coins   ( exec/Chainweb/Coins.hs, dist/build/chainweb-data/chainweb-data-tmp/Chainweb/Coins.o )
[ 2 of 11] Compiling Chainweb.Lookups ( exec/Chainweb/Lookups.hs, dist/build/chainweb-data/chainweb-data-tmp/Chainweb/Lookups.o )
[ 3 of 11] Compiling Chainweb.RichList ( exec/Chainweb/RichList.hs, dist/build/chainweb-data/chainweb-data-tmp/Chainweb/RichList.o )
[ 4 of 11] Compiling Chainweb.Worker  ( exec/Chainweb/Worker.hs, dist/build/chainweb-data/chainweb-data-tmp/Chainweb/Worker.o )
[ 5 of 11] Compiling Chainweb.Single  ( exec/Chainweb/Single.hs, dist/build/chainweb-data/chainweb-data-tmp/Chainweb/Single.o )
[ 6 of 11] Compiling Chainweb.Listen  ( exec/Chainweb/Listen.hs, dist/build/chainweb-data/chainweb-data-tmp/Chainweb/Listen.o )
[ 7 of 11] Compiling Chainweb.Gaps    ( exec/Chainweb/Gaps.hs, dist/build/chainweb-data/chainweb-data-tmp/Chainweb/Gaps.o )
[ 8 of 11] Compiling Chainweb.Server  ( exec/Chainweb/Server.hs, dist/build/chainweb-data/chainweb-data-tmp/Chainweb/Server.o )

exec/Chainweb/Server.hs:171:19: error:
    • Couldn't match type ‘Maybe RequestKey -> Handler [TxDetail]’
                     with ‘(Maybe RequestKey -> Handler [TxDetail])
                           :<|> (Text
                                 -> Text
                                 -> Int
                                 -> Maybe Limit
                                 -> Maybe Offset
                                 -> Handler [ChainwebData.AccountDetail.AccountDetail])’
      Expected type: Server TheApi
        Actual type: ((Handler [TxSummary]
                       :<|> ((Maybe Limit
                              -> Maybe Offset -> Maybe Text -> Handler [TxSummary])
                             :<|> ((Maybe Limit
                                    -> Maybe Offset
                                    -> Maybe Text
                                    -> Maybe EventParam
                                    -> Maybe EventName
                                    -> Maybe EventModuleName
                                    -> Maybe BlockHeight
                                    -> Handler [EventDetail])
                                   :<|> ((Maybe RequestKey -> Handler TxDetail)
                                         :<|> (Maybe RequestKey -> Handler [TxDetail])))))
                      :<|> (Handler ChainwebDataStats :<|> Handler Text))
                     :<|> Handler Text
    • In the second argument of ‘serve’, namely ‘(serverApp req)’
      In the expression: serve theApi (serverApp req) req f
      In the second argument of ‘($)’, namely
        ‘\ req f -> serve theApi (serverApp req) req f’
    |
171 |     serve theApi (serverApp req) req f
    |                   ^^^^^^^^^^^^^
[ 9 of 11] Compiling Chainweb.FillEvents ( exec/Chainweb/FillEvents.hs, dist/build/chainweb-data/chainweb-data-tmp/Chainweb/FillEvents.o )
[10 of 11] Compiling Chainweb.Backfill ( exec/Chainweb/Backfill.hs, dist/build/chainweb-data/chainweb-data-tmp/Chainweb/Backfill.o )
error: builder for '/nix/store/8j3b64gs4vvvb7h8pcb0fjn7sianyynm-chainweb-data-2.0.1.drv' failed with exit code 1

Build with Cabal

Dockerfile

Place it in the project's root directory:

FROM ubuntu:20.04
RUN apt-get update
RUN apt-get install -y wget curl build-essential git
WORKDIR /root
RUN wget -q https://downloads.haskell.org/~ghcup/0.1.17.8/x86_64-linux-ghcup-0.1.17.8
RUN mv x86_64-linux-ghcup-0.1.17.8 ghcup
RUN chmod +x ghcup
COPY . /project
CMD cd /project && ./docker-build.sh

Additional files

The build command, file docker-build.sh:

/root/ghcup install ghc 8.8.4
/root/ghcup install cabal 3.4.1
export PATH="$HOME/.ghcup/ghc/8.8.4/bin:$HOME/.cabal/bin:$HOME/.ghcup/bin:$PATH"
ghc --version
cabal --version
cabal update
cabal outdated
cabal build --only-dependencies
cabal build

Docker commands

docker build -t chainweb-data_cabal .
docker run -v cabal_root:/root -it chainweb-data_cabal

Result

Failed to build Boolean-0.2.4.
Build log (
/root/.cabal/logs/ghc-8.8.4/Boolean-0.2.4-2a8e7c88c49fc506101b3277f2946417e7c1acd5391d4e663c0491e645dfaed1.log
):
Configuring Boolean-0.2.4...
Preprocessing library for Boolean-0.2.4..
Building library for Boolean-0.2.4..
[1 of 3] Compiling Data.Boolean     ( src/Data/Boolean.hs, dist/build/Data/Boolean.o )
[2 of 3] Compiling Data.Boolean.Numbers ( src/Data/Boolean/Numbers.hs, dist/build/Data/Boolean/Numbers.o )
[3 of 3] Compiling Data.Boolean.Overload ( src/Data/Boolean/Overload.hs, dist/build/Data/Boolean/Overload.o )
/usr/bin/ld.gold: error: cannot find -lgmp
collect2: error: ld returned 1 exit status
`gcc' failed in phase `Linker'. (Exit code: 1)

Add column for miner reward txid to the blocks table

Miner Insertion Race

There seems to be a rare race condition involving the insertion of miner values. It's only likely to happen with a fresh db, when blocks with the same associated miner are being inserted.

Missing information about indexes required for indexer's endpoints

Indexer performance problems

Investigation

Naming

In the description below following names are being used for:

Postgres super-user: postgres,
database name: chainweb-data,
database user: db_user.

Increase debug query size

Increase the query size, so it is possible to display large queries:

sudo -u postgres psql chainweb-data
ALTER SYSTEM SET track_activity_query_size = 16384;
quit
systemctl restart postgresql

Intercept long-running query

Invoke the indexer's endpoint, which fails to return the data quickly:

curl http://localhost:8080/txs/search?limit=200&search=k:5adb16663073280acf63bc2a4bf477ad1391736dcd6217b094926862c72d15c9

Connect to the database and display the running SQL queries:

sudo -u db_user psql -d chainweb-data -U db_user -W
SELECT pid, age(clock_timestamp(), query_start), usename, query FROM pg_stat_activity WHERE query != '<IDLE>' AND query NOT ILIKE '%pg_stat_activity%' ORDER BY query_start desc;

Then analyze the query:

explain analyze SELECT "t0"."chainid" AS "res0", "t1"."height" AS "res1", "t0"."block" AS "res2", "t0"."creationtime" AS "res3", "t0"."requestkey" AS "res4", "t0"."sender" AS "res5", "t0"."code" AS "res6", "t0"."continuation" AS "res7", "t0"."goodresult" AS "res8" FROM "transactions" AS "t0" CROSS JOIN "blocks" AS "t1" WHERE (("t0"."block") = ("t1"."hash")) AND (("t0"."code") LIKE ('%k:5adb16663073280acf63bc2a4bf477ad1391736dcd6217b094926862c72d15c9%')) ORDER BY "t1"."height" DESC LIMIT 100 OFFSET 0

Analysis gives the following query plan:

"QUERY PLAN"
"Limit  (cost=556941.22..556952.89 rows=100 width=1064) (actual time=2328.803..2336.257 rows=81 loops=1)"
"  ->  Gather Merge  (cost=556941.22..556990.92 rows=426 width=1064) (actual time=2105.215..2112.657 rows=81 loops=1)"
"        Workers Planned: 2"
"        Workers Launched: 2"
"        ->  Sort  (cost=555941.20..555941.73 rows=213 width=1064) (actual time=2093.093..2093.099 rows=27 loops=3)"
"              Sort Key: t1.height DESC"
"              Sort Method: quicksort  Memory: 64kB"
"              Worker 0:  Sort Method: quicksort  Memory: 55kB"
"              Worker 1:  Sort Method: quicksort  Memory: 52kB"
"              ->  Nested Loop  (cost=0.56..555933.06 rows=213 width=1064) (actual time=521.487..2093.012 rows=27 loops=3)"
"                    ->  Parallel Seq Scan on transactions t0  (cost=0.00..554104.98 rows=213 width=1056) (actual time=521.416..2078.174 rows=27 loops=3)"
"                          Filter: ((code)::text ~~ '%k:5adb16663073280acf63bc2a4bf477ad1391736dcd6217b094926862c72d15c9%'::text)"
"                          Rows Removed by Filter: 1824218"
"                    ->  Index Scan using blocks_pkey on blocks t1  (cost=0.56..8.58 rows=1 width=52) (actual time=0.546..0.546 rows=1 loops=81)"
"                          Index Cond: ((hash)::text = (t0.block)::text)"
"Planning Time: 9.838 ms"
"JIT:"
"  Functions: 26"
"  Options: Inlining true, Optimization true, Expressions true, Deforming true"
"  Timing: Generation 3.805 ms, Inlining 210.140 ms, Optimization 366.445 ms, Emission 201.074 ms, Total 781.465 ms"
"Execution Time: 2348.104 ms"

The line -> Parallel Seq Scan on transactions t0 (cost=0.00..554104.98 rows=213 width=1056) (actual time=521.416..2078.174 rows=27 loops=3)"
indicates that the whole table needs to get sequentially scanned (Seq Scan) because of the lack of appropriate index on the code column.
After adding the necessary index the query plan is the following:

"QUERY PLAN"
"Limit  (cost=7239.73..7239.98 rows=100 width=1064) (actual time=145.033..145.048 rows=81 loops=1)"
"  ->  Sort  (cost=7239.73..7241.01 rows=511 width=1064) (actual time=145.032..145.040 rows=81 loops=1)"
"        Sort Key: t1.height DESC"
"        Sort Method: quicksort  Memory: 123kB"
"        ->  Nested Loop  (cost=832.52..7220.20 rows=511 width=1064) (actual time=121.291..144.978 rows=81 loops=1)"
"              ->  Bitmap Heap Scan on transactions t0  (cost=831.96..2834.55 rows=511 width=1056) (actual time=121.263..144.195 rows=81 loops=1)"
"                    Recheck Cond: ((code)::text ~~ '%k:5adb16663073280acf63bc2a4bf477ad1391736dcd6217b094926862c72d15c9%'::text)"
"                    Rows Removed by Index Recheck: 376"
"                    Heap Blocks: exact=405"
"                    ->  Bitmap Index Scan on ""transactions-code""  (cost=0.00..831.83 rows=511 width=0) (actual time=113.783..113.783 rows=457 loops=1)"
"                          Index Cond: ((code)::text ~~ '%k:5adb16663073280acf63bc2a4bf477ad1391736dcd6217b094926862c72d15c9%'::text)"
"              ->  Index Scan using blocks_pkey on blocks t1  (cost=0.56..8.58 rows=1 width=52) (actual time=0.009..0.009 rows=1 loops=81)"
"                    Index Cond: ((hash)::text = (t0.block)::text)"
"Planning Time: 0.405 ms"
"Execution Time: 145.100 ms"

The Seq Scan is now gone in favour of scanning the index Bitmap Index Scan on ""transactions-code"".

Solution

The discussion about the right index supporting generic LIKE queries is here:
https://stackoverflow.com/questions/1566717/postgresql-like-query-performance-variations
In summary, the use of GIN or GiST trigram index with the special operator classes provided by the additional module pg_trgm is the solution.

Install additional Postgres module

To install the needed module, log into database with super-user:

sudo -u postgres psql chainweb-data
create extension pg_trgm;
quit

Identify missing indexes

The analysis of the queries resulted in the following list of the indexes that need to get created:

Indexer endpoint	Table	Column	Index
/txs/tx?requestkey=[...]	transactions	requestkey	`btree`
/txs/tx?requestkey=[...]	events	requestkey	`btree`
/txs/search?search=[...]	transactions	code	`gin_trgm_ops`
/txs/events?search=[...]	events	qualname	`gin_trgm_ops`
/txs/events?search=[...]	events	paramtext	`gin_trgm_ops`

Create appropriate indexes

The commands to create the above indexes:

sudo -u db_user psql -d chainweb-data -U db_user -W

CREATE INDEX "transactions-requestkey" ON public.transactions USING btree (requestkey);
CREATE INDEX "events-requestkey" ON public.events USING btree (requestkey);

CREATE INDEX "transactions-code" ON public.transactions USING gin (code gin_trgm_ops);
CREATE INDEX "events-qualname" ON public.events USING gin (qualname gin_trgm_ops);
CREATE INDEX "events-paramtext" ON public.events USING gin (paramtext gin_trgm_ops);

quit

Should this section get included in the README?


** Performance
Some endpoints require the creation of additional indexes in order to work efficiently:

| Indexer endpoint         | Table        | Column     | Index          |
|--------------------------+--------------+------------+----------------|
| /txs/tx?requestkey=[...] | transactions | requestkey | `btree`        |
| /txs/tx?requestkey=[...] | events | requestkey | `btree`        |
| /txs/search?search=[...] | transactions | code       | `gin_trgm_ops` |
| /txs/events?search=[...] | events       | qualname   | `gin_trgm_ops` |
| /txs/events?search=[...] | events       | paramtext  | `gin_trgm_ops` |

The use of GIN or GiST trigram index with the special operator classes provided
by the additional Postgresql module `pg_trgm` is the solution.

*** Install additional Postgres module
To install the needed module, log into database with super-user:
#+begin_example
sudo -u postgres psql chainweb-data
create extension pg_trgm;
quit
#+end_example

*** Create appropriate indexes
The commands to create the above indexes:
#+begin_example
sudo -u db_user psql -d chainweb-data-db -U db_user -W

CREATE INDEX "transactions-requestkey" ON public.transactions USING btree (requestkey);

CREATE INDEX "transactions-code" ON public.transactions USING gin (code gin_trgm_ops);
CREATE INDEX "events-qualname" ON public.events USING gin (qualname gin_trgm_ops);
CREATE INDEX "events-paramtext" ON public.events USING gin (paramtext gin_trgm_ops);

quit
#+end_example

Migration Fails due to Expecting Signers to be in CWD

Migration fails due to "cwd" schema expected in haskell-src/db-schema/migrations/2.2.0.1_swap_signers_pkey.sql

2023-07-24T07:12:57.664Z [Info] [] Running migration: 2.2.0.1_swap_signers_pkey.sql
chainweb-data: SqlError {sqlState = "42P01", sqlExecStatus = FatalError, sqlErrorMsg = "relation \"cwd.signers\" does not exist", sqlErrorDetail = "", sqlErrorHint = ""}

Solution is to remove "cwd" schmea prefix.

Store hashes consistently in hex

chainweb-data=# select powhash from blocks limit 10;
                             powhash
------------------------------------------------------------------
 0000000000000156c931661d6ecfee72c7789bcc7792b63b316006f26fd0336f
 000000000001e07e120762afc29678c0278802ef4db297827169b45d70a8a414
 000000000000836c53022c32c1c166926a003af307159e183fdfd2567a543edd
 0000000000001958e57a41ea2ee544352d7e465d759ef0160b11aa35b9293e11
 BOQ_HOAVEqs80XEtWxdx0vovBFP2uaPPUgiErrsHqeQ=
 vSm9aAj1YIoix3QjHbL6DiKHqvKQjBFHX2_NH13lGzw=
 Fid3V5qk8Sa-haoMKRXtdttdahP776vKpRVYnwac07k=
 BxZ_SnT7nKRikhuwYiToe-klYiELjAVXkxwvTSt1zvs=
 Aip4EdsyovdLf2nRgygyh00Ng4kVAlYT1SA7zlzVYvY=
 nQNkADqnXctuHdjHQ79ss33pICYCUy90cESWZteiTEM=
(10 rows)

kadena-io / chainweb-data Goto Github PK

chainweb-data's People

Contributors

Stargazers

Watchers

Forkers

chainweb-data's Issues

Concern

The problem

Build with Nix

Dockerfile

Additional files

Docker commands

Result

Build with Cabal

Dockerfile

Additional files

Docker commands

Result

Indexer performance problems

Investigation

Naming

Increase debug query size

Intercept long-running query

Solution

Install additional Postgres module

Identify missing indexes

Create appropriate indexes

Should this section get included in the README?

Recommend Projects

Recommend Topics

Recommend Org