Light

notional-labs / cosmprund Goto Github PK

View Code? Open in Web Editor NEW

10.0 2.0 7.0 136 KB

prune data for tendermint/cosmos based chains

Go 99.21% Makefile 0.79%

cosmprund's Introduction

Cosmos-Pruner

This is a fork of cosmprund with some improvements:

support pebbledb
prune all stores of app state
support blockstore prunning
support tx_index prunning

Usage

# clone & build cosmprund repo
git clone https://github.com/notional-labs/cosmprund
cd cosmprund
make build

# run cosmprund 
./build/cosmprund prune ~/.gaiad/data --app=cosmoshub --backend=pebbledb --blocks=362880 --versions=362880 --compact=true

Flags:

data-dir: path to data directory if not default
blocks: amount of blocks to keep on the node (Default 10)
versions: amount of app state versions to keep on the node (Default 10)
app: deprecated! does not use for anything exccep some special chains.
cosmos-sdk: If pruning app state (Default true)
tendermint: If pruning tendermint data including blockstore and state. (Default true)
tx_index: set to false you dont want to prune tx_index.db (default true)
compact: set to false you dont want to compact dbs after prunning (default true)

cosmprund's People

Contributors

Stargazers

Watchers

Forkers

baabeetaa qmin80 clemensgg ryssroad kinrokinro kii-awesome iwbinb

cosmprund's Issues

disable compaction when prunning

for pebbledb only, no problem for goleveldb

fix_sei: tx_index

key is encoded differently

https://github.com/sei-protocol/sei-tendermint/blob/4b4be3151c250f4f5ece0f1d4b7962148ed4c8dc/internal/state/indexer/tx/kv/kv.go#L554

https://github.com/tendermint/tendermint/blob/v0.34.24/state/txindex/kv/kv.go#L559

ref:#12

panic: pebble: batch too large

panic: pebble: batch too large: >= 4.0 G

goroutine 733 [running]:
github.com/cockroachdb/pebble.(*Batch).grow(0x0?, 0x418bff?)
	/root/go/pkg/mod/github.com/cockroachdb/[email protected]/batch.go:1108 +0x129
github.com/cockroachdb/pebble.(*Batch).prepareDeferredKeyRecord(0xc00311cfc0, 0x32, 0x0)
	/root/go/pkg/mod/github.com/cockroachdb/[email protected]/batch.go:498 +0x74
github.com/cockroachdb/pebble.(*Batch).DeleteDeferred(...)
	/root/go/pkg/mod/github.com/cockroachdb/[email protected]/batch.go:600
github.com/cockroachdb/pebble.(*Batch).Delete(0xc00311cfc0?, {0xc0b69d6dc0?, 0x32?, 0xc0b69d6dc0?}, 0xc0b69dc750?)
	/root/go/pkg/mod/github.com/cockroachdb/[email protected]/batch.go:583 +0x31
github.com/tendermint/tm-db.(*pebbleDBBatch).Delete(0x11?, {0xc0b69d6dc0?, 0xc006a03800?, 0x459bd2?})
	/root/go/pkg/mod/github.com/baabeetaa/[email protected]/pebble.go:283 +0x2e
github.com/tendermint/tm-db.prefixDBBatch.Delete({{0xc003696d68, 0x11, 0x18}, {0x1568f40, 0xc005a5e270}}, {0xc0b69e6030, 0x21, 0xc000100400?})
	/root/go/pkg/mod/github.com/baabeetaa/[email protected]/prefixdb_batch.go:35 +0x117
github.com/cosmos/iavl.(*nodeDB).DeleteVersionsRange.func1({0xc0b69e4511, 0x31, 0x31}, {0xc0b69d0b00, 0x20, 0x20})
	/root/go/pkg/mod/github.com/cosmos/[email protected]/nodedb.go:305 +0x1e4
github.com/cosmos/iavl.(*nodeDB).traversePrefix(0xc000e84b10?, {0xc0b69880d0?, 0x1?, 0x1?}, 0xc006a03c00)
	/root/go/pkg/mod/github.com/cosmos/[email protected]/nodedb.go:521 +0x112
github.com/cosmos/iavl.(*nodeDB).traverseOrphansVersion(0xc0022c87e0?, 0xc005fd1b80?, 0x4d2774?)
	/root/go/pkg/mod/github.com/cosmos/[email protected]/nodedb.go:487 +0x6e
github.com/cosmos/iavl.(*nodeDB).DeleteVersionsRange(0xc0022c87e0, 0x1cd8a0, 0x205b6c)
	/root/go/pkg/mod/github.com/cosmos/[email protected]/nodedb.go:298 +0x457
github.com/cosmos/iavl.(*MutableTree).DeleteVersionsRange(0xc0003e2460, 0x1cd8a0, 0x205b6c)
	/root/go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:623 +0x4a
github.com/cosmos/iavl.(*MutableTree).DeleteVersions(0x1559d80?, {0xc01933c000?, 0x382cc, 0x382d6})
	/root/go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:611 +0x2ca
github.com/cosmos/cosmos-sdk/store/iavl.(*Store).DeleteVersions(...)
	/root/go/pkg/mod/github.com/cosmos/[email protected]/store/iavl/store.go:201
github.com/binaryholdings/cosmos-pruner/internal/rootmulti.(*Store).PruneStores.func1({0x155bae0, 0xc0083953a0?})
	/root/cosmprund/internal/rootmulti/store.go:453 +0x159
created by github.com/binaryholdings/cosmos-pruner/internal/rootmulti.(*Store).PruneStores
	/root/cosmprund/internal/rootmulti/store.go:450 +0x225

pebblecompact

compact after prunning so no need to use https://github.com/notional-labs/pebblecompact

dydx-testnet-4 apphash

dydx-testnet-4 got apphash after running prunning tool.

add celestia

compact flag as an option

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.