diff --git a/README.md b/README.md new file mode 100644 index 0000000..82ff0a0 --- /dev/null +++ b/README.md @@ -0,0 +1,73 @@ +# iavlread - extract data from the state of a Cosmos SDK blockchain + +This is a simple tool read state data from the snapshot of a [Cosmos SDK] block chain. The state is stored in the `application.db` leveldb database, in the form of an [IAVL tree]. This tool walks the IAVL tree to get values from the state, at a desired block height. + +### Installation + +It's really just two Python files. `iavlread` is the CLI tool, and `iavltree.py` is the library which actually handles the data structure. It can also be used as a package in Python code (`store.ipynb` shows a few examples). To use `iavlread`, it's easiest to just clone this repository and make a symlink to `iavlread` from `$HOME/bin` or other directory which in `$PATH`. + +### Usage + +I'm just going to show a couple of examples on a snapshot of the [Allora] testnet, as it can be downloaded e.g. [here](https://www.imperator.co/services/chain-services/testnets/allora). That's what I've been using this for, although I'd assume it works the same for other CosmosSDK blockchains. + +We assume that we're inside the snapshot, so the application db is at the path `data/application.db` from the current working directory. Otherwise, it can be specified with using `-d path_to_database`. + +To get the **maximum/minimum block height** contained in the snapshot + + $ iavlread max_height s/k:emissions/ + 5224814 + $ iavlread min_height s/k:emissions/ + 5221360 + +Here `s/k:emissions/` is the prefix for a specific IAVL tree, the one corresponding to the emissions module. Other prefixes are `s/k:mint/`, `s/k:bank/`, `s/k:staking/`, `s/k:acc/`. They should generally produce the same min/max height, but that is not guaranteed. + +To **count** the items in a keeper, use the `count` subcommand: + + $ iavlread count s/k:emissions/ + 19070982 + +We can also count only the items with a specific key (i.e. one item of the keeper). + + $ iavlread count s/k:emissions/ 62 + 9714538 + +Here 62 corresponds to the `latestOneOutInfererInfererNetworkRegrets` field, which has type `Map[Triple[uint64, string, string], TimestampedValue]`. So the keys for the individual item in the map are quadruples consisting of 62, an integer, and two strings. + +To **iterate through all items** whose key starts with 62, use the `iterate` command: + + $ iavlread -kQss -vpb,2=float iterate s/k:emissions/ 62 | head -n5 + ([62, 1, 'allo1004k7wqa4spns0wlct6mvxnmfysae07w2p75xc', 'allo1004k7wqa4spns0wlct6mvxnmfysae07w2p75xc'], [(1, 1655577), (2, -2.538390795862665)]) + ([62, 1, 'allo1004k7wqa4spns0wlct6mvxnmfysae07w2p75xc', 'allo123s56yuz7dkyh54gs7gstulrfzwj3d6ucwlwk2'], [(1, 1655577), (2, -2.9735713547935196)]) + ([62, 1, 'allo1004k7wqa4spns0wlct6mvxnmfysae07w2p75xc', 'allo136dfvvuhazgyqyls2f68qzr5l3p07uhnk9mmk0'], [(1, 1655577), (2, -2.4482888033843784)]) + ([62, 1, 'allo1004k7wqa4spns0wlct6mvxnmfysae07w2p75xc', 'allo15fvg8gk63u8ydf3znaaxrrukgwulezjz4azf68'], [(1, 1655577), (2, -2.4736050386063284)]) + ([62, 1, 'allo1004k7wqa4spns0wlct6mvxnmfysae07w2p75xc', 'allo16k9af4uu7vpm5hy68t6u62jylc5frv747mz5lw'], [(1, 1655577), (2, -2.4686292734004254)]) + +The option `-kQss` specifies the key format (a 64 bit integer `Q` followed by two strings `s`; see below). And `-vpb,2=float` specifies the value format: a protocol buffer whose field number 2 is a `float`. + +If we want to restrict to keys which start with 62 and 60 (i.e. get one-out regrets for topic 60 only) + + $ iavlread -kQss count s/k:emissions/ 62 60 | head -n5 + 10650 + $ iavlread -kQss -vpb,2=float iterate s/k:emissions/ 62 60 | head -n5 + ([62, 60, 'allo107zfy4xrp5plt0jmutaj9feer02v6r30amku78', 'allo107zfy4xrp5plt0jmutaj9feer02v6r30amku78'], [(1, 5070658), (2, 0.004613475335253211)]) + ([62, 60, 'allo107zfy4xrp5plt0jmutaj9feer02v6r30amku78', 'allo10q5h8afpwjh5x3vazxzwwkxfhpzfys9wxxw3q8'], [(1, 4785658), (2, 0.18922985130794248)]) + ([62, 60, 'allo107zfy4xrp5plt0jmutaj9feer02v6r30amku78', 'allo10vgaxk57dkk0fd255r3gxn5quwzxaqq95m2cz2'], [(1, 4837018), (2, 0.6616848257922094)]) + ([62, 60, 'allo107zfy4xrp5plt0jmutaj9feer02v6r30amku78', 'allo10w45atfjsh9q6vsk7mx74xh0pvuf8r42vnmt5p'], [(1, 4802038), (2, 0.010614832362991345)]) + ([62, 60, 'allo107zfy4xrp5plt0jmutaj9feer02v6r30amku78', 'allo12hnfamvwumkfm6dnc42rt8q3yevyqpuzkdtwat'], [(1, 4471438), (2, 0.2668976650081499)]) + +Or if we know the full key, we can use `get` instead of `iterate`. E.g. to get a balance from the bank module: + + $ iavlread -kbs -vint get s/k:bank/ 2 570DD38DC5BAF3112A7C83A420ED399A8E59C5FC uallo + 350 + +We can also get the value for at a different block height (by default, the max block height is used): + + $ iavlread -H 5221360 -kbs -vint get s/k:bank/ 2 570DD38DC5BAF3112A7C83A420ED399A8E59C5FC uallo + 10 + +Or we can get all past updates to the value (that are contained in the snapshot): + + $ iavlread -kbs -vint history s/k:bank/ 2 570DD38DC5BAF3112A7C83A420ED399A8E59C5FC uallo + 5224814 150 + 5224813 30 + 5224812 60