Deep history updates #694
Deep history updates #694
Conversation
| Archives supporting historical lookup are available to download from a Google Cloud Storage bucket. | ||
|
|
||
| :::important | ||
| Documentation in this section is preliminary and subject to change. |
There was a problem hiding this comment.
This comes in the following PR.
| ### Downloading non-pruned database | ||
| ## Observer configuration | ||
|
|
||
| A deep history squad has it's observers configured to retain the whole, non-pruned history of the blockchain. That is, the observers must have the following settings in their `prefs.toml`: |
| :::tip | ||
| As of October 2022, a public deep-history squad isn't yet available. The instance is being prepared and should be ready in November 2022. | ||
| ::: | ||
| The observing squads backing the public Gateways, in addition to being full history squads (serving past blocks, transaction and events up until the Genesis), also act as 3-epochs deep-history squads. That is, for **mainnet**, one can use https://gateway.multiversx.com to resolve historical account (state) queries, for the last 3 days. This interval is driven by the configuration parameter `[StoragePruning.NumEpochsToKeep]`, which is set to `4`, by default. |
There was a problem hiding this comment.
serving past blocks, transactions and events
| FullArchive = true | ||
|
|
||
| OverridableConfigTomlValues = [ | ||
| { File = "config.toml", Path = "DbLookupExtensions.Enabled", Value = "true" }, |
There was a problem hiding this comment.
we might think of defining a node operation for these settings
There was a problem hiding this comment.
Good point, thanks! Actually, we do have something: full-archive, db-lookup-extension, historical-balances - I will re-check their compatibility.
Edit: currently testing.
There was a problem hiding this comment.
--operation-mode=historical-balances seems perfect 🚀
configs.GeneralConfig.StoragePruning.Enabled = true
configs.GeneralConfig.StoragePruning.ValidatorCleanOldEpochsData = false
configs.GeneralConfig.StoragePruning.ObserverCleanOldEpochsData = false
configs.GeneralConfig.GeneralSettings.StartInEpochEnabled = false
configs.GeneralConfig.StoragePruning.AccountsTrieCleanOldEpochsData = false
configs.GeneralConfig.StateTriesConfig.AccountsStatePruningEnabled = false
configs.GeneralConfig.DbLookupExtensions.Enabled = true
configs.PreferencesConfig.Preferences.FullArchive = true
It does not override SnapshotsEnabled = true and PeerStatePruningEnabled = true, but those are the official default anyway.
Thanks @bogdan-rosianu @iulianpascalau (great simplification)!
|
|
||
| [comment]: # (mx-context-auto) | ||
| An alternative to downloading the non-pruned history is to reconstruct it locally (on your own infrastructure). The reconstruction process is based on the **[import-db](/validators/import-db/)** feature, which re-processes past blocks - and, while doing so, retains the whole, non-pruned accounts history. |
There was a problem hiding this comment.
We can also add the possibility to sync from the network by using the full archive flag in prefs.toml
There was a problem hiding this comment.
Added a short note. But now I think that would be easier for users from the community to use a regular sync instead of import-db. In a next PR, we can:
a) drop the details regarding reconstruction using import-db, and only describe the simpler approach (even if it takes a bit more time, depending on the availability of peers, plus the network traffic).
b) have the regular-sync approach described side-by-side with the import-db approach.
Which one should we pick?
| git clone https://github.com/multiversx/mx-chain-mainnet-config --branch=$CONFIG_TAG --single-branch --depth=1 config | ||
|
|
||
| # In most cases, this should be the tag found in the file "config/binaryVersion": | ||
| export BINARY_TAG=v1.6.13 |
There was a problem hiding this comment.
maybe get the compatible binary from the binaryVersion file after getting the configs?
| :::tip | ||
| Downloading the archives and extracting them might take a while. | ||
| ``` | ||
| ./node --import-db=./import-db --import-db-no-sig-check --log-level=*:INFO --use-log-view --log-save --destination-shard-as-observer 0 |
There was a problem hiding this comment.
--use-log-view is deprecated. Will be removed in the upcoming releases
| Now, get (download) and extract **two daily archives (snapshots)** for the shard in question (e.g. shard 0): one archive that was created on the epoch preceding the **chosen start epoch** (e.g. the archive created during epoch `1255 - 1 = 1254`); and one archive that was created on the epoch following the **chosen end epoch** (e.g. the archive created during epoch `1260 + 1 = 1261`). You can compute the correct URLs for the two archives manually, given the [**mainnet** Genesis time](https://gateway.multiversx.com/network/config) and the chosen epochs, or you can use the following Python snippet: | ||
|
|
||
| ``` | ||
| import datetime |
| We are now ready to start the reconstruction process :rocket: | ||
|
|
||
| ``` | ||
| ./node --import-db=./import-db --import-db-no-sig-check --log-level=*:INFO --use-log-view --log-save --destination-shard-as-observer 0 |
There was a problem hiding this comment.
also here, no --use-log-view
Description of the pull request (what is new / what has changed)
Simplifications. Document new reconstruction flow for deep-history.
For reference, see: multiversx/mx-deprecated-chain-deep-history#19.
Did you test the changes locally ?
Which category (categories) does this pull request belong to?