Releases: streamingfast/firehose-core
v0.1.3
This release bumps substreams to v1.1.9
Highlights
Substreams Scheduler Improvements for Parallel Processing
The substreams
scheduler has been improved to reduce the number of required jobs for parallel processing. This affects backprocessing
(preparing the states of modules up to a "start-block") and forward processing
(preparing the states and the outputs to speed up streaming in production-mode).
Jobs on tier2
workers are now divided in "stages", each stage generating the partial states for all the modules that have the same dependencies. A substreams
that has a single store won't be affected, but one that has 3 top-level stores, which used to run 3 jobs for every segment now only runs a single job per segment to get all the states ready.
Substreams State Store Selection
The substreams
server now accepts X-Sf-Substreams-Cache-Tag
header to select which Substreams state store URL should be used by the request. When performing a Substreams request, the servers will optionally pick the state store based on the header. This enable consumers to stay on the same cache version when the operators needs to bump the data version (reasons for this could be a bug in Substreams software that caused some cached data to be corrupted on invalid).
To benefit from this, operators that have a version currently in their state store URL should move the version part from --substreams-state-store-url
to the new flag --substreams-state-store-default-tag
. For example if today you have in your config:
start:
...
flags:
substreams-state-store-url: /<some>/<path>/v3
You should convert to:
start:
...
flags:
substreams-state-store-url: /<some>/<path>
substreams-state-store-default-tag: v3
Operators Upgrade
The app substreams-tier1
and substreams-tier2
should be upgraded concurrently. Some calls will fail while versions are misaligned.
Backend Changes
-
Authentication plugin
trust
can now specify an exclusive list ofallowed
headers (all lowercase), ex:trust://?allowed=x-sf-user-id,x-sf-api-key-id,x-real-ip,x-sf-substreams-cache-tag
-
The
tier2
app no longer uses thecommon-auth-plugin
,trust
will always be used, so thattier1
can pass down its headers (ex:X-Sf-Substreams-Cache-Tag
).
v0.1.2
Operator Changes
-
Added
fire<chain> tools check forks <forked-blocks-store-url> [--min-depth=<depth>]
that reads forked blocks you have and prints resolved longest forks you have seen. The command works for any chain, here a sample output:... Fork Depth 3 #45236230 [ea33194e0a9bb1d8 <= 164aa1b9c8a02af0 (on chain)] #45236231 [f7d2dc3fbdd0699c <= ea33194e0a9bb1d8] #45236232 [ed588cca9b1db391 <= f7d2dc3fbdd0699c] Fork Depth 2 #45236023 [b6b1c68c30b61166 <= 60083a796a079409 (on chain)] #45236024 [6d64aec1aece4a43 <= b6b1c68c30b61166] ...
-
The
fire<chain> tools
commands and sub-commands have better rendering--help
by hidden not needed global flags with long description.
v0.1.1
Backend Changes
- added missing
--substreams-tier2-request-stats
request debugging flag - added missing firehose rate limiting options flags,
--firehose-rate-limit-bucket-size
and--firehose-rate-limit-bucket-fill-rate
to manage concurrent connection attempts to Firehose.