Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limit block import queue by segments, not blocks #2837

Open
nazar-pc opened this issue Jun 10, 2024 · 0 comments
Open

Limit block import queue by segments, not blocks #2837

nazar-pc opened this issue Jun 10, 2024 · 0 comments
Labels
improvement it is already working, but can be better node Node (service library/node app) performance Related to performance measurement or improvement

Comments

@nazar-pc
Copy link
Member

One of the architectural decisions that was carried over from Substrate sync to our DSN sync is to limit import queue size in blocks. However, blocks can be of different size. Sometimes we have more blocks in a segment than the limit, sometimes less, but the goal is to limit memory usage.

As such I think we should track which blocks segments end at and download one or even more segments ahead of time, regardless of how many blocks are in there. It is actually nice that we deal with segments here due to their predictabe size, we can even try to download next segment while previous segment is still downloading if we have available concurrency to do that (for example if there is a few slow to download pieces left in the previous segment, we should be able to download the next segment already).

This will accelerate sync from DSN, though how much will depend on block size and how heavy blocks are.

@nazar-pc nazar-pc added improvement it is already working, but can be better node Node (service library/node app) performance Related to performance measurement or improvement labels Jun 10, 2024
@nazar-pc nazar-pc added this to the Protocol UX Improvements milestone Jun 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement it is already working, but can be better node Node (service library/node app) performance Related to performance measurement or improvement
Projects
Development

No branches or pull requests

1 participant