Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use a worker pool to process ApprovalDistribution messages #1941

Conversation

sandreim
Copy link
Contributor

@sandreim sandreim commented Oct 19, 2023

This should be merged to feature branch: https://github.com/paritytech/polkadot-sdk/tree/sandreim/network_scalability

The first step for implementing #1617

High level changes:

  • worker/worker pool abstractions in subsystem-util
  • refactor the approval-distribution subsystem to use a thread pool for processing the messages

Important: there are no logic changes and this is mostly plumbing work.

At this stage I want to gather more feedback on the approach in this PR and also for how to do the same for approval-voting. I currently see two options:

  • merge the approval-voting logic into the current approval-distribution worker
  • use a separate (and larger) pool for approval-voting logic

What's missing in this PR:

  • test fixes
  • new tests
  • documentation

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
…reim/refactor-approval-checking

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
@sandreim sandreim added T0-node This PR/Issue is related to the topic “node”. I9-optimisation An enhancement to provide better overall performance in terms of time-to-completion for a task. T8-parachains_engineering labels Oct 19, 2023
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
@paritytech-cicd-pr
Copy link

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-clippy
Logs: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4023605

serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Mar 26, 2024
…tytech#1941)

* actually clone client data by reference when clonning the client

* spelling

* clippy
serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Mar 27, 2024
…tytech#1941)

* actually clone client data by reference when clonning the client

* spelling

* clippy
serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Apr 8, 2024
…tytech#1941)

* actually clone client data by reference when clonning the client

* spelling

* clippy
serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Apr 8, 2024
…tytech#1941)

* actually clone client data by reference when clonning the client

* spelling

* clippy
serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Apr 8, 2024
…tytech#1941)

* actually clone client data by reference when clonning the client

* spelling

* clippy
serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Apr 8, 2024
…tytech#1941)

* actually clone client data by reference when clonning the client

* spelling

* clippy
serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Apr 8, 2024
…tytech#1941)

* actually clone client data by reference when clonning the client

* spelling

* clippy
serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Apr 9, 2024
…tytech#1941)

* actually clone client data by reference when clonning the client

* spelling

* clippy
serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Apr 9, 2024
…tytech#1941)

* actually clone client data by reference when clonning the client

* spelling

* clippy
serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Apr 9, 2024
…tytech#1941)

* actually clone client data by reference when clonning the client

* spelling

* clippy
serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Apr 9, 2024
…tytech#1941)

* actually clone client data by reference when clonning the client

* spelling

* clippy
serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Apr 9, 2024
…tytech#1941)

* actually clone client data by reference when clonning the client

* spelling

* clippy
serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Apr 9, 2024
…tytech#1941)

* actually clone client data by reference when clonning the client

* spelling

* clippy
serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Apr 10, 2024
…tytech#1941)

* actually clone client data by reference when clonning the client

* spelling

* clippy
serban300 pushed a commit to serban300/polkadot-sdk that referenced this pull request Apr 10, 2024
…tytech#1941)

* actually clone client data by reference when clonning the client

* spelling

* clippy
bkchr pushed a commit that referenced this pull request Apr 10, 2024
* actually clone client data by reference when clonning the client

* spelling

* clippy
@sandreim sandreim closed this Jul 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
I9-optimisation An enhancement to provide better overall performance in terms of time-to-completion for a task. T0-node This PR/Issue is related to the topic “node”.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants