Add bisection test #203
Open
Add bisection test #203
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Can't you already do that running the sendrecv_perf test, and setting NCCL_TESTS_SPLIT_MASK=(nranks/2)-1? Sure, that only works with nranks being a power of two; maybe your code is more generic. |
|
Being able to run this test when nranks is not a power of two is actually important to us |
AddyLaddy
reviewed
Apr 4, 2024
x41lakazam
force-pushed
the
bisection_test
branch
3 times, most recently
from
52ebfc6
to
42a079f
Compare
Add bisection to makefile Add bisection doc in performance.md
x41lakazam
force-pushed
the
bisection_test
branch
from
42a079f
to
ff27d6e
Compare
|
The code is ready to merge from our side |
AddyLaddy
reviewed
May 27, 2024
| } | ||
|
|
||
| int getPeer(int rank, int n_ranks){ | ||
| if (n_ranks % 4 == 0) |
There was a problem hiding this comment.
Why is n_ranks % 4 == 0 a special case?
There was a problem hiding this comment.
This allow cross rail bisection for rail-optimized topology
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In a bisection, each rank is paired with one other rank called 'peer', they both send and receive N messages, this number is bound to the
agg_itersparameter.The selection of this 'peer' rank is defined by the
getPeerfunction:https://github.com/x41lakazam/nccl-tests/blob/bisection_test/src/bisection.cu#L19