Performance comparisons: nc mesh vs conf. mesh

[artv3]

Folks, we seem to loose quite a bit of performance when going the nc-mesh machinery.

This PR will generate the data for the plot below:

when using a conforming mesh, we see the following performance:

With hypre branch: parcsr_local_trans

• a change in ex1

[artv3]

Turning on the profiler for the following run:
./ex1 -p 0 -o 2 -l 14 -d cuda

shows that most of the run-time for the non-conforming mesh is dominated by sorting algorithms?

While there are no sorting algorithms in the conforming mesh case:

I have the profiler data if you guys want to look at it with me.

[tzanio]

Do you have the stack trace for stable_sort_by_key_domino_phase1? I am surprised a sort is being called during Matvec...

[v-dobrev]

I see that HYPRE creates and destroys a cuSparse matrix every time the matvec function is called -- maybe that's where these sorts come from?

[v-dobrev]

Or maybe when we do MultTranspose, the transpose is created and destroyed every time?

[YohannDudouit]

I see that HYPRE creates and destroys a cuSparse matrix every time the matvec function is called -- maybe that's where these sorts come from?

That would explain why I see:

8.82326s  8.6400us            (533 1 1)       (256 1 1)        38  8.5000KB        0B         -           -           -           -  Tesla V100-SXM2         1        17  void stable_sort_by_key_local_core<int=256, int=4>(int, int, int*, int*, int*, int*) [15956]
8.82327s  1.4080us                    -               -         -         -         -      128B  86.698MB/s      Device           -  Tesla V100-SXM2         1        17  [CUDA memset]
8.82327s  1.4080us                    -               -         -         -         -        4B  2.7093MB/s      Device           -  Tesla V100-SXM2         1        17  [CUDA memset]
8.82328s  4.4160us              (2 1 1)       (256 1 1)        22      256B        0B         -           -           -           -  Tesla V100-SXM2         1        17  void cusparseIinclusive_localscan_core<int=256, int=4>(int, int*, int*, int*) [15961]
8.82329s  4.7360us              (1 1 1)       (256 1 1)        24      264B        0B         -           -           -           -  Tesla V100-SXM2         1        17  void cusparseIinclusive_scan_domino_v1_core<int=256, int=4>(int, int*, int*, int*, int*, int*) [15963]
8.82330s  4.3840us              (2 1 1)       (256 1 1)        16        4B        0B         -           -           -           -  Tesla V100-SXM2         1        17  void cusparseIinclusive_scan_merge_core<int=256, int=4>(int, int, int*, int*, int*) [15965]
8.82331s  12.832us            (533 1 1)       (256 1 1)        28       16B        0B         -           -           -           -  Tesla V100-SXM2         1        17  void stable_sort_by_key_merge_core<int=256, int=4>(int, int*, int*, int*, int*, int*, int*) [15967]
8.82332s  5.6320us            (533 1 1)       (256 1 1)        24        0B        0B         -           -           -           -  Tesla V100-SXM2         1        17  void stable_sort_by_key_stop_core<int=256, int=4>(int, int*, int*) [15969]

being called over and over.

[v-dobrev]

I'm not sure the cuSparse constructor (with external data) or action need to do sorting. The transpose construction on GPU, on the other hand, probably needs some sorting.

[liruipeng]

Do you have $A^T x$ anywhere?

[v-dobrev]

Yes, we call hypre_ParCSRMatrixMatvecT. Does that create and destroy the transpose each time when running on GPU?

[liruipeng]

Yes. you can avoid that by HYPRE_BoomerAMGSetKeepTranspose but it only makes sense in the context of AMG. I need to make a branch for changing this.

[tzanio]

If the transpose will be constructed either way, isn't it better to create and store the $P^T$ and then just multiply with it?

[liruipeng]

If the transpose will be constructed either way, isn't it better to create and store the P^T and then just multiply with it?

If it's worth paying the cost of constructing and storing the global $P^T$, yes.

[tzanio]

What's the difference in storage between that and the HYPRE_BoomerAMGSetKeepTranspose approach?

[liruipeng]

OK. I take part of it back. Storage wise, local and global $P^T$ are more or less the same I think (except for some parcsr info other than the matrix itself). But constructing $P^T$ globally is more expensive, which requires communications .

[tzanio]

Yes. you can avoid that by HYPRE_BoomerAMGSetKeepTranspose but it only makes sense in the context of AMG. I need to make a branch for changing this.

@liruipeng, let us know when you have this ready for testing

[tomstitt]

Do we know if this is as performant as using Umpire? Also, does this disable the use of unified memory?

[liruipeng]

No. We suggest using Umpire if you can. We probably will remove our device memory pool implementation in near future since even CUDA now has the "official" memory pool.

[artv3]

The default for hypre is no unified memory (ref. docs).
--enable-unified-memory Use unified memory for allocating the memory
(default is NO).

If I recall correctly I think Umpire does improve the performance a bit, not sure by how what factor compared to the internal memory pool framework.

[artv3]

@liruipeng do you know how much faster it is to use umpire?

[liruipeng]

@liruipeng do you know how much faster it is to use umpire?

Hard to give a number. It highly depends on the application.

[liruipeng]

Yes. you can avoid that by HYPRE_BoomerAMGSetKeepTranspose but it only makes sense in the context of AMG. I need to make a branch for changing this.

@liruipeng, let us know when you have this ready for testing

@tzanio @v-dobrev
For here, you only do matrix-transpose-vector, without matrix-transpose-matrix multiplication. Is it correct?

[tzanio]

For here, you only do matrix-transpose-vector, without matrix-transpose-matrix multiplication. Is it correct?

Yes, overall we evaluate the action of $P^T A P$ by

• hypre MatVec with $P$
• mfem matrix-free partial assembly action of $A$
• hypre MatVecT with $P$

The discussion here is for the last step

[liruipeng]

@artv3 @tzanio I created a new branch https://github.com/hypre-space/hypre/tree/parcsr_local_trans

The way I imagined to work is to call hypre_ParCSRMatrixLocalTranspose on $P$, so it generates the local transposes. And in the following matvecT the transposes should be reused without recreations. Of course, you can also do the global transposition hypre_ParCSRMatrixTranspose which I expected to be less efficient.

Give it a try and let me know. Thanks!

[tzanio]

Thanks @liruipeng !

[artv3]

@artv3 @tzanio I created a new branch https://github.com/hypre-space/hypre/tree/parcsr_local_trans

The way I imagined to work is to call hypre_ParCSRMatrixLocalTranspose on P, so it generates the local transposes. And in the following matvecT the transposes should be reused without recreations. Of course, you can also do the global transposition hypre_ParCSRMatrixTranspose which I expected to be less efficient.

Give it a try and let me know. Thanks!

Improved performance! Looking much better