Chunked C6 for Memory Efficiency

[marvinfriede]

No description provided.

[awvwgk]

Would it be better to provide an argument like memory and compute the chunk size based on the available memory?

[marvinfriede]

Would it be better to provide an argument like memory and compute the chunk size based on the available memory?

I thought about it, but decided against it for several reasons.

• First, it is not immediately clear, how much memory the calculations following the C6 build require (further usage of D3, gradients, ...). I played around with different chunk sizes and sometimes the calculation passed the C6 build, but still crashed later. So dynamically setting the chunk size might also cause memory issues later.
• Second, if the available memory is used, the results might not be reproducible due to other processes taking up resources.
• Third, I think, the memory is too crucial of a property to adjust automatically here. It should be the task of the user to properly set this up. With potentially non-reproducible behavior and OOM crashes at later stages, the problem might even become more difficult to diagnose for the user.