[DOC] Add best practices/advice with respect to using pool allocators

[wence-]

RMM has multiple pool-like allocators:

• a pool_memory_resource that wraps a coalescing best fit suballocator around an upstream resource;
• an arena_memory_resource that similarly wraps around an upstream resource but divides the global allocation into size-binned arenas to mitigate fragmentation when allocating/deallocating;
• and a cuda_async_memory_resource that uses the memory pool implementation provided by cudaMallocAsync. This one can avoid fragmentation because it is in control of the virtual address space.

Since these are all composable, one can happily wrap a pool_memory_resource around a cuda_async_memory_resource (or an arena, ...). But should one?

It would be useful if the documentation provided some guidance on which combinations make sense, and what typical allocation scenarios best fit a particular pool.

We should also recommend best practices for picking initial pool sizes: a bad choice here can lead to overfragmentation.

[harrism]

Side thought: Maybe we should experiment with replacing the cuda_memory_resource used for initial_resource with a cuda_async_memory_resource...

[wence-]

Side thought: Maybe we should experiment with replacing the cuda_memory_resource used for initial_resource with a cuda_async_memory_resource...

Maybe, though we'd have the usual static destruction problems, so we'd never explicitly free that memory pool.

It might also be problematic in the multiple library case where one library is not configured with a specific pool, so makes an allocation from the initial_resource which now builds a pool. And now we're conflicting (potentially) with other libraries that then set a pool up.

[harrism]

I was thinking by default the async resource uses the default pool, which we would not own. Maybe I'm misremembering how it's implemented.

[vyasr]

The async resource will use the pool managed by the CUDA driver, which we do not own and would probably be fine. Ideally everyone would use that and then all pooling would be handled by the driver. If we use the async mr by default and a different library does not but constructs their own pool manually using a different underlying allocation routine (e.g. cudaMalloc instead of cudaMallocAsync), then we could conflict.

[wence-]

In cuda_async_memory_resource we call cudaMempoolCreate and get a handle to a pool and use that to make our allocations. So that sounds like we own that pool.

[vyasr]

My mistake, I didn't realize that we were allocating from a specific pool that we created. The failure mode should still be relatively graceful if two processes both use the async allocation routines and one pool blocks another's growth. I don't think it will be as graceful if you mix and match async with non-async allocation, but I could be wrong there.

[harrism]

I believe the reason that cuda_async_memory_resource owns its pool is because we provide a non-owning MR: cuda_async_view_memory_resource. We could use the latter as the default resource with the default pool. That MR requires passing a cudaMemPool_t pool handle, and the default pool handle can be retrieved using cudaDeviceGetDefaultMempool.

Perhaps, however, we should wait to make this default change until we can start using the cuda_async_memory_resource from libcu++, which has a different design.

[bdice]

Just updating this since it's been a few months since the last reply.

My understanding of the CCCL memory resource design agrees with Mark's last statement. It seems like it will be sensible to have RMM's initial memory resource be something that acts like RMM's current cuda_async_view_memory_resource, constructed with the driver's default pool. As I worked on #1864 (Python bindings for the async view MR), I considered that design a bit more and it seems like a better "default" behavior than what RMM does today.

I'll need to have some follow-up conversations with the CCCL and cuda-python teams about this once we're ready to act on this idea. Probably not in the next release or two.

[bdice]

To address the original topic (documenting best practices for pool allocators), I think the original post has the right questions that users will want to understand. @wence- @vyasr @harrism Do you have any opinionated guidance off the top of your head that would help us answer those questions for users? Maybe we can consult Spark folks on arena? Who else has expertise in this area?

Specifically:

• When should one use RMM's pool adaptor (which requires a large pre-allocated pool) vs. using the CUDA async MR that uses a driver pool internally? (The docs on the CUDA async MR could probably stand to have a lot more detail here)
• When are arena MRs recommended? (Mitigating fragmentation -- but more info here would help)

[harrism]

The main reason to use the pool adaptor today is to make a pool of managed memory (until managed memory is available with cudaMallocAsync in all CUDA versions you need to support). However, the pool adaptor continues to be the default in a lot of libraries like cudf.

Last I checked (1-2 years ago) pool adaptor also still outperforms cudaMallocAsync for device memory for a certain distribution of allocation sizes (although I am foggy on the details).

I always wanted to create a fragmentation benchmark to compare the MRs on that metric, but I'm not sure how best to do it.