Multi-GPU support

[soumith]

MultiGPU support has been implemented in cutorch (and by extension all torch cuda libraries like cunn, cudnn etc.).

• Switch the device on the fly with cutorch.setDevice(devID)
• All cuda calls are asynchronous, and can be synchronized with cutorch.synchronize()

Example usage for tensors:

-- Let us do matrix addition for matrices sitting on two different GPUs
cutorch.setDevice(1)
matrix1 = torch.CudaTensor(10):fill(1)
print(matrix1) -- printing is a synchronous call, so you dont have to explicitly call cutorch.synchronize()
cutorch.setDevice(2)
matrix2 = torch.CudaTensor(10):fill(2)
print(matrix2) 
matrix2:add(matrix1) -- matrix1 is seamlessly copied onto GPU2 and added to matrix2
print(matrix2)

if you want to do data-parallel training of neural nets (including convnets), your training loop can run like this:

For each mini-batch:

1. load data (preferably using multiple threads, for example using [threads-ffi](https://github.com/torch/threads-ffi))
2. loop over GPUs (the loop below will be completely anynchronous, so will run parallely)
  2.1. model[gpuX]:forward
  2.2. criterion[gpuX]:forward
  2.3. criterion[gpuX]:backward
  2.4. model[gpuX]:backward
3. cutorch.synchronize()
4. accumulate GPUx's gradParameters to GPU1's gradParameters
5. do SGD on GPU1
6. copy back GPU1's parameters to GPUx
7. cutorch.synchronize() and print accuracy etc.

Loop back to 1 for next mini-batch

Also, to train ConvNets using multiple GPUs, I recommend using CuDNN for the convolution layers, as I've tested that they are completely asynchronous (meaning that the processing runs parallely on multiple GPUs)

Comments below describe the technical details of changes made. If you just want to use Multi-GPU, you can stop reading now.

[soumith]

Comments below describe the technical details of changes made. If you just want to use Multi-GPU, you can stop reading now.
What is missing?

cutorch.setDevice right now resets the random seed as well. This needs to be separated so that you can use multiple GPUs as needed
https://github.com/torch/cutorch/blob/master/init.c#L42

GPU-to-GPU copy now has to use a host-bridge. This needs to be changed to P2P GPU copy. This is really trivial to implement, in the cutorch initialization function, we just have to enable p2p for each GPU detected with this function:
http://developer.download.nvidia.com/compute/cuda/4_1/rel/toolkit/docs/online/group__CUDART__PEER_g9e5ea65a18938c2b8715a5602105c306.html

After that UVA takes care of everything else, copying tensors from one GPU to another is as simple as this:
cutorch.setDevice(1)
t1 = torch.randn(100):cuda()
cutorch.setDevice(2)
t2 = torch.randn(100):cuda()
-- UVA copy
t2:copy(t1)

Internally, Clement and us have multi-GPU support, and we will get the changes back to cutorch slowly (it will take time to isolate the commits and get approval etc), but if you are really adventurous, this is a couple of hours of work.

[soumith]

@stencilman @jonathantompson

[nicholas-leonard]

And I am guessing we should use this for our D2D memory copies: http://developer.download.nvidia.com/compute/cuda/4_1/rel/toolkit/docs/online/group__CUDART__MEMORY_g046702971bc5a66d9bc6000682a6d844.html#g046702971bc5a66d9bc6000682a6d844

This means that if I have kernel sequence A->B->C (one device 1), followed by device2device memcopy D , followed by kernels (one device 2) E->F->G, then eventually A->B->C of iteration t should run in parallel to E->F->G of the previous iteration (t-1), right? Otherwise, I don't see how this can be useful, other than allowing the use of more GPU memory. I mean ideally, you want those GPUs to work on different kernels concurrently. Say A->B->C are the first 3 modules of nn.Sequential, and E->F->G are the last 3.

[soumith]

@nicholas-leonard you dont need to do cudaMemcpyPeer explicitly anymore, UVA takes care of it.

[soumith]

Reference: http://on-demand.gputechconf.com/gtc-express/2011/presentations/cuda_webinars_GPUDirect_uva.pdf

[nicholas-leonard]

Wow. So you are right, it would be super easy to implement. We check for device UVA flag, then call cudaDeviceEnablePeerAccess for every combination of such devices. Easy.

[nicholas-leonard]

Still, would the two sequences in the above example be able to run concurrently?

[soumith]

yes they would run concurrently starting the next itreration as long as you have no blocking calls anywhere.
clement already removed all the blocking calls from the convnet pipeline in a few commits earlier this year (dont remember the exact hashes).

[szagoruyko]

@soumith I did the UVA init for GPUs and got rid of "cuda runtime error : an illegal memory access was encountered" errors while copying directly from one tensor to another, I'm not sure, however, that network calls are not blocking everywhere. How do we test it?

[soumith]

@szagoruyko profile it (with sys.tic()/sys.toc() or just with os.clock().
if model:forward() or model:backward() takes more then say 5-10ms, there is blocking calls.
you should only see a big timing number once you do cutorch.synchronize(), and until then all the cuda calls should not take any time at all.

[soumith]

@szagoruyko SpatialConvolutionMM seems to be randomly blocking sometimes, due to for-looped cublas calls. Use CuDNN, there is no blocking at all.

[clementfarabet]

Yeah, this is really annoying, I could never figure out why this gemm calls
are blocking sometimes, there's got to be a reason...

On Wed, Sep 24, 2014 at 12:26 PM, Soumith Chintala <notifications@github.com

wrote:

@szagoruyko https://github.com/szagoruyko SpatialConvolutionMM seems to
be randomly blocking sometimes, due to for-looped cublas calls. Use CuDNN,
there is no blocking at all.

—
Reply to this email directly or view it on GitHub
#42 (comment).

[soumith]

@clementfarabet I've even tried moving to cublasv2, and that didnt help. Maybe CuBLAS has a queue that gets filled? It's only conjecture, as we dont have source code.

[clementfarabet]

It probably does, in which case we would need to use streams

Clément

On Sep 24, 2014, at 12:30 PM, Soumith Chintala notifications@github.com wrote:

@clementfarabet I've even tried moving to cublasv2, and that didnt help. Maybe CuBLAS has a queue that gets filled? It's only conjecture, as we dont have source code.

—
Reply to this email directly or view it on GitHub.