faiss::gpu::runMatrixMult failure

[hellolovetiger]

The full log:
Faiss assertion err == CUBLAS_STATUS_SUCCESS failed in void faiss::gpu::runMatrixMult(faiss::gpu::Tensor<T, 2, true>&, bool, faiss::gpu::Tensor<T, 2, true>&, bool, faiss::gpu::Tensor<T, 2, true>&, bool, float, float, cublasHandle_t, cudaStream_t) [with T = float; cublasHandle_t = cublasContext*; cudaStream_t = CUstream_st*] at utils/MatrixMult.cu:141Aborted (core dumped)

I have successfully run demo_ivfpq_indexing_gpu, which I think the faiss was installed successfully.

[wickedfoo]

Possibility that you ran out of GPU memory?

[wickedfoo]

What were you trying to run?

[hellolovetiger]

train data shape: (2000000, 1000)
base data shape: (20000000, 1000)
query data shape: (1000000, 1000)
data type: float32

my code:

index = faiss.index_factory(d, "OPQ16_512,IVF4096,PQ16")
co = faiss.GpuClonerOptions()
co.usePrecomputed = False
index = faiss.index_cpu_to_gpu(res, 0, index, co)

index.train(xt)
del xt

index.add(xb)  # error happends here

My GPU memory is 8GB. I just tried the bench bench_gpu_sift1m.py, the same error.

[wickedfoo]

index.add(xb) # error happends here

instead of giving all of the (20000000, 1000) at once, try giving it in chunks of (10000, 1000) or so.
This is a issue that will be fixed at some point, the GPU side is less friendly unless you handle chunking the input beforehand, but eventually we'll handle that automatically.

[wickedfoo]

Only GpuIndexFlat* handles passing large amounts of data all at once for add or search at present.

[hellolovetiger]

I see. Actually I used numpy.memmap to load the data. Sorry, could you give me some guidance on how to chunk the input data that can be loaded with index.add?

[hellolovetiger]

Also, I notice that my GPU memory occupation in training is always about 20%. That's strange.

[hellolovetiger]

Just made some changes on the bench code bench_gpu_sift1m.py, still the same error. Populating top 10000 not work, either. Seems it is not memory issue. Maybe there is something wrong with the CUBLAS. By the way, do you have a plan to publish an official docker image to avoid some problems caused by installation?

#################################################################
#  Approximate search experiment
#################################################################

print "============ Approximate search"

index = faiss.index_factory(d, "IVF4096,PQ64")

# faster, uses more memory
# index = faiss.index_factory(d, "IVF16384,Flat")

co = faiss.GpuClonerOptions()

# here we are using a 64-byte PQ, so we must set the lookup tables to
# 16 bit float (this is due to the limited temporary memory).
co.useFloat16 = True

index = faiss.index_cpu_to_gpu(res, 0, index, co)

print "train"

index.train(xt)

print "add vectors to index"

index.add(xb[:10000])

[mdouze]

Hi
Note that the code above will not work for 1000-dim data (because 1000 is not a multiple of 64).
We do not have plans for a Docker image.

[hellolovetiger]

Hi, mdouze. Above code is from bench_gpu_sift1m.py. I used the data from http://corpus-texmex.irisa.fr/, following the instruction in https://github.com/facebookresearch/faiss/tree/master/benchs. I just wanted to check if the bench code works. Turn out to be the same error with my own.

[mdouze]

Ok, so this is the exact script bench_gpu_sift1m.py applied to the SIFT1M dataset and not your 20M*1000-dim dataset, correct?
On which type of GPU are you running this?

[hellolovetiger]

Yes for your first question.
My GPU is GeForce GTX 1080

[mdouze]

It could be the same bug as issue #8. Unfortunately we do not have the hardware to reproduce it, so we would be grateful if you could narrow down the error for us:

• Does it still crash in the add?
• If yes, could you add fewer vectors until it does not crash any more?
• could you set co.usePrecomputed = false and test again?
• could you reduce the 2 numbers in "IVF4096,PQ64" by powers of two until it does not crash any more?

[wickedfoo]

You can also try running cuda-memcheck on the bench_gpu_sift1m.py to see if anything gets printed out that does not look like the following:

========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaPointerGetAttributes. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib64/nvidia/libcuda.so.1 [0x2eea03]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x126239]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x16e44]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x1d066]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x1d1e2]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x1889f]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x194e5]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0xb504c]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x2332f]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x260d0]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0xf8cb]
=========     Host Frame:/lib64/libc.so.6 (__libc_start_main + 0xf5) [0x21b35]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0xf415]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaGetLastError. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib64/nvidia/libcuda.so.1 [0x2eea03]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x11de53]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x16e65]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x1d066]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x1d1e2]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x1889f]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x194e5]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0xb504c]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x2332f]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x260d0]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0xf8cb]
=========     Host Frame:/lib64/libc.so.6 (__libc_start_main + 0xf5) [0x21b35]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0xf415]

Another thing is to try resetting the GPU via nvidia-smi and trying again.

Also, you could try and investigate which CUDA shared libraries it is trying to load, to see if there is a mismatch if you have multiple CUDA SDK versions installed.

[wickedfoo]

Also, I notice that my GPU memory occupation in training is always about 20%. That's strange.

Faiss GPU reserves about 18% of available GPU memory up front for scratch space. This amount is controllable via StandardGpuResources, but it will run slower if you decrease it by a lot (due to cudaMalloc/cudaFree overhead). 1-2 GB of scratch space seems to be appropriate for most workloads.