GPU memory not cleaned up after off-loading layers to GPU using `n_gpu_layers`

[nidhishs]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

Please provide a detailed written description of what you were trying to do, and what you expected llama-cpp-python to do.

def generate_text(model_id:str, prompt:str) -> str:
    llm = LlamaCpp(model_path=f'./weights/{model_id}.bin', n_gpu_layers=40)
    output = llm(prompt)
    return output

After calling this function, the llm object still occupies memory on the GPU.

Current Behavior

Please provide a detailed written description of what llama-cpp-python did, instead.
The llm object should clean up after itself and clear GPU memory. The GPU memory is only released after terminating the python process.

Environment and Context

Please provide detailed information about your computer setup. This is important in case the issue is not reproducible except for under certain specific conditions.

• Physical (or virtual) hardware you are using, e.g. for Linux:

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         48 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  16
  On-line CPU(s) list:   0-15
Vendor ID:               AuthenticAMD
  Model name:            AMD Ryzen 7 5800X 8-Core Processor
    CPU family:          25
    Model:               33
    Thread(s) per core:  2
    Core(s) per socket:  8
    Socket(s):           1
    Stepping:            2
    Frequency boost:     enabled
    CPU(s) scaling MHz:  58%
    CPU max MHz:         4850.1948
    CPU min MHz:         2200.0000
    BogoMIPS:            7602.85
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 
                         constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt ae
                         s xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core
                          perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rd
                         t_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerp
                         tr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgi
                         f v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm
Virtualization features: 
  Virtualization:        AMD-V
Caches (sum of all):     
  L1d:                   256 KiB (8 instances)
  L1i:                   256 KiB (8 instances)
  L2:                    4 MiB (8 instances)
  L3:                    32 MiB (1 instance)
NUMA:                    
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-15
Vulnerabilities:         
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Not affected
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected

• Operating System, e.g. for Linux:

Linux name 6.2.13-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 26 Apr 2023 20:50:14 +0000 x86_64 GNU/Linux

[iactix]

I am encountering this too. Windows, CuBLAS, AMD CPU, RTX 1080. When the llm model is destroyed you get the RAM back but the VRAM stays occupied until the whole python script using it quits. That means you get out of memory doing a second inference this way. Which makes GPU acceleration unusable for me, currently. Kind of a big deal I think.

[gjmulder]

Is this a bug then in how llama-cpp-python is managing the shared lib libllama or a bug within libllama?

I suspect the latter, which means a bug needs to be logged with llama.cpp that reproduces the issue as simply as possible.

Does someone have a few lines of python to reproduce the problem? Ideally with informative output of nvidia-smi before, during and after the bug occurrence.

EDIT: Sorry, I see that the OP did provide some code. What happens if you do a:

del llm

before returning? My python-fu isn't that strong, but I suspect you need to explicitly destroy the object to destroy the reference to the code running on the GPU. Of course this looks horribly inefficient as every time the method is being called the model needs to be reloaded on the GPU.

[nidhishs]

I’ll make a minimal example and update you.

[iactix]

By now I am doing

llm.reset()
llm.set_cache(None)
llm = None
del llm
llm = None

Changes nothing. RAM goes down, VRAM keeps up.

ggml-org/llama.cpp#1456

[nidhishs]

This snippet should do the job:

from llama_cpp import Llama
import gc
import os

def measure_resources(func):
    def get_ram_usage(pid):
        ram = os.popen(f'pmap {pid} | tail -1').read().strip()
        return ram.split(' ')[-1]
    
    def get_gpu_usage(pid):
        gpu = os.popen(f'nvidia-smi --query-compute-apps=pid,used_memory --format=csv | grep {pid}').read().strip()
        return gpu.split(', ')[-1] if gpu else '0 MiB'

    def wrapper():
        pid = os.getpid()
        print('pid:', pid)
        pre_ram, pre_gpu = get_ram_usage(pid), get_gpu_usage(pid)
        print('pre_ram:', pre_ram, 'pre_gpu:', pre_gpu)
        func()
        post_ram, post_gpu = get_ram_usage(pid), get_gpu_usage(pid)
        print('post_ram:', post_ram, 'post_gpu:', post_gpu)

    return wrapper

@measure_resources
def generate_text():
    llm = Llama(model_path='./weights/oasst-30b.bin', n_gpu_layers=40)
    del llm
    gc.collect()

if __name__ == '__main__':
    generate_text()

Output:

pid: 13121
pre_ram: 720676K pre_gpu: 0 MiB
llama.cpp: loading model from ./weights/oasst-30b.bin
llama_model_load_internal: format     = ggjt v2 (latest)
llama_model_load_internal: n_vocab    = 32016
llama_model_load_internal: n_ctx      = 512
llama_model_load_internal: n_embd     = 6656
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 52
llama_model_load_internal: n_layer    = 60
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 9 (mostly Q5_1)
llama_model_load_internal: n_ff       = 17920
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 30B
llama_model_load_internal: ggml ctx size = 135.75 KB
llama_model_load_internal: mem required  = 25573.29 MB (+ 3124.00 MB per state)
llama_model_load_internal: [cublas] offloading 40 layers to GPU
llama_model_load_internal: [cublas] total VRAM used: 15307 MB
llama_init_from_file: kv self size  =  780.00 MB
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 | 
post_ram: 25209048K post_gpu: 16074 MiB

Interestingly RAM usage also doesn't go down until process is terminated? Perhaps I'm missing something.

[gjmulder]

Good repro! I patched it to use $MODEL.

You probably want to append to the bug llama.cpp/issues/1456, but they may ask which llama.cpp commit, which is non-obvious from the:

$ pip install --force-reinstall --ignore-installed --no-cache-dir --verbose llama-cpp-python

$ git diff repro*py
diff --git a/repro.py b/repro2.py
index 03f365a..d628b1d 100644
--- a/repro.py
+++ b/repro2.py
@@ -24,7 +24,7 @@ def measure_resources(func):
 
 @measure_resources
 def generate_text():
-    llm = Llama(model_path='./weights/oasst-30b.bin', n_gpu_layers=40)
+    llm = Llama(model_path=os.environ.get("MODEL"), n_gpu_layers=40)
     del llm
     gc.collect()
 
$ pip list | grep llama-cpp-python
llama-cpp-python              0.1.52

$ python ./repro2.py 
pid: 1750642
pre_ram: 718468K pre_gpu: 0 MiB
llama.cpp: loading model from /data/llama/7B/ggml-model-f16.bin
llama_model_load_internal: format     = ggjt v1 (pre #1405)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 512
llama_model_load_internal: n_embd     = 4096
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 32
llama_model_load_internal: n_layer    = 32
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 1 (mostly F16)
llama_model_load_internal: n_ff       = 11008
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size =  72.75 KB
llama_model_load_internal: mem required  = 14645.09 MB (+ 1026.00 MB per state)
llama_model_load_internal: [cublas] offloading 32 layers to GPU
llama_model_load_internal: [cublas] offloading output layer to GPU
llama_model_load_internal: [cublas] total VRAM used: 12602 MB
llama_init_from_file: kv self size  =  256.00 MB
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 | 
post_ram: 26360776K post_gpu: 13236 MiB

[nidhishs]

oobabooga/text-generation-webui/#2087 was able to fix the RAM not being released. Have we already integrated this change?

[iactix]

oobabooga/text-generation-webui/#2087 was able to fix the RAM not being released. Have we already integrated this change?

It seems to me they only fixed the RAM not VRAM? RAM was already freed when I tried this. Anyway it clearly seems to be a llama.cpp problem and I don't know how this can be open for a week or so. The fix must be a single line or something. Over there. As a workaround, I could imagine wrapping the model useage entirely into a thread and killing that after use could work to force it to free everything like it does when the pyhton script exits, but I have not tested it. I doubt it would be a fix llama-cpp-python could actually implement though.