Using llama.cpp, the entire context gets reprocessed each generation

[ClassicDirt]

When using llama.cpp, the entire prompt gets processed at each generation making things like chat mode unbearably slow. The problem compounds as the context gets larger and larger as well.

Perhaps using interactive mode in the binding might work? Or, maybe more likely, implementing something similar to the prompt fast-forwarding seen here.

[oobabooga]

I agree that this optimization would be beneficial, but it's not obvious to me how to implement it. Maybe someone can shed some light in a PR.

[djaffer]

The llama cpp is very slow.

I'm actually trying to look into this but it'll take some work to figure out how to implement it here as I'm more of a C programmer. What llama.cpp interactive mode does as far as I know is they only compute the new tokens the user provides rather than the entire prompt. Note that they don't have the option to edit prompts like us, so we will have to force a full recomputation if an edit occurs. As a start it may be simpler to first target saving the initial computation for our character prompt so at the very least those several hundred tokens won't be reprocessed every time.

As an aside OpenBlas works with the latest llama-cpp-python abetlen/llama-cpp-python#32 (comment) and in my tests I get a nice 2x improvement in ingestion speed. It's very helpful for long prompts.

[digiwombat]

Not a python guy either, but I might have it baseline working with the new interactive mode inclusion and the speeds are great.

I essentially gutted the llama_model_alternative.py script and used the newly merged interactive mode example.

Problem is, I don't know my way around the ins and outs of webui so I am only managing to dump it into the console as is.

The basics are this:

1. Copy over the high level chat example into llamacpp_model_alternative.py and the common.py classes that it uses into a new file named llamacpp_common.py. Here's a gist with the edited llama alt file.
2. Make functions in the chat example into class functions so they run.
3. Add a llama.cpp specific function to the generate_reply class in text_generation.py (basically just a copy of the current RWKV/llamacpp version. Mine looks like this:

if shared.is_llamacpp:
        for k in ['temperature', 'top_p', 'top_k', 'repetition_penalty']:
            generate_params[k] = generate_state[k]
        generate_params['token_count'] = generate_state['max_new_tokens']
        try:
            output = ""
            if shared.args.no_stream:
                shared.model.input(question)
                reply = shared.model.output()
                print(reply)
                if not shared.is_chat():
                    reply = original_question + apply_extensions(reply, 'output')
                yield formatted_outputs(reply, shared.model_name)
            else:
                if not shared.is_chat():
                    yield formatted_outputs(question, shared.model_name)

                # RWKV has proper streaming, which is very nice.
                # No need to generate 8 tokens at a time.
                shared.model.input(question)
                for reply in shared.model.output():
                    output = original_question + output + reply
                    print(output)
                    if not shared.is_chat():
                        reply = original_question + apply_extensions(reply, 'output')
                    yield formatted_outputs(reply, shared.model_name)

        except Exception:
            traceback.print_exc()
        finally:
            t1 = time.time()
            #original_tokens = len(encode(original_question)[0])
            #new_tokens = len(encode(output)[0]) - original_tokens
            #print(f'Output generated in {(t1-t0):.2f} seconds ({new_tokens/(t1-t0):.2f} tokens/s, {new_tokens} tokens, context {original_tokens})')
            return

5. Add if not shared.is_llamacpp: to line 39-ish and fix indent of chat.py since we don't need to encode for llamacpp in this mode.

That's basically it, I think. That should be outputting things into your console. I don't know the code structure or what it expects well enough to get past that at the moment, sorry for not helping more.

Keep in mind this is using 0.1.26 or 0.1.27 or whatever the latest wheel is from their repo (post-merge of abetlen/llama-cpp-python#15). Hopefully that is enough to get people with more time to dig into the code start. Sorry for not formatting as a pull request, I don't know the code well enough to beat it into shape but I still wanted to contribute.

Progress is currently being made on this issue at abetlen/llama-cpp-python#68, where they are making llama-cpp-python only compute the new tokens if the previous inputs have not been altered.

[Enferlain]

It feels like it uses the old CPU mode even when you're trying to run ggml models. Token speed is like 0.3s and it takes 3 minutes to get a reply that's like 10 words.

Loading llama-13b-ggml-q4_0...
llama.cpp weights detected: models\llama-13b-ggml-q4_0\ggml-model-q4_0.bin

llama.cpp: loading model from models\llama-13b-ggml-q4_0\ggml-model-q4_0.bin
llama_model_load_internal: format     = ggjt v1 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 2048
llama_model_load_internal: n_embd     = 5120
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 40
llama_model_load_internal: n_layer    = 40
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: n_ff       = 13824
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 13B
llama_model_load_internal: ggml ctx size =  73.73 KB
llama_model_load_internal: mem required  = 9807.47 MB (+ 3216.00 MB per state)
llama_init_from_file: kv self size  = 3200.00 MB
AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |

What hardware are you running it on? This is what I get with a 16GB i5-6500, with no AVX512 like you. I have BLAS which helps but this speed for the first generation is similar to what I get with the original llama.cpp.

llama.cpp: loading model from models/llama-13B-ggml/ggml-model-q4_0.bin
llama_model_load_internal: format     = ggjt v1 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 2048
llama_model_load_internal: n_embd     = 5120
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 40
llama_model_load_internal: n_layer    = 40
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: n_ff       = 13824
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 13B
llama_model_load_internal: ggml ctx size =  73.73 KB
llama_model_load_internal: mem required  = 9807.47 MB (+ 3216.00 MB per state)
llama_init_from_file: kv self size  = 3200.00 MB
AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 | 
Loading the extension "gallery"... Ok.
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Output generated in 81.47 seconds (0.49 tokens/s, 40 tokens, context 346, seed 1943500325)

Now if you're talking about continued generations then yes it will take several minutes each time as it has to recompute all the old tokens.

[oobabooga]

I think that the speed issues should be fixed after this d2ea925

See the discussion here abetlen/llama-cpp-python#44

[Answer-is-not-42]

I've managed to make cache work 3 times by typing in input, clicking "send dummy message" then "continue", then it stopped working. It's not consistent, can just decide to reprocess context out of the blue without (apparent to me) changes in what I'm doing.

I've been playing with the updated UI today and I see these inconsistencies too. Most of the time the cache works well but there are occasional times where it does a full regeneration even though the previous text remains unaltered. This happens when I'm only using the Generate button and nothing else.

Below is a particularly bad example with no editing taking place where I would expect a cache hit every time. I think I need to add some debug code and figure out what is actually being sent over to llama-cpp-python.

Output generated in 118.49 seconds (0.47 tokens/s, 56 tokens, context 384, seed 1483172622)
generate cache hit
Output generated in 33.98 seconds (0.97 tokens/s, 33 tokens, context 477, seed 1813301482)
generate cache hit
Output generated in 70.04 seconds (0.99 tokens/s, 69 tokens, context 570, seed 179168234)
Output generated in 130.12 seconds (0.20 tokens/s, 26 tokens, context 657, seed 1809474820)
generate cache hit
Output generated in 54.90 seconds (0.84 tokens/s, 46 tokens, context 742, seed 1525464511)
generate cache hit
Output generated in 50.84 seconds (0.75 tokens/s, 38 tokens, context 840, seed 1646691917)
generate cache hit
Output generated in 70.03 seconds (0.99 tokens/s, 69 tokens, context 917, seed 2034897901)
Output generated in 185.25 seconds (0.30 tokens/s, 56 tokens, context 1024, seed 1520712527)