llama : enable FA by default and disable it per-layer

[ggerganov]

See the discussion starting here: #9991 (comment) and the proposed solution here: #9991 (comment).

Additionally, switch to F32 precision for the K*Q matrix multiplication by default.

Marking this as good first issue as an opportunity for new contributors, but also it is kind of high priority, so we should probably implement this in a day or two if there is no progress. @slaren or @JohannesGaessler in case you already started to work on it, fill free to assign to the issue and finish it.

[slaren]

I will work on this as part of the changes to the model loader than I am working on. This is more complicated than it looks like, determining which layer will run the KV run on, then testing if the operation is supported, then updating all the session writing code to support per-layer v-trans would be very hard to do for new contributors without experience on the code base.

Additionally, switch to F32 precision for the K*Q matrix multiplication by default.

This can already be done separately.

[Dampfinchen]

With partial offloading, FA slows down generation speed significantly, though. So I wonder if it's a good idea to make FA the default, considering how the killer feature of llama.cpp is the fast CPU offloading.

I can provide benchmarks, if needed.

[JohannesGaessler]

Despite user reports I have never been able to reproduce and investigate the issue where FlashAttention is detrimental with partial offloading.

[Dampfinchen]

Despite user reports I have never been able to reproduce and investigate the issue where FlashAttention is detrimental with partial offloading.

I was investigating this further and I could not reproduce it using llama-bench. Token generation and prompt processing were both faster using llama-bench which goes completely against my real world experience. The token generation was 5.5 token/s with 20 layers offloaded for Mistral Nemo q4_k_s. Something is wrong here.

However, I noted something curious. Even increasing the prompt size from the default 512 to 8096 using the command -p, the token generation speed would not change using llama-bench. This doesn't make sense, as in real world experience the token generation is significantly faster when using a smaller prompt compared to a bigger one. So I've tried llama.cpp server.

With llama-server, this were my results:

 ./llama-server -m "D:\KI\LLMs\Mistral-Nemo-Instruct-2407-Q4_K_S.gguf" -t 6 -c 8096 -fa -ngl 20 -b 512 --host 127.0.0.1 --port 8080

prompt eval time =   11302.73 ms /  4716 tokens (    2.40 ms per token,   417.24 tokens per second)
       eval time =   41852.62 ms /   128 tokens (  326.97 ms per token,     3.06 tokens per second)
      total time =   53155.35 ms /  4844 tokens

 ./llama-server -m "D:\KI\LLMs\Mistral-Nemo-Instruct-2407-Q4_K_S.gguf" -t 6 -c 8096 -ngl 20 -b 512 --host 127.0.0.1 --port 8080

prompt eval time =   12661.91 ms /  4764 tokens (    2.66 ms per token,   376.25 tokens per second)
       eval time =   30581.22 ms /   128 tokens (  238.92 ms per token,     4.19 tokens per second)
      total time =   43243.14 ms /  4892 tokens
srv  update_slots: all slots are idle

Like I have expected, there is a drastic slowdown in terms of token generation. If you were not able to reproduce this, perhaps you were using llama-bench as well? Could you try sending a prompt with the size of 8096 tokens to llama-server and see if FA slows down token generation speed for you as well?

If by then you are still not able to reproduce the issue then perhaps the slowdown may be related to GPU architecture since I'm using Turing.

Theoretically I should be able to reproduce the issue using llama-cli as well, I may test this later today.

[slaren]

There is too much noise in the server/cli performance measurements to be used to draw any conclusions from it, that's one of the reasons llama-bench exists in the first place.

[Dampfinchen]

There is too much noise in the server/cli performance measurements to be used to draw any conclusions from it, that's one of the reasons llama-bench exists in the first place.

Shouldn't real world performance count more in this case? Something seems to be off with the llama-bench results. I'm seeing and experiencing the slower token generation in all of my use cases, regardless what noise there may be and thus I always turn FA off when using partial offloading.

Wouldn't you agree that 5.5 token/s on a 8096 context prompt using a 12B model (Nemo in this case) seems unrealistic on an RTX 2060 6 GB with just 20/41 layers offloaded?

If a benchmark is that far off of real world results then in my opinion, there's not much worth to it. Given the results, I suspect llama-bench may be hard coded to only use a context of 512 for token generation, regardless of the prompt size. It may very well be that the slower token generation with FA and partial offloading is an issue that only appears when using higher context sizes, in that case the results of llama-bench would not suffice to investigate this issue properly.