this format is no longer supported

[apcameron]

The latest merge breaks the old models.
We need the instructions on how to update them

./examples/chat.sh
main: build = 529 (b9fd7ee)
main: seed = 1683842865
llama.cpp: loading model from ./models/7B/ggml-model-q4_0.bin
llama_model_load_internal: format = ggjt v1 (pre #1405)
llama_model_load_internal: n_vocab = 32000
llama_model_load_internal: n_ctx = 512
llama_model_load_internal: n_embd = 4096
llama_model_load_internal: n_mult = 256
llama_model_load_internal: n_head = 32
llama_model_load_internal: n_layer = 32
llama_model_load_internal: n_rot = 128
llama_model_load_internal: ftype = 2 (mostly Q4_0)
llama_model_load_internal: n_ff = 11008
llama_model_load_internal: n_parts = 1
llama_model_load_internal: model size = 7B
error loading model: this format is no longer supported (see #1305)
llama_init_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model './models/7B/ggml-model-q4_0.bin'
main: error: unable to load model

[moh21amed]

same, right now just check out to older commit.

[redthing1]

Which older commit?

[moh21amed]

write this in llama.cpp directory from command line
git checkout cf348a6
it worked for me

[javea7171]

Competing forks and ecosystem fracture are looming. Needs a common file format and a plan for breaking changes. Without this, the community's time and money will be wasted.

[philpax]

I'm a maintainer of llm (a Rust version of llama.cpp and other models), and we're not entirely sure how we're going to handle this. We'd like to maintain compatibility with the previous models, but it doesn't seem like that's an option at all if we update to the latest version of GGML.

Is it possible to restore the old quantisation functions as an option for loading? I understand the desire to keep iterating, but there's an extant ecosystem of hundreds of models out there, and this is going to cause a lot of unnecessary disruption 😦

This will also cause a lot of unnecessary confusion for users given the use of the same names for the quantisation methods - it's very unclear if a q4_1 model released today will be compatible with the old or new quantisation methods.

[LostRuins]

Hi @philpax, I think this is because the llama converter is still tagging new models as file version 1 instead of version 2. See: #1405 (comment)

[philpax]

Hi @philpax, I think this is because the llama converter is still tagging new models as file version 1 instead of version 2. See: #1405 (comment)

Good to know - I haven't tackled it yet - but even still, what are your plans for loading both v1 and v2? Will you keep both versions of ggml and switch based on the version?

[LostRuins]

For KoboldCPP I have already prepared a GGML shim that contains refactored version of all the old quantizers

https://github.com/LostRuins/koboldcpp/blob/concedo_experimental/ggml_v2.c

Once this is ironed out I'll probably swap the dequant functions as necessary based on file version if possible.

I plan to maintain 100% backward compatibility with every single past ggml model.

[batmac]

I'm just here to confirm that it would be really nice to continue using all the quantized models out there! (for instance, all the "ggml" models on huggingface) :)

[ggerganov]

Wouldn't it be better instead of distributing the quantized models, to distribute the F16 models.
The F16 format is unlikely to change in the future, while the quantized formats are likely to change again in the future.

If the F16 models are distributed, then your software can decide when to re-generate a quantized model and when to pick one directly from the cache.

[philpax]

For us, personally, we're not distributing anything - we're supporting what's out there, to make sure that users can use a model, no matter how old it is. Users prefer downloading the quantised models because they're smaller and ready to go - the f16 models are often not accessible to them, being up to four times as large and requiring additional work.

I'm likely to take the same workaround @LostRuins has and bundle an older ggml (or a part of it) to keep supporting old models. For better or worse, those hundreds of models have proliferated to tens of thousands of users in the few months this has been active - we need to be able to either keep using those models, or to be able to update those models in a seamless and convenient way.

[ZenKemuri]

Sorry, I'm a newbie and yesterday was my fisrt day using LLaMA.cpp, the thing is that it looks like I decided to start in a very difficult moment. I ignore the majority of the terms and I got lost when I have to understand how to install these technologies on my PC.
Now... my question is, what if I want to try a model right now? What model should I download? Because I have the same issue, I did everything right! But it tells me that it can'ts load the model :(
So, what should I do? Do I have to quantize anything?

Sorry, I'm really interested but now I am more lost than yesterday

• Ubuntu 22.04
• i7-5410
• 8Gb RAM
• 4Gb VRAM GTM840-M

[grencez]

Wouldn't it be better instead of distributing the quantized models, to distribute the F16 models.

There are so many fine-tunes floating around. If someone wants to try a variety of them, grabbing the quantized versions directly makes a lot of sense storage-wise. Maybe if LoRAs loaded faster, more of those would be shared instead of weights, but iiuc we need to read a f16 model for good LoRA results... pretty slow.

Even with a good reason to distribute the quantized models, I think changing format like this is still reasonable because it's so easy to requantize. The popular models on huggingface were requantized within a day.

Anyway, the point of this bug is to give better advice in the error message, right? Would it make sense to link to docs on how quantize an f16 model?