Implement Together Computer's Red Pajama 3B Base/Chat model

[jimexist]

• announcement
• base model, 3B
• instruct model, 3B
• chat model, 3B

Hopefully this can be blazingly fast!

[jimexist]

Following #1333 (comment) maybe it's a good idea to start at ggml first

[keldenl]

+1 on this

[Green-Sky]

this should be another gptneox model. did someone look into running it? using https://github.com/ggerganov/ggml/tree/master/examples/stablelm

[Green-Sky]

$ bin/stablelm -m ../examples/stablelm/models/RedPajama-INCITE-Base-3B-v1/ggml-model-f16.bin -p "They"
main: seed = 1683366948
stablelm_model_load: loading model from '../examples/stablelm/models/RedPajama-INCITE-Base-3B-v1/ggml-model-f16.bin' - please wait ...
stablelm_model_load: n_vocab = 50432
stablelm_model_load: n_ctx   = 2048
stablelm_model_load: n_embd  = 2560
stablelm_model_load: n_head  = 32
stablelm_model_load: n_layer = 32
stablelm_model_load: n_rot   = 80
stablelm_model_load: ftype   = 1
stablelm_model_load: ggml ctx size = 7376.40 MB
stablelm_model_load: memory_size =   640.00 MB, n_mem = 65536
stablelm_model_load: ................................................ done
stablelm_model_load: model size =  5296.58 MB / num tensors = 388
main: number of tokens in prompt = 1
main: token[0] =   3726, They

They.G A,S.F2., L. A,.B.s G P. A N., to S. nail on the_A A. Stephen A.s. 20, (a- a/f, (x) ofl-in. 4. Stephen.s.,,.s G. su A. A. Stephen A, G. |
in,, G, W., B. and/wins_M. G. A r L. R. sack, P G. Ph.s,/
?/
G and / F, B-R, M.s,^C

Something is not right here.

edit:

RedPajama-INCITE-Base-3B-v1$ sha256sum ggml-model-f16.bin
95afa2ab90d9496a521c2916da0693a9b01d73d3e22ecc7120d5e3d5636ecbb4  ggml-model-f16.bin

[Green-Sky]

since they explicitly state pythia architecture, i tested pythia-70m-deduped, and that worked just fine.

$ bin/stablelm -m ../examples/stablelm/models/pythia-70m-deduped/ggml-model-f16.bin -p "They"
main: seed = 1683373243
stablelm_model_load: loading model from '../examples/stablelm/models/pythia-70m-deduped/ggml-model-f16.bin' - please wait ...
stablelm_model_load: n_vocab = 50304
stablelm_model_load: n_ctx   = 2048
stablelm_model_load: n_embd  = 512
stablelm_model_load: n_head  = 8
stablelm_model_load: n_layer = 6
stablelm_model_load: n_rot   = 16
stablelm_model_load: ftype   = 1
stablelm_model_load: ggml ctx size = 188.42 MB
stablelm_model_load: memory_size =    24.00 MB, n_mem = 12288
stablelm_model_load: ......... done
stablelm_model_load: model size =   134.41 MB / num tensors = 76
main: number of tokens in prompt = 1
main: token[0] =   3726, They

They could help us to find them.

I had the same feeling as my sister, but I felt like I could go to sleep. I felt like the old lady. She was my sister and I had no idea. But I realized it didn't matter.

It had been months and years.

My wife, my sister and my cousin, was really close. I had an intense feeling at the same time. I was so happy to see her. I had never felt anything so happy and so happy before I went to sleep. My mind wandered on the bed. I would sit down again, and I was on my own. I woke up and tried to do something else. I finally started to get up, and she was crying in the bed. She was still in the room, but when she came into the room and asked if she could give me a lift to her, she told me that she could not. I was so happy that she told me that she

[amirza1]

The problem is that stableLM uses the gptneoX with use_parallel_residual=True (so that each block is x + mlp(x) + attn(x)) and RedPajama uses use_parallel_residual=False. I implemented it in https://github.com/amirza1/ggml and will submit a PR

[CarlKenner]

Together Computer already did it here:

https://github.com/togethercomputer/redpajama.cpp/tree/support_redpajama

[keldenl]

I've also quantized for q4_0 -> q5_1 ggml the 3B red pajamas chat model https://huggingface.co/keldenl/RedPajama-INCITE-Chat-3B-v1-GGML, doing the rest and uploading them too soon

[limcheekin]

I've also quantized for q4_0 -> q5_1 ggml the 3B red pajamas chat model https://huggingface.co/keldenl/RedPajama-INCITE-Chat-3B-v1-GGML, doing the rest and uploading them too soon

Thanks for sharing the GGML models. I tried to deploy the q4_0 and q4_2 to AWS lambda, both models hitting the error loading model: missing tok_embeddings.weight. Not sure what missing, the same code works for the OpenLlama-7B model, you can check out the code here at https://github.com/limcheekin/serverless-ml/tree/redpajama-chat-3b-v1/redpajama-chat-3b-v1

Edited: my bad, just got the following answer on the readme:

This will NOT work with llama.cpp as of 5/8/2023. This will ONLY work with the GGML fork in https://github.com/ggerganov/ggml/pull/134, and soon https://github.com/keldenl/gpt-llama.cpp (which uses llama.cpp or ggml).

[keldenl]

@limcheekin yup, your only options currently are this PR gets merged or you use the fork in the ggml repo or if u use my hacky gpt-llama.cpp (which leans on ggml repo, will make it less tacky tonight)

anyways, here's RedPajamas 3B instruct model: https://huggingface.co/keldenl/RedPajama-INCITE-Instruct-3B-v1-GGML/

(i'll be quantizing the 7B models tonight)

[keldenl]

ggml for instruct 7b uploaded: https://huggingface.co/keldenl/RedPajama-INCITE-Instruct-7B-v0.1-GGML
only chat 7b left, i'll upload it later tonight

[ekryski]

@keldenl these probably need to be quantized and re-uploaded now that #1405 landed, which would likely require forking the Red Pajama fork and pulling in upstream llama.cpp changes.

[ekryski]

It looks like the most up to date stuff is here: https://github.com/togethercomputer/redpajama.cpp/tree/support_redpajama/examples/redpajama. However it needs to pull in the upstream quantization changes, so it's a bit of a mess right now.

@ggerganov @slaren @Green-Sky based on other discussions I've seen scattered across other issues/PRs, what might be best is to consolidate ggml and llama.cpp into a common low level llm repo that can support all these different models.

My C/C++ is so bad that I'm not gonna be much help there but happy to help in any other way to make this happen/share info.

[Green-Sky]

consolidate ggml and llama.cpp into a common low level llm repo

somewhat yea. but i feel like either more llm stuff is going to land in the ggml repo, or a separate ggml repo will be needed.

these probably need to be quantized and re-uploaded now

if you use the gpt-neox code example from the ggml repo, not yet since @ggerganov did not yet backport the ggml changes from llama.cpp.