too slow?

[manageseverin]

I'm using it under Windwos 11 with alpaca 7B
Ok, it's great overall, but I have a native cpp version (chat.exe) and it's running 2 times faster than your docker version.
Also, how to use the API ? I saw in docker something like 127.0.0.1:35272 - "GET /chat/5fe89704-c7ca-4a67-9ec2-f267689b0ffe/question?prompt=No%2C+it%27s+actually+14 HTTP/1.1" 200 OK
But where to look for proper API documentation ?

[mokahless]

Ubuntu 22. using 30B

While a significant amount of this issue appears to be CPU processing, I am watching resource usage and noticing every time it finishes an answer, it unloads the model from ram, and so for every question I ask it, it has to read the entire thing from disk into ram again. For my system, this wastes about 15 seconds every question. This portion seems to scale with the threads you give it, too. The same attempt with a quarter of the threads took ~25 seconds to move from disk to ram.

Seems to me this could be sped up by keeping the model in ram? Is that possible or is there something more complex going on that can't be improved upon?

[y12studio]

@manageseverin

Try this docs URL. Serge - Swagger UI http://localhost:8008/api/docs

Very easy to use, it works. But yes, very slow for me, too (30B model).
Takes about several minutes till the words begin to appear after I enter a prompt. But then the words appear quite after another.

edit: Looking at RAM consumption it seems the model is indeed unloaded after every response.
System: Windows 10 64-bit, Intel Core i7 8700K @ 3.70GHz, 32,0GB Dual-Channel DDR4 @ 1600MHz

[manageseverin]

On alpaca 7B model, it takes about a minute for the answer to start to appear. Win 11, Ryzen 5600G/16GB RAM
I thought that it depends on the history size it need to feed back, but no - even for the first time (when there is no history) it also takes the same time.

[magicmars35]

@nsarrazin Hi Nathan, do you think it's possible to load the model in RAM, and keep it in ram, as long as the user querries are submitted ?

[nsarrazin]

Yes it's on the list!

[alph4b3th]

hey, it's really slow! amd epyc 16gb ram, and a lot of delay. more than a minute to load the model to ram. (SSD)

[alph4b3th]

wtf? I have a powerful server! how heavy is this?

[futurepr0n]

I am also fairly beefy specs - I find it slow to run but I am guessing its due to the issue pointed out offloading the model. If we could keep it persistent it would work best, but I could see the reason for not wanting that maybe that its tricky to manage multiple sessions if you implement it right now? I don't know - maybe we could have a superuser option if we have the available ram to keep it loaded full time?

[alph4b3th]

the problem is not serge, it's in llama.cpp. Something doesn't work well with docker, I saw 'npx dalai serve' run and the model responded in 3-5 seconds.. with docker here on my server, it took between 18-60min to initialize and load to ram, and 8 minutes for the model to finish its relatively small response.

[Mattssn]

I am playing with the 30B Model, and I am having the same thing, I am running this on Docker on a pretty beefy box, but getting pretty slow response times. Something to maybe have a variable to keep the model alive for 10 minutes then shut it down for inactivity, tbh, my server really never passes like 14gb of memory usage, an option to keep it loaded as long as the docker container is running might be cool too, I know you said you are working on it, just wanted to give my feedback :)

[voarsh2]

Quite unusable to constantly need to load/unload 4GB's in RAM. Not everyone is on SSD either, so then you have to contend with waiting on disks as well to load in RAM for every chat submission.

I installed it outside of docker and did not get different results from those already mentioned above. What is it? Because I've seen some people running alpaca 7B and it loads and responds in seconds, and on my machine which is a powerful computer for servers, even the 7B is extremely slow, even consuming 6 cores (I upgraded from intel xeon to amd epyc which reduced the answer to 8 minutes and the load to 18 minutes)

Similar experience.
I've tried 7B and 30. I've fed it 50GB's of RAM, giving it 32 cores of CPU (32 x Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz (2 Sockets)) and on a 48 x Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (2 Sockets) - always takes several minutes to load the model back into memory after each response. Realistically, it takes a good 5-10 minutes between each simple response and even longer for typing out multi-line responses. I have a Ryzen 5x (12 x AMD Ryzen 5 3600X 6-Core Processor (1 Socket)) system, that doesn't meet the RAM requirement (a kubernetes node with existing workload taking up available RAM) so I won't bother even trying - the 2 CPU E5-2697 v2 system is on PAR with the Ryzen CPU system. I am not sure why the default threads are 4, increasing the threads for the model doesn't seemingly make responses any faster, but grind the CPU to starvation....

I have an RTX 260 GPU that I can't use for this project... a shame. Would much rather use a GPU for this type of task, even modern day CPU's struggle with this project.

Given there's no multi-server support/workers, I can't make use of the Kubernetes deployment beyond single node compute........ (there's no multi-server worker logic)

--- edit:
I did test on AMD Ryzen 5 3600X 6-Core - not much faster (but faster at printing the text out word by word. 🤷) by any means (2019 CPU vs 2013 CPU, although they are comparable).

[alph4b3th]

I installed it outside of docker and did not get different results from those already mentioned above. What is it? Because I've seen some people running alpaca 7B and it loads and responds in seconds, and on my machine which is a powerful computer for servers, even the 7B is extremely slow, even consuming 6 cores (I upgraded from intel xeon to amd epyc which reduced the answer to 8 minutes and the load to 18 minutes)

[alph4b3th]

I discovered that the problem is in the compilation of the new version of llama.cpp, in which the parameters passed by the compiler are making the software slower, and that an older version like: https://github.com/nomic-ai/gpt4all
it works faster because it doesn't have these optimizations

[johncadengo]

@voarsh2 you mentioned the default threads are 4. Where is this located? Is there a way to change it?