I’ve been looking into self-hosting LLMs or stable diffusion models using something like LocalAI and / or Ollama and LibreChat.
Some questions to get a nice discussion going:
- Any of you have experience with this?
- What are your motivations?
- What are you using in terms of hardware?
- Considerations regarding energy efficiency and associated costs?
- What about renting a GPU? Privacy implications?
Quite some AI questions coming up in selfhosted in the last few days…
Here’s some more communities I’m subscribed to:
And a few inactive ones on lemmy.intai.tech
I’m using koboldcpp and ollama. KoboldCpp is really awesome. In terms of hardware it’s an old PC with lots of RAM but no graphics card, so it’s quite slow for me. I occasionally rent a cloud GPU instance on runpod.io Not doing anything fancy, mainly role play, recreational stuff and I occasionally ask it to give me creative ideas for something, translate something or re-word or draft an unimportant text / email.
Have tried coding, summarizing and other stuff, but the performance of current AI isn’t enough for my everyday tasks.
Thanks for the post, super appreciate the posting of other communties. I think this is a great way to grow Lemmy and create discoverability for niche communities, I’ll keep that in mind myself on future opportunities.
Have been using llama.cpp, whisper.cpp, Stable Diffusion for a long while (most often the first one). My “hub” is a collection of bash scripts and a ssh server running.
I typically use LLMs for translation, interactive technical troubleshooting, advice on obscure topics, sometimes coding, sometimes mathematics (though local models are mostly terrible for this), sometimes just talking. Also music generation with ChatMusician.
I use the hardware I already have - a 16GB AMD card (using ROCm) and some DDR5 RAM. ROCm might be tricky to set up for various libraries and inference engines, but then it just works. I don’t rent hardware - don’t want any data to leave my machine.
My use isn’t intensive enough to warrant measuring energy costs.
So you access the models directly via terminal? Is that convenient? Also, do you get satisfying inference speed and quality with a 16GB card?
Mostly via terminal, yeah. It’s convenient when you’re used to it - I am.
Let’s see, my inference speed now is:
- ~60-65 tok/s for a 8B model in Q_5_K/Q6_K (entirely in VRAM);
- ~36 tok/s for a 14B model in Q6_K (entirely in VRAM);
- ~4.5 tok/s for a 35B model in Q5_K_M (16/41 layers in VRAM);
- ~12.5 tok/s for a 8x7B model in Q4_K_M (18/33 layers in VRAM);
- ~4.5 tok/s for a 70B model in Q2_K (44/81 layers in VRAM);
- ~2.5 tok/s for a 70B model in Q3_K_L (28/81 layers in VRAM).
As of quality, I try to avoid quantisation below Q5 or at least Q4. I also don’t see any point in using Q8/f16/f32 - the difference with Q6 is minimal. Other than that, it really depends on the model - for instance, llama-3 8B is smarter than many older 30B+ models.
Thanks! Glad to see the 8x7B performing not too bad - I assume that’s a Mistral model? Also, does the CPU significantly affect inference speed in such a setup, do you know?
If your CPU isn’t ancient, it’s mostly about memory speed. VRAM is very fast, DDR5 RAM is reasonably fast, swap is slow even on a modern SSD.
8x7B is mixtral, yeah.
I’ve installed Ollama on my Gaming Rig (RTX4090 with 128GB ram), M3 MacBook Pro, and M2 MacBook Air. I’m running Open WebUI on my server which can connect to multiple Ollama instances. Open WebUI has it’s own Ollama compatible API which I use for projects. I’ll only boot up my gaming rig if I need to use larger models, otherwise the M3 MacBook Pro can handle most tasks.
Is that 128GB of VRAM? Because normal RAM doesn’t matter unless you want to run the model on the CPU, which is much slower.
That’s 128GB RAM, the GPU has 24GB VRAM. Ollama has gotten pretty smart with resource allocation. Smaller models can fit soley on my VRAM but I can still run larger models on RAM.
Any tips on how to get stable diffusion to do that? I’m running it through Krita’s AI Image Generation plugin, and with my 6GB VRAM and 16GB RAM, the VRAM is quite limited if I want to inpaint larger images, I keep getting ‘out of VRAM’ errors. How do I make it switch to RAM when VRAM is full? Or with Jan for that matter, how can I get it to partially use RAM and partially VRAM so I can get it to run models larger than 7B?
I have https://github.com/oobabooga/text-generation-webui for LMS and https://github.com/lllyasviel/stable-diffusion-webui-forge for stable diffusion models.
They just run on 8g vram so I turn one off to use the other, lol.
Both work well for my needs
I tried out ollama. It was trivially easy to set up.
Stable diffusion is a bit more work, but any power user should be able to figure it out.
Picked up an AMD instinct mi25 to try and do just that. Can get easy-diffusion working after some cussing and voodoo. Cannot get rocm to do ANY llm of any kind, feels like a waste of video ram
Also have a tesla p4 that runs most text-to-image models rather well, but have been unsuccessful at any llm either, even oobabooga can’t seem to run on it.
Have given up because the software stack keeps advancing and leaving my hardware behind. I don’t have $3000 for an a100 or $1300 for an mi100 sooo… until the models can run on older/less powerful hardware, I’m probably sitting out of this game. Even though I’d love to be elbow deep in this one.
Sounds like a rather frustrating journey for you.
It has been. I started in this because I liked picking up kick ass enterprise hardware really cheap and playing around with what it can do. Used enterprise hardware is so damn expensive now, it’s cheaper and easier to do everything with consumer products and use the rx6700 in my gaming rig. Just don’t want that running llms and always on.
I have a Asus laptop with a GTX 1660 ti with 6GB VRAM. I use Jan for LLMs, only the 7B models or lower are small enough for my hardware though, and Krita with the AI Image Generation plugin for image generation, most things work in it, except it fails with an ‘out of VRAM’ error if I try to inpaint an area more than about 1/8 of my canvas size.
Currently seeing this up at work. Ollama with danswer as a rag / frontend. On a bunch of Nvidia L4 /L40 on kubernets
Is pretty plug and play honestly.
I run ollama on my laptop in a VM with open web UI. It works great and I have plenty of models to choose from.
I recently was playing around with TTS and it is pretty solid as well. I am thinking about taking the smaller phi models and throwing it onto my pine64 quartz64 for a portable AI assistant while traveling. The only potential problem is the time it takes to process.
Ive been playing with the nixified.ai project, which packages two web interfaces for LLMs and image generation. Im also looking into Tabby.ml for code assistant as well. I haven’t gotten deep, but these all look like promising options for utilitizing a server’s hardware but offering the functionality across the network.
You want https://tabby.tabbyml.com/ instead of tabby.ml
