publication croisée depuis : https://lemmy.world/post/1474932
Hi there.
I wanted to run LLMs locally on my server (for better privacy), and was wondering if:
- I could use Intel ARC/AMD GPUs - these are often less expensive and AMD has open source drivers, which is something I like.
- If a PCIe x4 Gen 3 slot would be enough (it’s an x16 slot with x4 speeds) - this is an important consideration.
- Would 8GB of RAM (in the GPU, I believe it’s called VRAM?) be enough?
I’m looking at language models to train on my Reddit and Lemmy content, in an aim to make it write like me (and maybe even better than me? Who knows). I don’t quite know which models I will train, or how I will do so (I certainly won’t be writing anything from scratch), but I was wondering; with the explosion of FOSS AI models, maybe something like this would be possible with the hardware constraints I mentioned above?
Does the speed of the connection between the GPU and the CPU really matter in such applications?
Thanks!
Yup; hopefully there are some advances in the training space, but I’d guess that having large quantities of VRAM is always going to be necessary in some capacity for training specifically.
I hope some GPU manufacturer starts allowing removable RAMs. 4 x 8 GB DDR5 might not be too bad given PCIe speeds aren’t a bottleneck. If I could upgrade the RAM to 64 GB later, I’m ready to give $10k at 3080 level perf. Intel ARC people I hope you are already doing this!
I don’t know anything about GPU design but expandable VRAM is a really interesting idea. Feels too consumer friendly for Nvidia and maybe even AMD though.