Dual 3060s are an option. LLMs can be split across GPUs reasonably well.
3090s used to be like $700 used, but ironically they’ve gone up in price. I got mine for around $800 awhile ago, and stuffed it into 10L PC.
Some people buy used P40s. There are rumors of a 24GB Arc B580. Also, AMD Strix Halo APU laptops/mini PCs can host it quite well, with the right software setup… I might buy an ITX board if anyone ever makes one.
Also, there are 12GB/6GB VRAM distillations too, but 24GB is a huge intelligence step-up.
That’s minimum a $1000 GPU if you go red team or $1500 for green.
Dual 3060s are an option. LLMs can be split across GPUs reasonably well.
3090s used to be like $700 used, but ironically they’ve gone up in price. I got mine for around $800 awhile ago, and stuffed it into 10L PC.
Some people buy used P40s. There are rumors of a 24GB Arc B580. Also, AMD Strix Halo APU laptops/mini PCs can host it quite well, with the right software setup… I might buy an ITX board if anyone ever makes one.
Also, there are 12GB/6GB VRAM distillations too, but 24GB is a huge intelligence step-up.
Totally forgot the 3090 had 24GB. It’s definitely still enthusiast territory though.
For sure.
The 14B distillation is still quite good, and usable on like 10GB GPUs. Maybe 8 with the right settings.