I’ve been back’n’forth for a while whether to invest in a second 5060Ti to be able to go dual GPU on my main workstation. The target has always been to be able to run Qwen 3.6 27B at a usable model quant, kv-cache quant and context size.

Today I tried out llama.cpp with the rpc option. I sort of knew at the back of my head that there were projects doing cross-computer inference but I didn’t know it was built into llama.cpp.

Sure, this requires that you have another machine on your local network (latency matters) with a GPU - but in my case I actually have two. One server that has a 12GB A2000 and one Bazzite box with an AMD something also with 12GB VRAM.

I just compiled the latest llama.cpp HEAD with the rpc option enabled on my headless server and launched my main llama-server pointing to it. With the 12GB of the A2000 in the mix I’m now able to comfortably run the 27B model Q4_K_M, Q8_0 kv-cache and 140000 context size, and I’ve still not maxed out the available VRAM.

PP at ~400 (a bit low, will see if that can be tuned*) and TG at 25 (MTP and ngram enabled) makes this perfectly usable. That’s the token/second during inference I would’ve expected from a dual GPU setup in the workstation anyway.

So, if you also happen to have another machine on your network with spare VRAM - try out llama rpc. I have no idea why it took this long for me to do.

*) This is literally my first run with random defaults - tuning starts now