BeefAndPoultry@lemmus.org to LocalLLaMA@sh.itjust.worksEnglish · 1 month agounsloth/DeepSeek-V4-Flash-0731-GGUF · Hugging Facehuggingface.coexternal-linkmessage-square8linkfedilinkarrow-up126arrow-down11file-text
arrow-up125arrow-down1external-linkunsloth/DeepSeek-V4-Flash-0731-GGUF · Hugging Facehuggingface.coBeefAndPoultry@lemmus.org to LocalLLaMA@sh.itjust.worksEnglish · 1 month agomessage-square8linkfedilinkfile-text
minus-squarehummingbirdlinkfedilinkEnglisharrow-up6·1 month agoHuh the unsloth Q4_K_XL variant running with 5-6 t/s on a 16GB 9070 XT, 128 GB RAM and an NVME ssd. This is shockingly usable 🤯
minus-squareDacoTacolinkfedilinkEnglisharrow-up3·1 month agoDid some more testing on it? This isnt a small model and i havent found the space to save it on my machines yet haha
minus-squareBeefAndPoultry@lemmus.orgOPlinkfedilinkEnglisharrow-up3·1 month agomake sure that holds up with large context, you might need to step down to Q3 (which I’ve heard is still good for this model, many people are even using IQ2)
Huh the unsloth Q4_K_XL variant running with 5-6 t/s on a 16GB 9070 XT, 128 GB RAM and an NVME ssd. This is shockingly usable 🤯
Did some more testing on it? This isnt a small model and i havent found the space to save it on my machines yet haha
make sure that holds up with large context, you might need to step down to Q3 (which I’ve heard is still good for this model, many people are even using IQ2)