Hi all, if you’ve not heard, there’s been a Caveman compression version of Qwen 3.6-27b and Qwen 3.6-35b-3ab.

https://huggingface.co/ProCreations/grug-27b-gguf

https://huggingface.co/ProCreations/grug-35b-v2

https://huggingface.co/ProCreations/grug-35b-v2-gguf

I’ve been playing around with 35B and I am able to run it on my ancient Quadro P1000 4gb at ~10tok/s.

But beyond that, the quality of the reasoning and the token discipline / output is actually higher than a stock, in my opinion.

You can read the benchmarks above; I am also uploading a HTML file here for your consideration.

(Sorry for the Limewire link; I dunno where else to share throw-away files. It’s HTML, though the side by side probably works better if you actually download it)

Anyway…I’m doing more testing right now…but so far, this is a good cook.

Or -

Grug good. Me like.

  • SuspiciousCarrot78@aussie.zoneOPEnglish
    6·
    14 hours ago

    Oh and if anyone is interested in my local settings

      -t 12 ^
  -tb 12 ^
  -ngl 28 ^
  --n-cpu-moe 41 ^
  --no-mmap ^
  --mlock ^
  --cache-type-k q4_0 ^
  --cache-type-v q4_0 ^
  -c 16384 ^
  -b 256 ^
  -ub 128 ^
  --host 0.0.0.0 ^
  --port %PORT% ^
  --ui-mcp-proxy

    Turbo quant is not supported on this GPU (too old),