• brucethemooseEnglish
    2·
    6 days ago

    • Exl3 quantization at 3.4bpw or so, KLD very close to the higher Q3/close to Q4 GGUFs.

    • TabbyAPI as a backend

    • Split each layer, experts are profiled and hot ones are run on GPU during hybrid inference.

    I have a Ryzen 7800 with 128GB DDR5, which is how it fits. The “cold” MoE experts are run on the CPU, but this is very sparse, which is how it stays fast. Batch size of 8192 keeps prompt processing fast even though it is loading experts from RAM for that.

    There’s also a lot of system optimization to get it there, like allocating large pages at boot, running the backend at a high priority, keeping background RAM/VRAM usage minimal so it will fit, sizing the quantization just right and such.

    And this was not possible… a few weeks ago? DeepseekV4 just came out, and exllama got support for hybrid inference very recently. Before that, I could only get 8-12 toks/sec with a big MoE like this.

    • fancy_pants@reddthat.comEnglish
      2·
      5 days ago

      Thank you, this is a lot of new names for me and a great starting point for research. This field changes so fast

      • brucethemooseEnglish
        1·
        5 days ago

        Exllama is old! Ironically. It was around during the Llama2 days, hence the name.

        The paper behind its quantization method is pretty old too, and it had KV cache quantization with hadamard rotation like a year before Google’s TurboQuant paper made it popular. llama.cpp (the engine behind GGUFs) isn’t as cutting edge as people think, unfortunately.

        I’m not disagreeing, I just find it amusing how “oldschool” some of this stuff is now.