@[email protected] to [email protected]English • 7 months agoLLM ASICs on USB sticks?lemmy.mlimagemessage-square14fedilinkarrow-up127arrow-down16file-textcross-posted to: aicompanions
arrow-up121arrow-down1imageLLM ASICs on USB sticks?lemmy.ml@[email protected] to [email protected]English • 7 months agomessage-square14fedilinkfile-textcross-posted to: aicompanions
Source: nostr https://snort.social/nevent1qqsg9c49el0uvn262eq8j3ukqx5jvxzrgcvajcxp23dgru3acfsjqdgzyprqcf0xst760qet2tglytfay2e3wmvh9asdehpjztkceyh0s5r9cqcyqqqqqqgt7uh3n Paper: https://arxiv.org/abs/2406.02528
minus-squareSmorty [she/her]linkfedilink1•3 months agoSomething similar to this already kinda exists on HF with the 1.58 bit quantisation which seem to get very similar performance to the original Llama 3 8B model. That’s essentially a two bit quanitsation with reasonable performance!
minus-square@[email protected]linkfedilinkEnglish2•3 months agoThat’s really interesting, gonna try out how well it runs
Something similar to this already kinda exists on HF with the 1.58 bit quantisation which seem to get very similar performance to the original Llama 3 8B model. That’s essentially a two bit quanitsation with reasonable performance!
That’s really interesting, gonna try out how well it runs