Open weight models are not far behind and good enough to get software developers jobless, but there is a huge difference in what you can do with 1M context and 500token/sec or 60k context and 40token/sec.
Sure! End the AI Bubble and i’ll go ahead and buy a bunch of H200’s.
But for real, if you structure your tasks well, you can get by for most things with local models, and for a small business in IT, investing in dedicated hardware you own is a smart move. The employer of an acquaintance of mine bought some inference hardware for his 5 engineer business, and they rely mostly on that. Around 80% of all tasks are done by a GLM5.3 model, and while they currently still use Claude for some things, the percentage of external inference has dropped dramatically in the last months, because people have adapted on how to structure tasks for the local model.
Noone can explain to me how Claude, OpenAI and the NeoClouds plan to become profitable businesses when you can reduce inference costs by 90% by getting your own inference - and it’s not even that hard to set up. if you already have IT staff for your inhouse stuff, they will probably manage.
Someone will come up with a decent accelerator eventually. Right now we’re in the pre-soundblaster days where audio sythesis is being done by the CPU but for neural networks.
We’re repurposing graphics hardware, these are primitive times.
I have to disagree here. We are on the absolute edge of current physics regarding miniaturization - without a massive breakthrough in quantum physics on how we can use subatomic particles for transistors we are nearing the maximum we can shrink these circuits. These are already optimized for exactly this kind of math - a Vera Rubin will never be able to render a game. Also, the mathematical basis (a ton of matrix multiplication) does not give much hope that we can replace all this math with something simpler. Maybe a radical redesign taking a completely different approach can help, but i wouldn’t count on that - it would mean reinventing the whole damn thing on a completely new base.
I can’t really see any explanation when i take the above into account.
Open weight models are not far behind and good enough to get software developers jobless, but there is a huge difference in what you can do with 1M context and 500token/sec or 60k context and 40token/sec.
Sure! End the AI Bubble and i’ll go ahead and buy a bunch of H200’s.
But for real, if you structure your tasks well, you can get by for most things with local models, and for a small business in IT, investing in dedicated hardware you own is a smart move. The employer of an acquaintance of mine bought some inference hardware for his 5 engineer business, and they rely mostly on that. Around 80% of all tasks are done by a GLM5.3 model, and while they currently still use Claude for some things, the percentage of external inference has dropped dramatically in the last months, because people have adapted on how to structure tasks for the local model.
Noone can explain to me how Claude, OpenAI and the NeoClouds plan to become profitable businesses when you can reduce inference costs by 90% by getting your own inference - and it’s not even that hard to set up. if you already have IT staff for your inhouse stuff, they will probably manage.
Someone will come up with a decent accelerator eventually. Right now we’re in the pre-soundblaster days where audio sythesis is being done by the CPU but for neural networks.
We’re repurposing graphics hardware, these are primitive times.
Also, there, I explained it for you.
I have to disagree here. We are on the absolute edge of current physics regarding miniaturization - without a massive breakthrough in quantum physics on how we can use subatomic particles for transistors we are nearing the maximum we can shrink these circuits. These are already optimized for exactly this kind of math - a Vera Rubin will never be able to render a game. Also, the mathematical basis (a ton of matrix multiplication) does not give much hope that we can replace all this math with something simpler. Maybe a radical redesign taking a completely different approach can help, but i wouldn’t count on that - it would mean reinventing the whole damn thing on a completely new base.
I can’t really see any explanation when i take the above into account.