Very serious. Your personal amount of usage means nothing at all in this conversation.
It is entirely about tokens per watt. The amount of energy the memory operations involve scale incredibly well when people are accessing the same object in memory simultaneously. Last I looked it was around a 10x difference for the same models efficiency.
If you want me to be your personal search engine you’ll need to wait a bit, im making dinner right now and would rather look for the articles on my desktop.
Very serious. Your personal amount of usage means nothing at all in this conversation. It is entirely about tokens per watt. The amount of energy the memory operations involve scale incredibly well when people are accessing the same object in memory simultaneously. Last I looked it was around a 10x difference for the same models efficiency.
Hold up. Are you talking about caching? Because if you are… yeah. That has nothing to do with the model and everything to do with the service layer around the model. The same service layers can be - and have been - implemented in tools like Lemonade Server, llama.cpp, Ollama, etc.
And I really do want to know your sources.
Mine say GPT 5.5 is probably using quite a lot more than 0.34 Wh per query (0.34 Wh is what Sam Altman claimed for the then-current version of GPT in June of 2025, but he hasn’t released numbers since then and no one has done an independent analysis). With Claude, an independent estimate from last year pegged Sonnet at 0.8 Wh for a short prompt, 2.8 Wh for a medium one, and 5.5 Wh for a long one. Current numbers are, again, almost certainly much higher. And just for fun, there’s DeepSeek (which I’ve never used and never would use), with the reasoning-tuned DeepSeek-R1 hitting a whopping 29 Wh for a complex query.
Meanwhile, small, open models are probably in the 0.07 - 0.2 range, depending on the model, the hardware it’s running on, and the nature of the query. Of course, there are much weightier open models too, with ones like Llama 3.1 405B using about 9 Wh for a medium-length prompt. On the other hand… who is going to run that on their local machine?
Look… If I’m wrong, and using local models the way I do - sparingly and infrequently - really does consume more electricity than using Claude Code, I want to know. I have no problem whatsoever with eschewing AI models entirely, since I despise all of them. But given how tight-lipped OpenAI and Anthropic are about energy consumption per average prompt, and what independent analyses have estimated, I am highly skeptical that they are acting as some sort of paragons of environmental stewardship.
Not talking about caching (though there would be some decent memory savings due to that on general platforms like ChatGPT and tools like Codex). I am talking about large batch sizes, which are concurrent requests all accessing the same memory at the same time. The model is loaded once onto the GPU(s) and then many simultaneous requests can read that memory at the same time. When those requests are all processing their responses simultaneously, the energy per token drops off a cliff.
And yes, running a smaller model would generally take less power, but thats not really a fair comparison. Small models just wont give you the same results as larger ones. You need to compare it apples to apples. If you want to compare your local Qwen model running on your laptop, you compare those numbers to larger systems supplying that same qwen model to thousands of people. Just because we are comparing cloud services to local doesn’t automatically mean GPT 5.6 vs Qwen 3.6 27B. There are plenty of cloud AI providers running all sorts of models and sizes.
As for one of the articles I learned alot of this from originally, this is one I recommend going through. It really goes deep into the whole topic: https://arxiv.org/html/2601.22076v1
Very serious. Your personal amount of usage means nothing at all in this conversation. It is entirely about tokens per watt. The amount of energy the memory operations involve scale incredibly well when people are accessing the same object in memory simultaneously. Last I looked it was around a 10x difference for the same models efficiency.
If you want me to be your personal search engine you’ll need to wait a bit, im making dinner right now and would rather look for the articles on my desktop.
Hold up. Are you talking about caching? Because if you are… yeah. That has nothing to do with the model and everything to do with the service layer around the model. The same service layers can be - and have been - implemented in tools like Lemonade Server, llama.cpp, Ollama, etc.
And I really do want to know your sources.
Mine say GPT 5.5 is probably using quite a lot more than 0.34 Wh per query (0.34 Wh is what Sam Altman claimed for the then-current version of GPT in June of 2025, but he hasn’t released numbers since then and no one has done an independent analysis). With Claude, an independent estimate from last year pegged Sonnet at 0.8 Wh for a short prompt, 2.8 Wh for a medium one, and 5.5 Wh for a long one. Current numbers are, again, almost certainly much higher. And just for fun, there’s DeepSeek (which I’ve never used and never would use), with the reasoning-tuned DeepSeek-R1 hitting a whopping 29 Wh for a complex query.
Meanwhile, small, open models are probably in the 0.07 - 0.2 range, depending on the model, the hardware it’s running on, and the nature of the query. Of course, there are much weightier open models too, with ones like Llama 3.1 405B using about 9 Wh for a medium-length prompt. On the other hand… who is going to run that on their local machine?
Look… If I’m wrong, and using local models the way I do - sparingly and infrequently - really does consume more electricity than using Claude Code, I want to know. I have no problem whatsoever with eschewing AI models entirely, since I despise all of them. But given how tight-lipped OpenAI and Anthropic are about energy consumption per average prompt, and what independent analyses have estimated, I am highly skeptical that they are acting as some sort of paragons of environmental stewardship.
Not talking about caching (though there would be some decent memory savings due to that on general platforms like ChatGPT and tools like Codex). I am talking about large batch sizes, which are concurrent requests all accessing the same memory at the same time. The model is loaded once onto the GPU(s) and then many simultaneous requests can read that memory at the same time. When those requests are all processing their responses simultaneously, the energy per token drops off a cliff.
And yes, running a smaller model would generally take less power, but thats not really a fair comparison. Small models just wont give you the same results as larger ones. You need to compare it apples to apples. If you want to compare your local Qwen model running on your laptop, you compare those numbers to larger systems supplying that same qwen model to thousands of people. Just because we are comparing cloud services to local doesn’t automatically mean GPT 5.6 vs Qwen 3.6 27B. There are plenty of cloud AI providers running all sorts of models and sizes.
As for one of the articles I learned alot of this from originally, this is one I recommend going through. It really goes deep into the whole topic: https://arxiv.org/html/2601.22076v1