It turns out these clusters are being used very inefficiently, seeing how Qwen 2.5 was trained with a fraction of the GPUs and is clobbering models from much larger clusters.
One could say Facebook, OpenAI, X and such are “hoarding” H100s but are not pressured to utilize them efficiently since they are so GPU unconstrained.
Google is an interesting case, as Gemini is getting better quickly, but they presumably use much more efficient/cheap TPUs to train.
![[Kevin Comics] Zuckerberg AI](https://sh.itjust.works/pictrs/image/2896b29d-18c5-4302-89fa-c5c9cb8b8aa7.png){kind=link}
It turns out these clusters are being used very inefficiently, seeing how Qwen 2.5 was trained with a fraction of the GPUs and is clobbering models from much larger clusters.
One could say Facebook, OpenAI, X and such are “hoarding” H100s but are not pressured to utilize them efficiently since they are so GPU unconstrained.
Google is an interesting case, as Gemini is getting better quickly, but they presumably use much more efficient/cheap TPUs to train.