Honestly good for them. US tech CEOs deserve to have their lunch eaten for ducking the industry into stagnation with their short sighted greed.
In one story they’re using PTX on Nvidia H800s. In another they’re on Huawei chips.
Which is it? Are we all just hypothesising?
Not the best on AI/LLM terms, but I assume that training the models was done on Nvidia, while inference (using the model/getting the data from the model) is done on Huawei chips
To add: Training the model is a huge single-cost expense, while inference is a continuous expense.
Wait, so after you train, you don’t need all those fancy Nvidia chips?
They should make one place where there is an overabundance of geo thermal energy and train all models there…
Yes, so R&D and finalizing the model weight is done on NVIDIA GPUs (I guess you need an excessive amount of VRAM).
Inference is probably gonna be offloaded to consumers in the end where the NPU is taking care of the inference cost (See Apple, Qualcomm etc)
Yes. You still need similar ones if you want to run the models really fast, but not nearly the same amount or cost. That’s how people run LLMs on their laptops. You don’t even need a GPU, a multi-core CPU is sufficient, just not very fast at it.
You do need great hardware, but it depends on your use case. If you want the full 671 billion parameter R1, you need to run it on specialized hardware that has enough RAM.
If you want to run R1 on a phone, you could get the 1.5B parameter R1 running as well. But the quality of results and the speed of response diminish significantly depending on the model and the hardware you use.
In Iceland they run their Bitcoin Mining facilities fully on geothermal energy. I wouldn’t be surprised to find the EU exploring there options regarding new data centers built on renewable energy for quite some time. For now it is a lot faster to train the models within existing data centers that already have the hardware while everyone is actively competing.
Meanwhile governments and corporations are trying to pull money out their ass (cutting important programs) to move mountains and create AGI, of which we have no evidence this is the way to accomplish that.
An unknown quantization of R1 is running on the 3rd iteration of outdated 7nm hardware taken from Sophgo’s work with TSMC last year?
Is this meant to be impressive or alarming? Because I’m neither.
I’m not going to parse this shit article. What does interference mean here? Please and thank you.
That’s a very toxic attitude.
Inference is in principle the process of generation of the AI response. So when you run locally and LLM you are using your GPU only for inference.
Yeah, I misread because I’m stupid. Thanks for replying, non-toxic man.
Training: Creating the model
Inference: Using the modelInference? It’s the actual running of the ai when you use it, as opposed to training.
Sorry. I forgot to mention that I’m dumb.
