The CUDA moat is pretty deep, but the primitive are starting to solidify and almost no one uses CUDA directly. Increasingly popular libraries are going multi-backend (thanks Apple silicon).

My guess is that as soon as cheap accelerators with LARGE memory banks hit the market, the libraries will support whatever API those need and CUDA dominance will be essentially shattered forever.

But we are not there yet because making good numerical hardware is fucking hard.