I’ve been having trouble with this too.

So far my best assessment is that the problem stems from the Chromecast not playing well with my router. Resetting the Wi-Fi on my router tends to fix the problem, but it’s seriously annoying.

Separately, I also have issues with the play/pause not working when casting and some other weird interface behaviors sometimes. But all in all, it’s worth complaining, but it still works better than everything else.