I think it is behavioral isolation. Performing a mating dance wouldn’t be abnormal behavior on its own, but the key takeaway from the question is that the meaning of this behavior is indecipherable to other members of the species. Without being able to communicate their desire to mate, the individual will fail to reproduce and this new dance won’t be passed on to the next generation.

The motion of the dance would carry no meaning to others of the same species, and this individual will be isolated by that behavior as a result.