I just got up from conversation with a couple of older black men, that I said “well I got to go back to work and start cracking the whip.” And it occurred to me then that it was probably a really insensitive stupid thing to say.
Sadly, it hadn’t occurred to me until it’s already said.
Considering this is a term related to animals, it is not particularly offensive. Or at least not the way you think it might be. While there are some phrases that are obviously offensive, pretty much any phrase, regardless how innocent, there will be someone who could construed it as offensive. I doubt this fellow considered it anything but that of work animals.