Don’t apologise, it’s a phrase people use and you weren’t being racist. Euphemism treadmill is slippery by definition, if we’re all mates there’s no need to get too into language policing.