I think if Calvin did, he would have been smart enough to not destroy the stones because you’d need to periodically snap again as reducing the population by a % once only turns back the clock and eventually you’ll be back where you were before the snap. And if he had the stones, that attack on him wouldn’t have stood a chance.