I feel that the downsides people bring up can be mitigated pretty well. It’s been mentioned elsewhere in this thread that the majority of ecological impact is from strays, so you can negate most of that harm just by having your cat fixed. Danger to the cat itself; use judgment about how safe it is where you live, how adapted the cat is to that sort of environment, and consistently keep it indoors at night. Being responsible doesn’t mean always going along with what other people think or want.

If it was a dog, different story since they can be much more destructive to other people’s pets and property, but a cat I think this is a personal choice.