Not to be cold and scientific, but if:

A growing body of scientific evidence shows that infertile men are more prone to disease and tend to die younger than fertile men. The incidences of cancer and cardiovascular disease are also higher among infertile men. And beyond fertility, quality sperm can also protect the health of children, and even grandchildren, by reducing the risk of passing on lifestyle-influenced genetic changes that can result in a host of health problems, from obesity to neurodevelopmental disorders. An infertile man could even pass on his infertility to his offspring during medically assisted fertilization.

As a population, like every other species, we got here because the species passed on the traits that helped pass on traits. I feel like passing down infertility over and over is going to eventually cause sterile offspring.