You’re right. It will build up. But, it also can be diluted, used up by plants/microbes and will dissipate into the atmosphere pretty quickly. Few months to a year for sure, provided fresh urea isn’t being supplemented to the area.
If you need to speed it up, spray vinegar then grind up charcoal and turn it into the soil. Finally, wet w water from the hose.