I tell my clients it comes from prison as a sign you were looking for sex.

Close and possibly true for some, but apparently it was more just that they weren’t allowed to have belts to hold up their pants. When people got released, they continued to wear them loose because they were comfortable. The trend then spread to hip hop and other cultures in the 90s.