From 1984 to 2023, the median cost of a new home rose 423%, more than double the rate of overall price inflation.