I would argue that under the right conditions it could negatively impact physical growth. As social creatures, humans can experience a lot of stress from isolation. I think it well understood that intense prolonged stress can have many negative impacts on the body. It could negatively impact physical development in that way.
I think an argument could also be made that isolation might result in poorer nutrition than having a stable support system of friends, relatives, etc., which could then potentially stunt physical growth.
I would argue that under the right conditions it could negatively impact physical growth. As social creatures, humans can experience a lot of stress from isolation. I think it well understood that intense prolonged stress can have many negative impacts on the body. It could negatively impact physical development in that way.
I think an argument could also be made that isolation might result in poorer nutrition than having a stable support system of friends, relatives, etc., which could then potentially stunt physical growth.