And then at some point we went from that to starting mass labor projects around aligning rocks with these celestials.

My theory is that organized slavery arose when some elders had a mental break when someone nudged their stone circle for the Xth time, so they decided to go big with it.