The human brain contains functionally and anatomically distinct networks for representing semantic information in each sensory modality, and a separate, distributed amodal conceptual network. In this study we examined the spatial organization of visual and amodal semantic functional maps. The pattern of semantic selectivity in these two distinct networks corresponds along the boundary of visual cortex: for visual categories represented posterior to the boundary, the same categories are represented linguistically on the anterior side. These results suggest that these two networks are smoothly joined to form one contiguous map.