The swap from spears was part of a move away from Greek-style hoplite formations. The Romans began to split their troops into smaller units for maneuvering in rough terrain, and the change to swords as a primary weapon was part of that. Roman troops excelled in close-combat against Greek-style hoplites (and later Macedonian-style phalangites when they could close, but considering phalangites are carrying around a massive pike, that’s less surprising), and their performance against typically-spear-wielding barbarians and Eastern troops was generally excellent as well.

The exceptionally large shield used by the Romans, and their generally aggressive tactics (as opposed to the more defensive formations of hoplite-style armies), made the shortsword a deadlier weapon than the spear. Essentially, you’ve got a small guy with a big shield pushing as close to you as he can so he can shiv you - if you have a spear (or a longer sword, for that matter), he’s in the more advantageous position, as he can more easily control the movement of your spear with his shield than you can control the movement of his sword, which he can comfortably withdraw and handle behind his shield.