That's the intention of "once you use a weapon, you have to keep it till next turn." Fire your bow, you can't insta-swap for a shield. But if you're firing a bow and an enemy closes the distance, you can swap for a shield and rapier (on your turn). And then when the enemy dies, stow the shield and rapier to attack with the bow again.
And yes, there absolutely should be harder rules against armored casting. As in, you just can't do it unless you get a class feature that lets you cast while armored/using a shield. How would I word this? I dunno. Half casters exist, clerics and druids exist. But someone smart can figure out an elegant solution.