No, but you do have to worry about finding the one you want.
If there are 50 arrows in your quiver, each with a different spell, how do you find the one you want? Did you label them? If so, are you going to stop in combat and flip through your quiver trying to find the right arrow?
As a DM, I might rule that finding a
specific arrow from your quiver takes longer than merely grabbing
any arrow, limiting you to one shot a round or possibly less. That would balance ammo pretty easily; sure you have a stored spell for every occasion, but you're going to need a couple rounds' preparation time before you can use it.
(I know this isn't RAW, but it seems an entirely sensible houserule.)