Armor doesn't use the Torso slot. It uses the Body slot (as in your whole body). They don't conflict. You can wear the vest under the armor.
EDIT: Of course, what the DM says is what happens. I just posted this in case your DM misunderstood the official rules on equipping magic items. Listen to your DMs, kids!