I feel like you must be.
It doesn't matter if they can wear metal armor. They will not wear metal armor.
Fighter: "Here, wear this half-plate because you're proficient."
Druid: "No. I will not put that on. It is unnatural for man to be clad in metal."
Whether or not he physically can wear it is irrelevant, because he won't put it on.