Interesting stuff... I don't think we really disagree. My point is: there is no reason to limits the types of casters; you could easily have an Int Warlock or Sorcerer, or even a Wis-based one (of course, balance is a concern).
You make some interesting points about Innate spells being harder to swap, but other than that... there is nothing stopping a deity from giving a new power every day, for example, or giving you a small list of powers that can never be changed. Both are possible.
There is no reason why there can be a "White Mage" character that can magically heal because he read a formula in an ancient book. Also, he cannot wear medium armor. The things aren't necessarily tied to one another.
Here is another post about the subject, but is concerned with OSR more than 5e.
The idea is that you could learn spells from a book, form a deity, a patron, another wizard's scroll, etc. Doesn't matter - you could create all types of wizards, sorcerers, etc.