I don't think the problem is necessarily that some challenges require magic to solve as it is that there are few challenges that magic can't solve. At least personally, I'd be fine with the non-casters occasionally standing back and letting the casters do their thing if the roles were reversed about as frequently.