Fighter's not the only one though. If your challenge requires magic to solve, there's a whole slew of classes that can't contribute, and if your challenge doesn't require magic to solve, then it's not a high magic challenge, so sounds to me like the limiting factor is more with high magic challenges than with the whole array of non-magical classes.
Also note I never specified fighter, that's your own inclusion.