I mean, I would argue that a challenge where half the party doesn't participate would automatically disqualify said challenge as meaningful, since half the party isn't being engaged. You have half a meaningful challenge, because even if it's awesome fun for the half that get to engage, the other half will associate no meaning with the challenge, because to them it may as well have not existed in the first place.