Watch / play that again. You obviously missed a lot, like how the only alternative is genocide the other way. Seriously, if the choice is between a total genocide of all humans, or a total genocide of random race X that I feel sympathy for, I sure will kill every last one of race X. Because at the end of the day I might be willing to sacrifice a lot of humans for peace and future cooperation, but I will not doom the human race to extinction.
Besides, that situation was at least to 90% the Quarian's own fault.