Dwarves appear to be much more heavily concentrated in the north than anywhere else, and there are not only easily a dozen ethnic groups, there's also dozens of sentient species contributing to the total.
Think about how many trolls and goblins and squidy things there are.
If anything, 500 million seems like lowballing it.