Well.. as i recall.. to avoid inbreeding you need something along the line of at least 50 individuals. And thats while running a strict breeding program. Even if all 4 of the guys had a child with each of the 2 womens, then it would at most take a couple generations before siblings began to pair. Only way this would have been possible is if very early on other people began to break free of the stone curse.