I think you've got it backwards - it's not that all Ogres are use for manual labor and thus they become big and strong, it's that all Ogres are big and strong and thus usually used for manual labor. Some ogres are laborers, some are gladiators, some are guardians - the only important thing is being big and strong (technically big and tough). If you were used for manual labor and didn't become a huge brute, then you weren't an Ogre.