It's an explicit plot point that he's only resistant to blunt damage due to his rubber in the early parts of the story. It just never ends up mattering because injuries to characters in One Piece only exist to make it more dramatic when they keep fighting, so they never actually hinder anyone.