You aren't wielding it at all. You're just carrying it with you. You aren't making attacks with it, you're just running around with a rowboat. And as others have noted, you can actually move faster running while carrying the ship than you can while rowing it in the water, so logically it wouldn't make sense for it to do less damage.
The real question is, why in god's name did they decide getting hit by a rowboat should deal 4d10 damage in the first place?