While I perfectly agree with you, there is a limit. I wouldn't class having the same strength as the strongest power lifter as being supernaturally strong though.
Suppose there was the same physical disparity between our average strength fencer and the strongman as there was between a 12 year old and a MMA fighter at their peak - then it doesn't just start being an issue of strength, but speed as well and adding supernatural resilience so that the fighter can apply that strength with no issues just becomes unfair.
If we increased the size of this disparity even further then it stops being a contest and more a matter of how long the weaker person can survive or is forced to retreat.
I was thinking more of his Band of Hawks days, where he's running around with an oversized sword although not at the size of the Dragonslayer.
Edit: Thinking about Berserk some more, there's a perfect scene in the TV series that illustrates Galloglaich's and my points. When Griffith first meets and duels Guts, he's by far the superior swordsman and defeats Gats handily.
It's not until Gats does something completely off the wall that illustrates his strength and determination (not to mention absolutely bat-crap insane levels of aggression) that he even comes close to beating Griffith.
I've only seen the TV series - based on what you've said, I may have to get hold of the movie.