Are you sure that you multiply each and every modifier? In the first game, at least, i'm fairly certain that it added all of your modifiers together and then applied them depending on whether the total was positive or negative.