You'd be a mediocre archer at best, it's up to you whether the utility of being a straight classed Bard is worth that to you.

You're tying yourself to a handcrossbow with this build, so depending what your table is like that could hamper things for you in a number of ways.