Skirmish only increases damage 4 times though so instead of being behind 50% of the time, you're only behind ~25% of the levels, and not by anything nearly as powerful as higher level spell slots. Instead, you're behind by +1d6/attack damage; 3.5 - ~14 damage. Relevant but nothing major.
More relevant earlier than later since it presents a larger percentile fraction of your composite damage, but as you don't have multiple attacks early, you don't need the dip then either so you won't be left behind until when it's not such a major factor anymore. And, of course, the fact that you get to full attack with Skirmish consistently (as opposed to "occasionally" or "never") means you're way, way ahead in damage compared to anyone without the dip.