4 rounds of combat in an adventuring day sounds kind of ridiculous.
I could see maybe 4 rounds in a single combat, especially if everybody unloads, but in a whole day seems kind of extreme levels of 10 minute adventuring day. I would say that we probably usually have 4-10 rounds in a single combat, and in a day that involves combats, there are usually several [Most days do not involve combat]. I definitely understand an argument that short rests might as well not exist so Fighter doesn't get to alpha strike more than Sorcerer, but Fighter isn't actually relying on being able to recharge her burst abilities on a short rest to outperform the long rest Sorcerer, only that there's 7 or more rounds of combat in a day where there's a fight.
As far as being a melee centered rogue, it's not just the caster showing you up:
As mentioned, even a basic fighter build [any type, PAM & GWM] is going to downrange 204 average potential damage on the opening/action surge rounds and 102 on a regular turn. Shooty Rogue, on the round they Assassinate [first round], outputs 20d6+2d10+15 [107] [or some combination thereof, since most of the damage is coming from Sneak Attack] on the first round of combat and drops to 10d6+1d10+15 [61] on each round thereafter. This isn't favorable at all. Without Assassinate, Rogue doesn't even get the damage spike on turn 1 [I don't know what Swashbuckler gets, though].
IMO, expecting the use of cantrips for damage is only really acceptable in the calculation if the alpha strike and follow-on rates are actually exceeding the every-turn always-active floor for the noncaster. Given that this isn't true, I think that casters just need the potential in their low end slots to grow with level.
[as a side note, without twinning spells the burst damage out of a spellcaster isn't even that high]
If we took, for example, Fireball [which is common and good when it becomes available but is obsolete not too long afterwords] and scaled it as it used to be [1d6/caster level], then it would do 20d6 [80] damage at 20th level. This would be pretty acceptable for a 20th level blast effect, and alleviate the problem of high-level casters having 6 turns of useful damage output [which still for the most part doesn't actually exceed noncasters] before dropping off to irrelevance in that regard and mostly being useful for teleporting places.