Dawnguard has some interesting new mechanics, especially for vampires, but the plot is kind of meh. Dragonborn is a lot of fun. Hearthfire is very skippable, but is interesting if you like collecting a lot of crap and then organizing it in a big house you designed yourself.
As far as your characters, no you would be starting fresh. Mods as well (RIP space core).
As far as differences go, SE doesnt crash as much on my system, but i dont know if thats a personal thing or actually something they improved in the SE.