I think one of the difficulties with the house in Raven Rock is that by the time you find it, you're usually high enough level that you already own a functional house in a more convenient spot. You can get Breezehome lickety-split if you give the main quest even token attention.
It also doesn't help that Solstheim's idea of a friendly welcome is a mouthful of dust and three Ash Spawn throwing a corpse party. Probably turns people off the whole "chasing down the cultists" idea for a while. Personally, I took it as a challenge to throw an even bigger party, only mine had more axes and frantic, primal shrieking. I call it my "war face".