There is evidence of armed conflict between neanderthals and early humans. I don't think it's clear whether the warfare or simply being outcompeted was the primary cause of neanderthal extinction.
As for the interbreeding, well that implies a very ugly backstory.
That's not how it works at all. It's not like a videogame where you snowball off your early position and get richer faster and faster. Civilisations and societies rise and fall, and who is richest changes from time to time. It is widely regarded that Africa is the most resource rich continent in the world - google it.
Why so resistant to the possibility that beings that have inherently worse abilities than other beings might end up worse off?