also remember that castles tend to be lagging indicators.

Some family of Barons have been barons for umpteen generations and are deeply linked to their traditional homes....they may even share a name with the town/village that has grown up around their castle...but with some good marriages, better investments, the friendship of certain king etc....suddenly they are now dukes...sure they now hold the amount of land that comes with being a duke but they are still likely to keep their main residence at the same location even if there are now "better" places for it.

plus there are any number of reasons to build castles. A good garrison castle that can't be bypassed (because then the garrison could cut off the invaders supply lines) should probably be near the boarders along natural invasion routes (which will probably correlate nicely to trade routes but not always) for example.

Also castles tend to need supplies and services. Doubly true if they are also a luxury manse. So people tend to see such a place as a good place to set up a business. Now those business owners become a market in and of themselves so more people come....so older castles tend to attract a town.

but the short answer to your question is all of the above....different castles will have different reasons.