Depends what you are building them for:

To control a town or city?

In the city, preferably on a site previously occupied by houses of your political opponents

To protect and levy tolls on trade?

By the river or road in question; in some cases you can make a bridge your castle

To project power?

In this case it's just a defensive site where your rangers, cavalry, whatever can rest and resupply before heading out into whatever area you are trying to control

Pick a decent defensive site on the borders of the area - will probably not be near an existing settlement

To protect a town or city?

Probably not necessary because a town worth defending should be walled anyway

As a place of refuge?

Needs to be big enough for population to flee to - and to hold their animals - and also to be close enough to get to.

As pure defence?

Chances are the best defensive site isn't near anything worth defending and an attacking enemy can just bottle you up and then ignore you.

As a showpiece residence?

Has to be somewhere you want to live and where people can get to

As an administrative centre?

Should be close to the population it serves

Most likely though you won't have a perfect site for "a" castle - so you might have several smaller fortifications and one or more main ones.

This is especially true with a heredity nobility who are going to pick up odd bits of land here and there