It really depends on what the Lord is trying to accomplish by building his castle. Castles are extremely expensive, so most people won't build unless they have a good reason. They probably want to be close enough to have easy access to the sources of their wealth, as well as serving as a reminder to the peasants as to who is in charge.

Given the tiny holdings you describe, most lords will probably have something more like a fortified McMansion than an actual castle, probably initially built out of wood, with subsequent generations replacing the wood with stone as funding allows.