Drew, feel free to to build all the way up against my castle. If it's not inside stone walls it's not important. (This goes for gargoyle too!)
Oh, but the castle does have a courtyard on the south side. The random number generator prototype is there, and it's also where I make cake because the ditch makes it easy to milk cows. So leave that alone please
It's in a hollow surrounded by mountains, so feel free to build on the mountains. Everything else is fair game, I would enjoy it very much if the area was more populated.
All the castle's mining operations are on the north side so don't worry about hitting anything underground, either.