I have all the villagers blocked inside their houses at the moment – I was thinking about taking on an Outpost and I'd surely get struck with the bad voodoo during battle – so there's little chance a raid would kill the population. There's like a dozen or so iron golems (they just show up randomly to get in the way or trample crops
), so there's that additional protection as well. As long as the raiders spawn outside the village you should be OK.