Endermen are killed by water so I've made sure my buildings have moats as a defense. They're annoying, but they're too weak to be scary.
Having friendly mods not despawn has made farming a possibility, but has caused some problems. I needed wool but there were no sheep around, but when I went exploring I came across this:
