Hostility chasing away participants

I have come and gone a few times over the last decade. Every time I have left it was because of vague hostility, or some sort of territoriality. I'm doing bug hunting lately, and the same problems are making it much less fun than it normally is. Again, I wonder if the stress is unhealthy and I should quit for a while.