I've never seen a hermit do damage to an anemone, but it wouldn't hurt.
In my pico tank, I have a hermit that crawls into my rose bubble and takes any sort of food. I had to feed the crab before I feed the 'nem, so I brought it home to a larger tank. Now it crawls into my rock flower and I still have to feed the crab before the 'nem.
Just have to be aware of what they are doing. We all got to eat.