TheFatKat has already covered the no HP glitched mobs - they have been there since I started
The other issue you mention is a lot more rare than the glitched mobs one, but it has also been around for as long as I can remember.
It seems to be that there are certain spots where the mob can't "see" you.
You hit it and get aggro, but as the mob can't see you it automatically resets & goes back to full health.
The solution to this one is to move away, then come back at a different angle and try again.
If that doesn't work, just try a different mob