This also happens to me occasionally. I can't find any pattern to explain it, though.
Sometimes I lose the buff when I leave TW, sometimes when I die for the first time in TW, sometimes when I leave arena and sometimes when I change channels (which can also make me gain love buffs as lilveviruchi mentioned).