As far as I can tell there isn't a numerical way to calculate it -- except that the damage that you do DOES result in malice. Paladins have +aggro spells to keep and hold aggro, which means that other DPS classes need to back off a little to ensure that the paladin keeps aggro. Necros (possibly wizards too, I'm not certain,) have anti-aggro spells. It would be nice if they would add some kind of aggro meter in-game, they're still working things out so maybe they'll add it later. It would be even harder to calculate, too, because many pally spells increase aggro by a percentage while the skill is active, but is that adding to a percent of the base damage or the aggro that has already amassed? And do the values add together... is it 100% + 35% plus another 50% of that total? Or 100% plus 35% plus another 50% of the base 100%? At this point it seems that the percents and numbers are simply to indicate how high the level of malice is -- i.e., if you want to keep aggro, re-cast this spell a lot, and also to let the players know which spells collect aggro (if two paladins are fighting and only one is tank, the non-tank needs to know which ones not to use).
About heal aggro, from my sage-experience it seems to be less than the heal value, however, in AOE situations, if someone is getting "ganged up on" and hasn't dealt damage to the mobs that are hitting (i.e., the person hasn't gotten any aggro except by passing by,) they will mob up on the healer quickly. The same is true of adds or spawns in instances when there isn't someone dealing with them. You get aggro the same way when you AOE heal, and it also seems that if you buff someone to reflect damage, you get the aggro from that reflected damage. However, if there is someone standing by and the healer hasn't hit any of the mobs that are now hitting him/her, the aggro is very easily stolen, so it seems that the heal malice value is pretty low compared to the heal value.
The best way to get through big bosses with a tank is to give them plenty of time to get aggro. Healers that are well-equipped can easily keep the heals going a few extra seconds to ensure the tank doesn't lose aggro. This usually means waiting at least two spell-rounds, to give them time to cool off and be re-cast. That way the tank has a good head-start on aggro collecting (especially important since paladins have low mp and bosses often drain it). Having at least one person ONLY on the spawns is another good tactic, since the healer cant' fight off spawns/heal him/herself AND heal the tank at the same time. World bosses get more complicated because there are so many spawns, so many tanks, etc. Best way to go about that is to make as many raid parties as there are people attacking so that the healer can keep tabs on who needs heals -- the AOE spells and strong spawns means that they're usually not just focusing on the tank, in fact, quite the opposite, since it's often the tanks that can fend for themselves in that situation better than the other classes.
Hope that helps, sorry I can't give you more definite info.
I would add though that when people add something that says "I'm wondering, too" it acts as a bump to push your post back to the top of the list, where more people are likely to read it and comment. So it's not necessarily a bad thing