So basically, there is a 6.9% chance to follow in Magi's footsteps.
The probability of losing is .8 on each occurrence.
0.8*0.8*0.8*0.8*0.8*0.8*0.8*0.8*0.8*0.8*0.8*0.8 = .0687
With gambling, it is usually smart to look at your chances of losing your investment rather than identifying the potential gains. There is nearly a 7% chance that you are throwing 12 gold away.
Advanced Transmutation of an Orange through 12 occurrences, there is a 28% chance that you are throwing 60 gold away.
Advanced Transmutation of a Red through 12 occurrences, there is a 69% chance that you are throwing 120 gold away.
You have to be okay with trashing the gold if you're going to gamble with it.
But don't forget that Gem transmutation is a little bit better in each case than for Equipment.
Edit: typo in the Red equation, corrected to be 120 gold versus 150. NumPad jumped right up and hit me officer.