All that means is you haven't proofed your armor against attacks on your spirit.Ah, but if it's done right the ghost couldn't get in the armor in the first place.
It's a Catch-22: improper proofing means enemy spellcasters can suck out your soul, but doing it right means that you can't get the ghost out of the armor.