Apparently being "sent back until your task is done" involves a good bleach-based detergent. (Also, a beard trim and some hair straightener.)

Honestly, he might have been "Gandalf the White" all along. Walking the earth for "three hundred lives of men," far from good washing machines, I'm sure his robes got rather dingy.