I agree with the sentiment and even own some opposite-gender clothing that I love. However, some clothing is cut certain ways to flatter stereotypical shapes so you may want to be aware if a shirt if cut to accentuate breasts or cut for certain shape hips.

Just pay a bit more attention to the cut of the clothing and you'll be fine though; everybody is shaped a little different so it may work for you. My aunt wore mens jeans basically her whole life, she said they were cheaper and more durable.