I’m going to say that somewhere between guys not wanting to wear their work clothes outside of work, and the outfits being a major symbol of workers during the time the IWW were being demonized (the IWW were the "International Workers of The World) baby clothes became a thing.

Overalls and jumpsuits and softer forms of them are easy to put babies… Cartoons even sometimes depicted them with having a poop flap at the back, so men were being told not to wear them, babies were wearing them, and they were being used as a political symbol of international communism and “lowly” blue collar work like mining or steel work.

… however women, would later (within our youth obsessed culture) want to look cute and young, and at other points in time, wanted to be seen as viable and able workers… And women were already in the habit of borrowing fashions from men.

Thus there were cultural rejection factors for men, and cultural attraction factors for women.