I have pants/trousers/leggings in my closet that range in size from M to XL. When you put them together, they are the same physical size.
In contrast, my karate gi pants are a size 4, based solely on my height. There is no gender-specific version. I've been wearing that size 4 since 2008. I know if I got another pair of gi pants, size 4 will fit. A size 5 is for a taller person, a size 3 is for a shorter person.
Women's clothing sizes are a black pit of madness.