For me it’s the result of lasting trauma from when I was bullied as a child for wearing “short shorts”. Not that I had a choice in the matter because my mom was buying all my clothes at the time and wasn’t about to replace a bunch of perfectly good (in her opinion) shorts. My only recourse was to just wear jeans all the time from then on until I was able to buy my own clothes but by then I was more comfortable in jeans all the time.