If it’s what you want and it’s not hurting anyone’s else, go for it.

Some people will judge you for it, but this is true of any choice. Since crocs (with socks) has become pretty common, I’d say socks and jandals are fair game pretty much anywhere so long as it’s practical (probably not acceptable on a building site).