End users aren’t the customer.

Yup, and there are a lot less bakers around now that machines do most of it.

No, I think there are fewer bakers.

Is this a grammer thing? I’m fairly certain I can use “a lot less”.

You can use “less” when it’s a non-discrete plurality, such as water or sand (ignoring the technical fact that these can now be observed as discrete components below the macroscopic level) or money. It’s vastly more messy to have 1.78 bakers, and their families get really upset about it, so it’s safer just to use “fewer.”