It depends a lot on your setup and there is no golden rule. Just print small section of your models and try it out. It can be that your outer contours are more accurate than hole dimensions. You can use some slicer tuning (like hole horizontal expansion) or just modify 3D model. I prefer learning my printer errors and then just tuning my 3D models