Or just bring an empty bottle through and fill it up at the tap/water fountain?

Some people think tap water tastes bad

I wouldn’t drink tap water outside of continental Europe. Maybe the original OP is simply in a third world country like the US.