Or just bring an empty bottle through and fill it up at the tap/water fountain?

Some people think tap water tastes bad

I wouldn’t drink tap water outside of continental Europe. Maybe the original OP is simply in a third world country like the US.

The tap water in Canberra, Australia is the tastiest I've tried out of the ~20-50 municipalities I've sampled in Australia, Western and Southern Europe, the US, China, Taiwan and Vietnam.

It tasted better in Canberra before the 2013 bushfire, back when it was filtered through pine needles, before they removed all the pine farms