I think it’s a matter of perspective.

In your time example, I see it as time going forward, not up. It’s progressing. When you scroll with the steam deck you’re “progressing” through files, which are normally organized in a descending, downward way.

I had a coworker who would say “scroll up” or “scroll down” depending on if he wanted the document/content to move up or down.