Just realized that ALT+D on a #fishshell displays the visited directories, so yo can jump back and forth with `prevd <number>` and `nextd <number>`.
Previously I used `dirh` command or `ALT+←→` for moving through the history. I still do if it's a very recent dir.