My personal photos were an absolute mess. I had a Google Takeout, an old iPhoto library, a number of CF cards and USB sticks with backups. Part of the mess was that there were overlaps between these piles of photos... I first started out with find/md5sum+shell but quickly switched to #fdupes (https://github.com/adrianlopezroche/fdupes). Very quick and the interactive delete mode is quite helpful.