On Local Photo Management and the Command Line

Picasa and iPhoto

Picasa and iPhoto were great apps. Both were free. Both allowed you to manage your photos locally and both allowed you to take pictures with photo cameras, or your phone, and sync them then you got home. Over time our phones synced via the cloud to these apps.

We lost the habit of getting home and ingesting photos because everything was done automatically. We took pictures and they appeared in Picasa and iCloud and we didn't think about it too much.

This was the gateway habit that led to some of us having cloud first photo libraries rather than local ones. One of the factors that led to this is that Picasa synced automatically to the cloud, as did iPhoto.

This was excellent because for years our photographs were stored locally and in the cloud. That is, until I libraries grew and became too big for laptop hard drives. At this point we had to choose. Get a laptop with a larger hard drive, delete photos, or spend more on iCloud.At 3 CHF per month spending more on iCloud was an easy choice.

It's later, when the library grew beyond 200GB that Google One became an interesting proposition. You could get two terabytes for 100 CHF per year. That head room is a huge luxury, for as long as you can backup your photos from the cloud back to a local volume.

The issue is that you can't, and you couldn't. It's only recently that I really managed to export my photos from Google Photos and Flickr, and after two weeks of experimenting and learning.

The Issue

Cloud services are great for synching all your photos and videos, as well as all the photos you get from whatsapp, the screenshots and more.They're not good when it comes to re-organising files.

If friends and family share photos via Whatsapp, Signal or you download videos from TikTok or Flickr, they're all combined into your own photos. This makes a lot of noise twice. The first time in iCloud and Google Photos but the second time in your whatsapp history.

If people share photos and videos whatsapp downloads them to its own backup, as well as to your own photo gallery if it allows you to, which I recommend for one reason. Whatsapp has a nasty memory of taking 100MB or more per chat. This noise is from photos, videos, pdfs, gifs and more. You might have a copy in Google Photos, in Apple Photos and potentially Immich, Photoprism and other photo clouds.

Hard to tidy

Google Photos, Apple Photos, Immich and Photoprism are great at automatic cataloguing but not at helping you tidy up the mess they help you create. For a start Immich and Apple photos make a tremendous mess of your photos files and hierarchy if you give them free reign. You go from a neatly organised hierarchy to a machine friendly mess that you need to clean up if you choose to move away from them.

With Apple Photos and Google Photos I find it excruciatingly hard to "spring clean" when storage gets low. With iPhoto I noticed that files are almost immediately backed up to iCloud so that if you migrate to Immich and Photoprism you download an entire library, every time immich or Photoprism crash and need to be repopulated. This often takes a day of keeping the phone's screen on. That's why having a local library is key and why kDrive is a great tool and a better solution

The Local Advantage

With kDrive, as with Google Photos, Photoprism, Immich, iPhoto and others you save your photo to a cloud, but unlike with them, with kDrive you have a hierarchical folder structure that you can download and work on via command line tools for batch operations, or visually for manual tidying tasks.

Exiftool

If your files are fresh from Google Takeout, the immich folder structure or other you can use a command line prompt to reorganise everything chronologically and more.

Jdupe

With Jdupe you can look for duplicates automatically. With Immich I noticed that I had 27,000 duplicates to sort through. In some cases they're triplicates and in other cases the duplicates are thumbnail duplicates. To do this sorting, manually, with the Immich tool would take weeks or months. With Jdupe it takes a few seconds to a few hours depending on how many duplicates there are.

rsync

With rsync you can transfer files between volumes with ease and convenience. The computer does the work in the background, backing up to a local drive, and a remote drive.

The Tailscale Caveat

If you're synching gigabytes of files use the local ip address, rather than tailscale because tailscale will throttle you after a certain amount of data transfer, I suspect. It's also a lot faster to do things locally. If you do sync remotely sync it locally first, and then move the drive to the remote location.

Visual Sorting and Find

While waiting for rsync to complete certain jobs I went through libraries manually and noticed patterns. I asked Gemini to create a command to help move webp, png and mp4 files with one pattern from my photo library to a secondary photo library that I can sort through at another time. In one instance that removed 130 gigabytes of noise.

The Motivating Push

I abandoned iCloud as my Single Source of Truth for my photos when my photos reached more than 200 GB and shifted towards Google Photos. With two gigabytes of storage I enjoyed the luxurious feeling. I enjoyed it until I saw that I could get 6TB for 67 CHF from Infomaniak and that's when I spent a long time migrating off of Google Photos.

They make it very hard because you can't just download a chronological list of folders and files as you can with kDrive.

Almost a Terabyte to Sort

My Apple, Google, and Flick libraries came to almost a terabyte of data, most of it duplicates. Sorting through it by hand would take months. Using the tools above, once I had a workflow prepared, with the tools listed above took days. Now my library is 370-390gb.

And Finally

27,000 Duplicates in Immich

I tried ingesting from mobile phones and an old immich library but in so doing I ended up with 27,000 dulicate pairs that I would have to sort through by hand. This task would take months. By removing all the duplicates, before ingesting into Immich I will save weeks of tedious work.

JDupe and Peace of Mind

My iCloud library hasn't been the single source of truth for years, due to the 200gb limit. For a while Google Photos was, until I downgraded the plan, and then it became a former single source of truth. Now I hope that Flickr will have filled many of the gaps. On a drive or two I have old iPhotolibraries.

If required I can open the package, extract the original. Run exiftool to create a chronological library, and repeat until all my libraries are consolidated, and then I can import them to my main photo library, and ingest them to Immich and Photoprism

Conclusion

With command line tools you can consolidate photo libraries from multiple sources into a single source of truth, and move on. By maintaining this single source of truth, and backing it up to kDrive, Google Drive or even iCloud you ensure that it is complete, and easy for immich, Photoprism, or some other tool to ingest.

Migrating from Google Photos to Immich Via Google Takeout

In an ideal world Google Photos, iCloud Photos, MyCloud (Swisscom) and other photo backup solutions would back up your photos into a directory structure that you can access and download from with ease and convenience. Unfortunately non of them want to offer that. That is why finding a workflow to get data out is worthwhile.

Remember, a cloud solution, where you can't get media files back out, is not a backup solution. It's just a synching solution. That's why Immich and kDrive are interesting alternatives.

The Problem

If you're like me you have been taking videos and photos for decades and in that time you have taken gigabytes worth of photos and terabytes worth of video. If we save everything to a hard drive then we're fine but because we're in the age of laptops and mobile phones we're backing up to the cloud, and applications are offloading our data from MacOS, iOS and Android. The result is that we no longer have local copies of files and that's where Google Takeout comes in.

Google Takeout

Google Takeout is a tool provided by Google One to export data from Google services to have a local backup. A few years ago when I first considered leaving Google One for kDrive, because that solution was cheaper I found that exporting all my data would be time consuming.

The reason for this is that I had over a hundred zip files to download from Google Drive and that would take hours, if not days to do. Instead, at the time, I was able to backup Google Photos directly to Infomaniak kDrive. With kDrive I could download the files automatically via the kDrive desktop app. I won't elaborate any further because that solution is no longer available.

Takeout Options

Google Takeout gives you the option of downloading 1GB, 2GB, 10GB, 50GB zip files and more. If you have 360 GB of files, as I do, the most convenient solution is to download the 50GB files as you only have eight files to download. With the 2gb version I would have had 186 files to download. It takes half a day or more for the files to be ready.

Unzipping and Metadata

Safari will want to automatically "open" the zip files but I chose not to have this option because I want to keep the zips. I downloaded all eight files, and before doing anything else I backed up the eight files to a second disk as a backup. I then unzipped all eight zips. This takes time so let it work overnight. This is a good time to work on something else.

Folders and Json

Google Takeout creates directories for each album but it also creates albums per year. This means that plenty of files are duplicated, at the very least.

My natural instinct, as a media asset manager was to see this file organisation and tidy up. Don't make this mistake if you're going to use Immich CLI or Exif tool. The mistake is not critical but if you want to save time then it is not required.

The JSON files that are created by Google Takeout expect the folder structure that it creates. If you tidy things up then Exif tool will look for files and not find them because it's looking in "Google Takeout 5" whereas you have everything in "Google Takeout". The result is that exif will fail to update exif data for thousands of photos.

Getting AI Help

Using the Exif tool might be daunting, and that's where Google Gemini or another AI solution can help. Sometimes you can read the fabulous manual but feel confused. With Gemini, Le Chat by Mistral, or other you can ask for help and it will.

Of course, with AI, it might say "Use this prompt" and it might be wrong. That's why it's good to test on a small folder first, see if there are any errors. If you see there are errors you can fix them. When things work well, then you can try in a second and third directory.

The Limitation

With event specific folders that span a single folder this approach worked fine but it's when I was exporting "photos from 2010" that I noticed that it was looking for the folder structure that I had habitually destroyed as a media asset manager.

That's why you keep the zip files. With the zip files intact you can extract the files and folders and start again, this time knowing that the command prompt is the right one and you can let Exif tool work in the background.

Double Checking

When Exiftool is done there are three ways to see if the operation was a success. The first is the summary after running the command. The second is to check individual files to see if the creation date is correct and the first test is to upload to immich and see if the files are uploaded with the correct date and time. When these three steps are complete you can scale up.

Populating Immich

The Fun part is when you start populating Immich. It is gratifying to see images sorted by date and time and for people to be recognised, and for us to see if we remember the names of friends from 15-20 years ago. It's also fun to see how mobile phones and photo cameras have improved over the years. It's also nice not to be trapped by iCloud or Google Photos anymore.

Have a Test Instance or a Test Account

I recommend having a test instance of Immich or at the very least a test account. If something goes wrong it's better to see it go wrong on a test instance rather than in production. I'd rather make sure all photos load into the right year, before seeing thousands of photos added to the day the Google Takeout archive was created.

Things to Watch Out For

• Duplicates: If you edited the rotation, adjusted histograms or anything else you might have three or four copies of the same image showing up.

• Creation Date: With some photos I found that they are sorted by the creation date for the zip, rather than their original creation date. In these cases it's worth checking to see if they are duplicates or if you need to correct the date and time information.

• Keep the Zips: If you make a mistake you can start over

• Keep the file and folder structure: The JSON files are optimised for Google Takeout Folder structures.

• Test Small and then scale up

And Finally

When I hit the 200 GB limit many years ago I upgraded to the 2TB plan and I was happy with it, but I didn't use all of that space so I was paying for more than I needed. As many people have said before me, if iCloud, Google One and other solutions had more tiers then we could upgrade one small step at a time, rather than catapulting from 200 GB to Terabytes.

Another point is that with larger file export sizes breaking out of Google Photos is almost easy, in contrast to before. When you had too download hundreds of zip files, unzip them, and then re-consolidate everything you were speaking of a huge investment in time. Now, within a day or two, you can be out of Google Photos and into Immich.

One of my key reservations about using cloud synching tools as a backup solution is that it was very hard to get data out, whereas now it is considerably easier.

Upon completion of this process, if I tidy up my e-mails I could revert to Google's free tier.

Initial release of ExifPlugin for Foswiki: reads EXIF data from images

https://foswiki.org/Extensions/ExifPlugin

#foswiki #wiki #opensource #perl #ExifTools #exif

@pixouls both #ExifTools and #ImageMagick can indeed be used for that.
While my #gemini post (gemini://fixato.org/gemini-tips.gmi) focused specifically on GPS metadata, Delightly Linux has a post aimed at stripping any metadata from images
But if you want to do that directly on your phone, you'd either have to find an app that wraps around it, or have access to a console such as #Termux to run the commands from your command-line.

#ObscuraCam might also be a useful app on Android phones; it claims to also strip identifiable metadata (note: I've personally never verified this claim), as well as has various blurry or blocking tools to hide faces or other identifiable visible aspects in the phone (works pretty well!).
ObscuraCam project's home page
ObscuraCam on F-Droid
ObscuraCam on Google Play Store
@ijatz_La_Hojita