On Local Photo Management and the Command Line

Picasa and iPhoto

Picasa and iPhoto were great apps. Both were free. Both allowed you to manage your photos locally and both allowed you to take pictures with photo cameras, or your phone, and sync them then you got home. Over time our phones synced via the cloud to these apps.

We lost the habit of getting home and ingesting photos because everything was done automatically. We took pictures and they appeared in Picasa and iCloud and we didn't think about it too much.

This was the gateway habit that led to some of us having cloud first photo libraries rather than local ones. One of the factors that led to this is that Picasa synced automatically to the cloud, as did iPhoto.

This was excellent because for years our photographs were stored locally and in the cloud. That is, until I libraries grew and became too big for laptop hard drives. At this point we had to choose. Get a laptop with a larger hard drive, delete photos, or spend more on iCloud.At 3 CHF per month spending more on iCloud was an easy choice.

It's later, when the library grew beyond 200GB that Google One became an interesting proposition. You could get two terabytes for 100 CHF per year. That head room is a huge luxury, for as long as you can backup your photos from the cloud back to a local volume.

The issue is that you can't, and you couldn't. It's only recently that I really managed to export my photos from Google Photos and Flickr, and after two weeks of experimenting and learning.

The Issue

Cloud services are great for synching all your photos and videos, as well as all the photos you get from whatsapp, the screenshots and more.They're not good when it comes to re-organising files.

If friends and family share photos via Whatsapp, Signal or you download videos from TikTok or Flickr, they're all combined into your own photos. This makes a lot of noise twice. The first time in iCloud and Google Photos but the second time in your whatsapp history.

If people share photos and videos whatsapp downloads them to its own backup, as well as to your own photo gallery if it allows you to, which I recommend for one reason. Whatsapp has a nasty memory of taking 100MB or more per chat. This noise is from photos, videos, pdfs, gifs and more. You might have a copy in Google Photos, in Apple Photos and potentially Immich, Photoprism and other photo clouds.

Hard to tidy

Google Photos, Apple Photos, Immich and Photoprism are great at automatic cataloguing but not at helping you tidy up the mess they help you create. For a start Immich and Apple photos make a tremendous mess of your photos files and hierarchy if you give them free reign. You go from a neatly organised hierarchy to a machine friendly mess that you need to clean up if you choose to move away from them.

With Apple Photos and Google Photos I find it excruciatingly hard to "spring clean" when storage gets low. With iPhoto I noticed that files are almost immediately backed up to iCloud so that if you migrate to Immich and Photoprism you download an entire library, every time immich or Photoprism crash and need to be repopulated. This often takes a day of keeping the phone's screen on. That's why having a local library is key and why kDrive is a great tool and a better solution

The Local Advantage

With kDrive, as with Google Photos, Photoprism, Immich, iPhoto and others you save your photo to a cloud, but unlike with them, with kDrive you have a hierarchical folder structure that you can download and work on via command line tools for batch operations, or visually for manual tidying tasks.

Exiftool

If your files are fresh from Google Takeout, the immich folder structure or other you can use a command line prompt to reorganise everything chronologically and more.

Jdupe

With Jdupe you can look for duplicates automatically. With Immich I noticed that I had 27,000 duplicates to sort through. In some cases they're triplicates and in other cases the duplicates are thumbnail duplicates. To do this sorting, manually, with the Immich tool would take weeks or months. With Jdupe it takes a few seconds to a few hours depending on how many duplicates there are.

rsync

With rsync you can transfer files between volumes with ease and convenience. The computer does the work in the background, backing up to a local drive, and a remote drive.

The Tailscale Caveat

If you're synching gigabytes of files use the local ip address, rather than tailscale because tailscale will throttle you after a certain amount of data transfer, I suspect. It's also a lot faster to do things locally. If you do sync remotely sync it locally first, and then move the drive to the remote location.

Visual Sorting and Find

While waiting for rsync to complete certain jobs I went through libraries manually and noticed patterns. I asked Gemini to create a command to help move webp, png and mp4 files with one pattern from my photo library to a secondary photo library that I can sort through at another time. In one instance that removed 130 gigabytes of noise.

The Motivating Push

I abandoned iCloud as my Single Source of Truth for my photos when my photos reached more than 200 GB and shifted towards Google Photos. With two gigabytes of storage I enjoyed the luxurious feeling. I enjoyed it until I saw that I could get 6TB for 67 CHF from Infomaniak and that's when I spent a long time migrating off of Google Photos.

They make it very hard because you can't just download a chronological list of folders and files as you can with kDrive.

Almost a Terabyte to Sort

My Apple, Google, and Flick libraries came to almost a terabyte of data, most of it duplicates. Sorting through it by hand would take months. Using the tools above, once I had a workflow prepared, with the tools listed above took days. Now my library is 370-390gb.

And Finally

27,000 Duplicates in Immich

I tried ingesting from mobile phones and an old immich library but in so doing I ended up with 27,000 dulicate pairs that I would have to sort through by hand. This task would take months. By removing all the duplicates, before ingesting into Immich I will save weeks of tedious work.

JDupe and Peace of Mind

My iCloud library hasn't been the single source of truth for years, due to the 200gb limit. For a while Google Photos was, until I downgraded the plan, and then it became a former single source of truth. Now I hope that Flickr will have filled many of the gaps. On a drive or two I have old iPhotolibraries.

If required I can open the package, extract the original. Run exiftool to create a chronological library, and repeat until all my libraries are consolidated, and then I can import them to my main photo library, and ingest them to Immich and Photoprism

Conclusion

With command line tools you can consolidate photo libraries from multiple sources into a single source of truth, and move on. By maintaining this single source of truth, and backing it up to kDrive, Google Drive or even iCloud you ensure that it is complete, and easy for immich, Photoprism, or some other tool to ingest.

Immich and Picasa – Past and Future

Reading Time: 3 minutes

Those of us that are geeky, and old enough, will remember when apps were local and we saved data to internal and external hard drives. We would take photos with a camera, or a phone, and once we got home we would download them to our computer, using apps like Picasa and iPhoto. Both apps would help us to organise them chronolgically, and eventually by location, people in photos and much more.

Eventually Picasa and iPhoto moved to the cloud, and more and more of our photos were managed for us. They were offloaded from our phones and our laptops, to live in the cloud, controlled by Apple and Google.

Over time, as Google Photos picked up steam, so Picasa lost users, until Google pulled the plug. After that we were stuck with iPhoto. The issue with iPhoto is that it required gigabytes of storage, and our laptops didn't have that much space. Neither did our phones. Our content emigrated to the cloud. We lost control and access to our own media.

Eventually, many yearss later self-hosting came along via Immich, Photoprism and other solutions. Immich and Photoprism are the Picasa of the cloud age. By this I mean that in the age of mobile phones, and laptops, we have offloaded photo management from our laptops, and mobile phones to online services. For a while these online services were owned by others. With Immich, and Photoprism, if we ignore NextCloud and other solutions, we are once again self-hosting our photos and other media files. We are once again moving away from the cloud.

Over two years I have moved from Raspberry Pi 4 towards Raspberry Pi 5 8GB and then 16GB before moving back to 8GB. I also had an immich instance on my laptop but stopped when I filled up the laptop's storage. It makes sense to store photos on a Pi.

Self-hosting a solution such as immich is similar to self-hosting via Synology or other solutions, but on a budget. It is also more involved, because it requires gaining an understanding of Linux, docker containers and more. It requires you to update apps when required, and if something goes wrong to debug.

It also requires you to learn how to backup that data, or migrate it from one storage medium to another. Over time you get better at it, and that's when you can remove backup solutions such as Google Photos.

Monthly Fees and Shortage of Space

Lightroom, Google Photos, Photos, for Mac, and other apps all have the same issue. They store your photos in the cloud, and in many cases retrieval is slow and clumsy. In the case of iCloud you need a volume with as much space as the library occupies. Most laptop drives are too small, The second problem is retrieval time. When I tried to recover photos from iCloud I saw that it would take days, if not weeks.

With Google Photos, and Flickr, if you write a simple shell script you can download photos within an extended period of time, but several zip files at a time.

Lightroom and other apps are burdened with monthly fees, which means that you pay every month, whether you use it a lot, or a little. You also get trapped, and that is what I don't like. If I want to shift from one solution to another I want it to be fast.

That's why Immich is good. With Immich you store a local version at home, and you can backup to a cloud service. The day that one cloud service is cheaper than another you cancel Service A and move to Service B without worrying about retrieving the data, because the cloud version is just an offsite backup, not the primary. It's because cloud solutions try to hold on to data, that I feel they should be a backup cloud solution rather than a primary cloud solution.

By self-hosting your primary cloud solution you have the data locally, and you can back that data up locally too. The cloud is just the offsite third copy.

And Finally

With the shift from Desktop apps to web apps, and from free solutions to paying solutions Picasa became Google Photos and Mac OS Photos relies on online storage rather than laptop storage, The result is that with web apps, self-hosting solutions replace locally installed laptop and desktop apps. In the multidevice era it makes sense for us to install apps on our own web servers, and to access them via Tailscale when away from home, and via the local network when at home.

Immich is designed for this, specifically. It allows you to say "use 192.168.1.2 when on home wifi and use immich when not on home wifi." It has automated the switch.

In essence, desktop apps are now self-hosted apps.

#app #cloud #free #Google #immich #local #Picasa

#Picasa wurde schon vor 10 Jahren von #Google gekillt und es hat immer noch Fans. Mein Blogpost "Picasa3 Datenbank retten" ist seit Jahren einer der meistgeklickten Artikel und ich habe dazu sogar gerade wieder einen Kommentar bekommen.

Ich würde das auch immer noch benutzen, wenn ich noch Windows nutzen würde. Das war einfach richtig gut und es war nicht in der Cloud. Es wollen nicht alle Leute ihre Bilder ausschließlich in der Cloud haben.

Schade, dass bisher niemand Picasa als #OpenSource nachgebaut hat: Lokale Datenhaltung, schlichte, übersichtliche Oberfläche, lokale Geschichtserkennung.