Week 2 update on the local S3 server.
Filesystem scanner landed. Three levels: L1 discovers files
L2 collects size and mtime
L3 streams MD5 (ETags) and SHA-256 (content hash) in one pass
Uploads via S3 API get full metadata immediately. Files on disk get indexed progressively. (inotify triggered)
Delete from disk = gone from S3.
Also: SigV4 auth, multipart uploads stream to disk, CopyObject, tagging, conditional requests, range reads.
Building an S3-compatible server for local filesystems. Point it at a directory, get an S3 endpoint.
Files stay where they are. Works with rclone, AWS CLI, any S3 SDK. When the object store knows every file's content hash, duplicates are just a query.
Started to deduplicate photos on my NAS. Realized the S3 API unlocks more: backup tools, sync workflows, dev tests.
Building an S3-compatible server for local FS. Started with the duplicate photos problem. Realized the real value is making local files accessible via S3 API.
1st milestone: `shoebox ~/Photos` serves an S3 endpoint. `aws s3 ls` returns actual files. PutObject, GetObject, DeleteObject, ListObjectsV2 all working. SQLite metadata layer + filesystem ops with symlink safety.
Next: SigV4 auth, multipart uploads, scanner.
#selfhosted #selfhosting #homelab #rustlang #opensource #buildinpublic #S3
20 years of photos across a pile of external drives, now on a 6-drive btrfs array. Phone backups via NextCloud, old laptop migrations, scanned childhood photos from the 80s. Some duplicated 3x across forgotten folders.
Tried a bunch of dedup tools, they all want to reorganize my files, upload them somewhere, or just fall over.
An object store that tracks content hashes makes duplicate detection a query. Building one in Rust.
