Damian LettieOct 6, 2024
Christian Selig
Beware UserDefaults: a tale of hard to find bugs, and lost data https://christianselig.com/2024/10/beware-userdefaults/
Beware UserDefaults: a tale of hard to find bugs, and lost data

Excuse the alarmist title, but I think it’s justified, as it’s an issue that’s caused me a ton of pain in both support emails and actually tracking it down, so I want to make others aware of it so they don’t similarly burned. Brief intro For the uninitiated, UserDefaults (née NSUserDefaults) is the de facto iOS standard for persisting non-sensitive, non-massive data to “disk” (AKA offline). In other words, are you storing some user preferences, maybe your user’s favorite ice cream flavors? UserDefaults is great, and used extensively from virtually every iOS app to Apple sample code. Large amount of data, or sensitive data? Look elsewhere! This is as opposed to just storing it in memory where if the user restarts the app all the data is wiped out.

Oct 5, 2024 at 10:57pmIvory for Mac
Show threadCody KriegerOct 5, 2024

@christianselig completely unrelated to the specific symptoms you’re detailing, but i have had issues with NSUserDefaults not persisting changes on macOS in years past that led me to write my own settings serialization mechanism.

i could reproduce the issue only sparingly, and never did figure out the root cause, but it caused a ton of support requests from users. the moment i shipped my bespoke settings serialization, the support emails dropped to zero. i have not trusted it since.

Show threadChristian SeligOct 5, 2024
@codykrieger That's great to hear, hoping that will be my result too haha
Show threadRodrigo Cardoso BuskeOct 6, 2024
@christianselig good too know about this problem, but about your solution, wouldn’t be better to use an actor instead of a queue? Using sync may lock the thread anyway right?
Show threadChristian SeligOct 6, 2024
@Robuske It would lock a background thread only enough to perform minor file IO, shouldn't be noticeable. I swear I remember reading actors weren't recommended for file IO nor do I believe they guarantee any form of serial execution
Show threadRodrigo Cardoso BuskeOct 6, 2024

@christianselig they do guarantee serial execution by nature of how they work
“only one task to access their mutable state at a time”

https://docs.swift.org/swift-book/documentation/the-swift-programming-language/concurrency/#Actors

That said, am still learning the new concurrency system, so I don’t know about they being bad for IO, what I have seen is that it might be a better idea to use @‘MainActor instead of really creating an actor since a lot of Apples APIs need to be run on the main thread

Documentation

Show threadRodrigo Cardoso BuskeOct 6, 2024
@christianselig I have read my answer again and realized that it is actually only guaranteeing one access, not that it would be in the same order, I think I read that somewhere, but is not present in that part of the documentation 🤔
Show threadChristian SeligOct 6, 2024
@Robuske Ah found it. Yeah there’s some interesting discussion in here about that problem https://forums.swift.org/t/actors-that-serialise-file-access/66652
Actors that serialise file access

Technically speaking even that is not guaranteed. Swifts actors are just not FIFO today. If a high priority task arrives and others are normal, it may get to execute before the others. It was designed this way, in order to facilitate serving those high priority work as soon as possible. And even allowing an “skip the work, we no longer need it!” Messages to jump in front of the queue etc… But yes, it means we just don’t — in the general sense of the word — have FIFO in actors today. If all y...

Swift Forums
Show threadtaichimasterOct 6, 2024
@christianselig @Robuske Regardless of dispatch queue or actor, it only protect access within a single process. To guard against data races against multiple process (like app and an app extension), I believe we have to use `NSFileCoordinator`.
Show threadChristian SeligOct 6, 2024
@taichimaster @Robuske if writing atomically the os should be able to handle that aspect, no?
Show threadGwendal RouéOct 6, 2024
@christianselig @taichimaster @Robuske The OS won't protect you from everything. Say both processes want to increment a value. P1 reads 0, increments in memory, is about to write 1. But now P2 does the same: it reads 0 because P1 did not write yet. In the end both processes write 1. Zero incremented twice should give 2, not 1. NSFileCoordinator can prevent this bug.
Show threadChristian SeligOct 6, 2024
@groue @taichimaster @Robuske Oh definitely, I meant more so from data corruption, that is a great point though, I'll probably implement NSFileCoordinator now, cool little API
Show threadGwendal RouéOct 6, 2024
@christianselig @taichimaster @Robuske Yeah, with that spicy Objective-C flavor from 2011 😅
Show threadDanielOct 6, 2024
@christianselig @taichimaster @Robuske depends, but in general, the answer is no if writing to intersecting files happens from different processes:
Atomic writing doesn’t prevent a contending write to finish first, and being overwritten by the slower writer. (For the same process, your queue can prevent that. The missing details in the articles snippets make it impossible to judge, if that’s the case.)
If you want to be sure no accidental overwrites happen, SQLite would do the trick nicely.
Show threadChristian SeligOct 6, 2024
@danyow @taichimaster @Robuske Right, but you're only looking at out of sequence overwriting occurring, right? Not actual data corruption due to concurrent writes? That's still worth protecting against, but just want to make sure I understand
Show threadDanielOct 6, 2024
@christianselig @taichimaster @Robuske corruption in the sense of “can no longer deserialise a given key” would be prevented by atomic writes, correct.
Show threadJason HowlinOct 6, 2024

@christianselig @Robuske it actually blocks the calling thread. Here you could optimize by only blocking the calling thread on a read. You could change the write to async.

However, you're right, it's likely trivial.

The queue or lock hides the blocking to the caller, while the actor exposes it by requiring an await (for which the caller may need to create an async context (ie Task).

Show threadChris Wu 🪐Oct 6, 2024

@christianselig @Robuske Interesting!! The sample project for WeatherKit from two years ago used actors for saving and loading forecast data.

https://developer.apple.com/documentation/weatherkit/fetching_weather_forecasts_with_weatherkit

Fetching weather forecasts with WeatherKit | Apple Developer Documentation

Request and display weather data for destination airports in a flight-planning app.

Apple Developer Documentation
Show threadChristian SeligOct 6, 2024
@MuseumShuffle @Robuske It might work, but at least in its current implementation I'd worry about stuff like Task priority inadvertently leading to file writes being out of order leading to data loss https://forums.swift.org/t/actors-that-serialise-file-access/66652/17
Actors that serialise file access

Technically speaking even that is not guaranteed. Swifts actors are just not FIFO today. If a high priority task arrives and others are normal, it may get to execute before the others. It was designed this way, in order to facilitate serving those high priority work as soon as possible. And even allowing an “skip the work, we no longer need it!” Messages to jump in front of the queue etc… But yes, it means we just don’t — in the general sense of the word — have FIFO in actors today. If all y...

Swift Forums
Show threadJonathan WillingOct 6, 2024
@christianselig Interesting, I can also confirm this happens with my Live Activities after a device restart. I’m not sure how common this scenario is (reboot during an activity), but still worrisome. Thanks for the post!
Show threadChristian SeligOct 6, 2024
@willing Yeah, the fact it can happen at all is the bonkers part!
Show threadGrapheelOct 6, 2024
@christianselig Thanks for the warning ! Never had to use it so far, and now never will 😬
Show threadJoefrey Kibuule, MDOct 6, 2024
@christianselig Thanks for saving me some heart ache when I work on my first independent app in a bit.
Show threadPaul Haddad Oct 6, 2024
@christianselig last time I ran into something like this, it wasn’t pre warming it was push notifications with one of them flags that causes background processing. Possible in your case?
Show threadKhaos TianOct 6, 2024
@paul @christianselig +1, had the similar experience with background refresh triggered by push notification while the phone rebooted... It's really annoying 😅
Show threadJim DusseauOct 6, 2024
@paul @christianselig +1, this is my guess
Show threadChristian SeligOct 6, 2024
@paul No push notifications in my app for better or worse, the user reports tend to be right after reboot within a Live Activity context, but I bet push notifications could totally cause it too!
Show threadgroxxOct 6, 2024

@christianselig yep. There were similar-ish data loss issues with it back when I did iOS dev in 5+.

I ended up making my own then too. It's extremely easy, and then you know the full behavior. Highly recommended.

Show threadHendrikOct 6, 2024
@christianselig Thanks for the warning, I had no idea and always considered UserDefaults one of the (few) APIs that “just work”. But it seems like this issue has been around since iOS 7 https://developer.apple.com/forums/thread/15685
NSUserDefaults values lost on back… | Apple Developer Forums

Show threadChristian SeligOct 6, 2024
@hendrik_kueck Yeah probably just got more prevalent over time
Show threadAlexOct 6, 2024
@christianselig
1. we also had a hard time figuring out why our app lost data. Did cost us a lot of reputation. We had a mechanism to figure out whether it's the first start of the app, which relied on UserDefaults. When it was a first start we "cleared" the keychain in order to avoid using keychain data from previous installations...!
Show threadAlexOct 6, 2024

@christianselig
2.
In our case it was the companion app on the connected Apple Watch, which started our app, even while the phone was locked!
Easy enough to solve the problem once you know about what you describe so good in your article.
Now we just create an empty file in the app's data container on first start and just need to check existence. No need to read the file.

But until we found it we looked like idiots. :-)

Show threadNick LockwoodOct 6, 2024
@christianselig oh god, this is horrifying
Show threadRobert AtkinsOct 6, 2024
@nicklockwood @christianselig And the apparent lack of documentation is the cherry on the cake.
Show threadNick LockwoodOct 6, 2024

@christianselig btw, I saw some discussion in the comments around the use of a DispatchQueue for thread safety. I do think that's overkill, and actors would be also since it would make the API async and more awkward to use.

I'm fairly sure the recommended approach for something like this is still just to use a simple lock (NSLock works fine and is fast enough).

Show threadChristian SeligOct 6, 2024
@nicklockwood My comment was referencing you probably don't need a concurrent queue for this (specifically for writing with a barrier lock or something) if you're keeping your files pretty small. DQ in general is nice for ensuring some level of serial operations though, I don't think actors provide that. Honestly I don't think for something this simple you're gonna be hitting any noticeable performance between methods of how you're synchronizing access tbh
Show threadNick LockwoodOct 6, 2024
@christianselig yes, I completely agree
Show threadnSonicOct 6, 2024

@christianselig @marcoarment
Holly crap?! What on earth … WHAT?!
Thank you so much for this finding and writing about it 🙏

(I got some old apps to develop them further and historically they are using an self made file storage … I was thinking about changing this for the iOS versions for some time but know I let everything as it is! Wow)

Show threadHendrikOct 7, 2024
@christianselig I came across some interesting info in the SO post linked below that might help your solution 1: It seems that when the protectedDataDidBecomeAvailableNotification fires you should call UserDefaults.resetStandardUserDefaults() to force it to read the now available data. Though Apple’s documentation for this method states “This method has no effect and shouldn’t be used”. So maybe this used to work but doesn’t anymore? 🤷‍♂️
https://stackoverflow.com/a/20840053
NSUserDefaults losing its keys & values when phone is rebooted but not unlocked

We are currently experiencing the following weird issue with our iPhone app. As the title says, NSUserDefaults is losing our custom keys and values when phone is rebooted but not unlocked, and this...

Stack Overflow
Show threadHendrikOct 7, 2024
@christianselig And at least in the Swift implementation of Foundation resetStandardUserDefaults() really is a noop:
https://github.com/swiftlang/swift-corelibs-foundation/blob/e55e1d88001997be830fbc01086564431d405dad/Sources/Foundation/UserDefaults.swift#L92
swift-corelibs-foundation/Sources/Foundation/UserDefaults.swift at e55e1d88001997be830fbc01086564431d405dad · swiftlang/swift-corelibs-foundation

The Foundation Project, providing core utilities, internationalization, and OS independence - swiftlang/swift-corelibs-foundation

GitHub
Show threadChristian SeligOct 7, 2024
@hendrik_kueck Really interesting thread, thank you for mentioning this, cool to see other folks running into this issue years ago haha! I think my issue is that having to wait for protectedDataDidBecomeAvailable alone is a dealbreaker, let alone having to pray through weird methods that UserDefaults actually properly responds to the change and loads the disk store into memory :/
Show threadHendrikOct 7, 2024
@christianselig Yeah, I agree. The solution further down in that thread to just crash the app in this (hopefully rare) situation is interesting too. Though I’d be scared to deploy that.
Show threadmhalttuOct 7, 2024
@christianselig Thank you for an excellent post on this topic! I’ve seen the same issue even though I store my settings in a file. I’m a bit hesitant to disable encryption for my use-case. What do you think of moving the initialization code (that depends on the settings file) from init to applicationWillFinishLaunching?
Show threadChristian SeligOct 7, 2024
@mhalttu That area still isn't safe, nor is didFinishLaunching. Why are you worried? You shouldn't be storing anything sensitive in there, and even if you are, it's decrypted the first time the user unlocks their device after reboot, if they lock it again it's still decrypted
Show threadmhalttuOct 9, 2024
@christianselig may I ask why do you say that method is not safe? According to https://developer.apple.com/documentation/uikit/app_and_environment/responding_to_the_launch_of_your_app/about_the_app_launch_sequence “Prewarming executes an app’s launch sequence up until, but not including, when main() calls UIApplicationMain(_:_:_:_:)” and applicationWillFinishLaunchingWithOptions happens only after that.
About the app launch sequence | Apple Developer Documentation

Learn the order in which the system executes your code at app launch time.

Apple Developer Documentation
Show threadChristian SeligOct 9, 2024
@mhalttu See my blog post, the docs are just wrong haha
Show threadmhalttuOct 9, 2024
@christianselig I had missed that part. Wow. The fact that the documentation has been wrong for years blows my mind. Thanks again!
Show threadChristian SeligOct 9, 2024
@mhalttu Mine too haha. No problem!
Show threadSaagar JhaOct 8, 2024
@christianselig Hello ActivePrewarm my old friend
Show threadDrew OlbrichOct 23, 2024
@christianselig Ugh, I was aware of this issue in the context of the keychain, but I didn't know or I had forgotten it applied to `UserDefaults` as well. Thank you for the writeup.
Show threadChristian SeligOct 23, 2024
@drewolbrich Same! At least the keychain APIs expose it nicely
Show threadDrew OlbrichOct 23, 2024
@christianselig Based on your experience so far, do you feel that for a vanilla app that doesn't use anything like Live Activities or widgets or any sort of extension, and which doesn't access UserDefaults until UIApplicationDelegate/application(_:didFinishLaunchingWithOptions:) is called, then UserDefaults would behave as expected in all cases and never fail, including shortly after the device reboots?
Show threadChristian SeligOct 23, 2024
@drewolbrich No I wouldn't trust that at all unfortunately to be honest, app prewarming already seems to behave in unexpected ways (not following what the documentation says), for instance appDidFinishLaunching can already have UserDefaults in a bad state, and who knows what other magic circumstances can cause it. I just really can't trust it anymore https://stackoverflow.com/questions/71025205/ios-15-prewarming-causing-appwilllaunch-method-when-prewarm-is-done
ios 15 prewarming causing appwillLaunch method when prewarm is done

We had an app in production which was reporting very high time to interact(tti) for ios 15 prewarm. TTI = timewhenViewController is loaded - mainStartTime mainStart time is measured inside AppDeleg...

Stack Overflow
Show threadDrew OlbrichOct 23, 2024

@christianselig Good grief.

OK how about, if I leave UIApplicationDelegate/application(_:didFinishLaunchingWithOptions:) empty, and only access UserDefaults after UIWindowSceneDelegate/scene(_:willConnectTo:options:) is called.

That should be safe, right? It's probably what Apple implicitly expects for an app based on scenes.

Show threadDrew OlbrichOct 23, 2024
@christianselig It's glorious how in Xcode if you create a new empty iOS project, it helpfully creates AppDelegate.swift and SceneDelegate.swift for you, and the app delegate's didFinishLaunchingWithOptions includes a cheerful comment inviting "customization", and yet it seems this comment should actually read "customization...but for the love of all that is holy, no code that references UserDefaults, unless you want to waste multiple days of debugging time several months from now".
Show threadChristian SeligOct 23, 2024
@drewolbrich I wish I only spent days on this one >_<
Show threadJohn DavisOct 24, 2024

@christianselig @drewolbrich

I wonder how this impacts @.appStorage and @.sceneStorage

Show threadChristian SeligOct 24, 2024
@its_john_davis @drewolbrich In those cases it should update dynamically upon UserDefaults becoming available as it's effectively live updating
Show threadChristian SeligOct 23, 2024
@drewolbrich That's called too in an unsafe state I believe, try with a Live Activity after reboot :) To be totally clear I'm saying I would not trust UserDefaults to really be safe in essentially *any* state given that the API isn't clear/built to be safe around this kind of thing