Mastodawn

Val Packett 🧉

In preparation for daily driving #MobileLinux I've been thinking a lot about what must be done for reliable and power-efficient push notifications.

Spoiler alert: while UnifiedPush may be a relevant service for some apps, it's not where platform dev focus should be. We want a future full of p2p apps that reject permanently-addressable "servers" entirely, after all! And centralization is not where the bulk of the power-saving magic is anyway. The "magic" is in the fact that the SoC can be in deep sleep and the modem will still wake it up with an interrupt when data arrives on an open connection. It should be fine to have apps' own service processes listen for notifications!

My rough sketch of a to-do list would be:

making sure wakeups don't turn the display on xD
research into what's needed to set up filtering on the modem for which sockets can wake the SoC up (but initially, fine to just rely on "nothing else has open sockets anyway, only the background services waiting for pushes" maybe?)
easy API for establishing the push connection specifically over mobile data if available (since only modem supports wakeup well rn)
support for robust background services: unlike what the Background portal offers now, let #Flatpak apps install systemd-user services, which would have metadata connecting them to the .desktop entry, making them introspectable and accountable via settings GUIs (not via control center popups! they shouldn't show up as "annoying left-over in-process thing possibly eating battery"! they're a different thing, expected to run permanently!)
actually getting apps to separate push notification listeners into background services

#postmarketOS #linuxmobile #freedesktop

Sebastian Krzyszkowiak 2d ago

@valpackett > making sure wakeups don't turn the display on xD

That'll get us, like, 90% there usability-wise already 😜 AFAIR gnome-settings-daemon simply turns it on unconditionally on resume from system suspend.

@valpackett I had similar thoughts, together with "wake up only bare minimum" (single CPU core, modem and ADSP, leave DSI, CSI, I2C/SPI/UART blocks in suspend (if possible, not sure if this can be done... I guess you can tell pinctrl to leave pins in pull-down mode?)

We don't need to wake up the entire SoC for this, which would drastically reduce the power draw (and therefore the battery life)

Val Packett 🧉1d ago

@elly just not turning on the display would leave DSI off, and if touchscreen is a panel-follower also that with the SPI/I2C used for it. that's all already managed by runtime pm!

USB of all things just is not there yet, it does pm_runtime_forbid() still (you can override that by echoing auto into power/control for all usb, xhci-hcd and related phy nodes and pray xD)

not enabling more than one CPU core until there's actual activity is very interesting yeah. somewhat related is using scx-lavd as the scheduler which does try to bunch tasks together onto one cluster to keep the others in suspend as much as possible. but hard "only 1 core online until the session is open" would be a nice big hammer :D

oh oh I also forgor for the list: bg services will need to have some way of opening the push connection specifically over the mobile data interface if available (unless specific-enough wakeup is possible with wifi after all)

✧✦Catherine✦✧1d ago

@valpackett @elly how does this solve the problem of "five applications independently maintain event sockets resulting in approximately five times the amount of wakeups"? the advantage of something like gms (no idea if unified push actually does it) is that it can batch events from unrelated sources into one wakeup

in theory, if you terminated all of the connections, including TLS, on the modem, you could achieve the same level of power savings. in practice i absolutely do not want to terminate TLS on the modem, that's horrifying

@whitequark @valpackett then you will be horrified to find out that smasnug modems (Samsung phones with Exynos SoC, Google Pixel 6 and newer) have a modem where ARM SoC (Control Processor, "CP") running ShannonOS (RTOS) is connected over PCIe and it's running a full IP stack on that RTOS.

AFAIK UnifiedPush does exactly that, I've seen a presentation about it and it acts kinda like GSF do.

✧✦Catherine✦✧1d ago

@elly @valpackett I'm not surprised at all that this exists (the modem is obviously capable of it and this is obviously desirable from a vendor's point of view), but I'm very surprised that there is both the desire and the capability to use that sort of stuff in FOSS ROM components

is this mechanism at least somewhat portable? do you want to keep using it? if the answers are yes then yeah aggregation on the modem is the way to go, I'm just surprised to see FOSS-minded people be okay letting that blob handle anything important

@whitequark @valpackett CP has BootROM that verifies signatures, I tried bit-flipping the firmware package it loads from modem partition and that broke the modem

✧✦Catherine✦✧1d ago

@elly @valpackett well yeah it better do that

✧✦Catherine✦✧1d ago

@elly @valpackett i found this slide deck which doesn't say anything about terminating connections on the modem, is there a different presentation? couldn't find anything in UP docs either

@whitequark @valpackett Ah, that's not what I meant. Sorry, I meant two different things.

UnifiedPush works completely in userspace

✧✦Catherine✦✧1d ago

@elly @valpackett okay yeah then to clarify my earlier statement: it horrifies me because the modem is some blob we don't control that is typically tested and built much more shoddily than the AP firmware (do they normally enable NX and stack cookies on it yet?), not because there's anything wrong with the abstract concept of 'brains in the modem'

@whitequark On Shannon specifically: NX is enabled, it has stack cookies and they started enabling some LLVM sanitizers. No CFI/PAC/ASLR. Slightly better than other modems, but whether their custom RTOS actually implements any of it *correctly* is anyone's guess.

@whitequark @valpackett @elly Why including TLS? I thought TLS doesn't require inband acks, so after TCP-acking one can insert an arbitrary delay before the TLS layer sees the data, no?

✧✦Catherine✦✧1d ago

@robryk @elly @valpackett i'm not a TLS expert so i won't comment on that part; the reason i think practically speaking you'd end up wanting to terminate TLS on the modem is because to minimize the time the SoC spends out of sleep you'd want the messages passed onto the SoC to be complete (no half-read ones where you need to wait for the rest) and useful (no passing along exchanges like "got data? no data!"). and i don't believe you can ensure that something you receive in a TLS envelope is complete because reading from a TLS stream may involve writing to the socket (i think for reasons like rekeying?)

maybe there's a way to half-ass it where it mostly works most of the time, but i've a distaste for halfassing greenfield systems

Demi Marie Obenour 1d ago

@whitequark @robryk @elly @valpackett I’d use asymmetric encryption and treat the modem as untrusted. Rely on it being behind an IOMMU and move its driver into a sandboxed userspace process.

Val Packett 🧉1d ago

@whitequark @robryk @elly IMO, batching and minimizing the time spent processing and all that is pennies. If you get the SoC to sleep for hours on end while 30-40 feral sockets are open and quietly waiting for something to happen, that's like the vast majority of the possible savings. Upon wakeup it can spend an entire minute turned on and it'll be fine. Recent SoCs just don't draw that much power when idle, this phone (SM7325/Yupik) measures 0.2W at the battery while responding to SSH commands over WiFi. That for a minute on every push, but CX rail collapsed while sleeping otherwise, would be super good. (Currently it doesn't even do CXSD yet, so suspend is not that much better than just being idle xD Still lasts an entire day without heavy screen-on use.)

@valpackett in my case fp5/pmos roughly uses two times more energy compare to the stock android. I'd like to learn and know for sure where the energy goes. powersupply is great app, but show only total current energy consumption. would be nice to get deeper into details. Please share thoughts how to know more.
thanks!

Val Packett 🧉1d ago

@Oleksii that's kinda basically impossible :) :) :) there isn't really any granular energy measurement in the hardware, heck all the hw shares power rails in ways that would make per-rail measurement not-so-useful.

fun fact, even what macOS reports as power usage by apps is kind of a guesstimate based on resource usage and factory measurement of how that vaguely translates into energy.

@valpackett thank you for sharing. Just for analogy. if we are talking about basic server use case without gpu, only cpu, ram and storage. In this case vast majority uses CPU i guess. of course linux mobile is way more complicated case, but at least its my thoughts.

@valpackett have no idea what is happening under the hood, but in case of kubernetes you have some control over cpu i guess its effect total energy consumption of server quite a lot.

ohh another use case. Im using watt app on stock android its very helpful when i need to save battery.

@valpackett I had an unhinged plan to write an XMPP client for the modem that just wakes up the phone SoC when there's data

@valpackett I am not a hardline systemd hater, but I do want to say I think having flatpak expect a user instance of systemd is a bad idea, given that flatpak exists in large part to make apps work independently of what's installed outside each app's sandbox.

Demi Marie Obenour 1d ago

@valpackett I’d love to be using mobile desktop-based Linux, but that will need a security model that is better than “yolo” 😆.

Val Packett 🧉1d ago

@alwayscurious yeah :) I wonder what is your threat model roughly wrt. mobile?

Demi Marie Obenour 1d ago

@valpackett These are at least some of the things missing from desktop Linux:

Disk encryption using the secure element and hardware-wrapped inline encryption keys that are tied to an OS signed by a specific key. Without this, disk encryption is usually very simple to brute force.

Protection against USB-based attacks. These are the main attacks used by Cellbrite. GrapheneOS (and only GrapheneOS) provides very strong protections against this.

Strong verified boot with distrust of persistent state. Android has it. Desktop-based Linux systems don’t.

As-strong-as-feasible application sandboxing. Flatpak’s sandbox is disabled by many popular apps. Android doesn’t allow that.

Security researchers seriously looking for vulnerabilities so they can be fixed.

New code in system services being written in safe languages (Rust, Kotlin, Java) rather than in C or C++. New code in applications is usually written in Kotlin or Java, which are both memory safe.

Forcing applications to migrate to new (and incompatible) APIs over time, rather than allowing them to stay on old APIs that provide excessive access.

Desktop Linux is maybe 15 years behind on these things, give or take a few years.

The hardest part of the problem to solve isn’t technical, though. It’s developer mindset. The Android team at Google is paid to be concerned with security. Desktop Linux developers are generally way more concerened with getting stuff to work at all. This doesn’t mean that they don’t care about security, but rather that it isn’t anywhere near as high a priority. For instance, Flathub is not willing to kick out apps that aren’t strongly sandboxed, and Flatpak doesn’t strictly enforce the use of runtime permissions rather than install-time permissions.

You can be secure or you can be compatibile with existing applications. You can’t have both unless you are willing to provide fake versions of everything the app thinks it has access to but doesn’t. Android has repeatedly shown it is willing to impose stronger and stronger restrictions, and Google Play Store policies force app developers to accept new restrictions on their apps as a condition of being able to update them. I know that the Google Play Store monopoly is bad, but it really does prevent app developers from doing lots of really bad things.

I think that Spectrum could solve many of the technical problems. It’s designed for verified boot from day 1, uses VMs for sandboxing, and uses Rust for its services. Unfortunately, it still doesn’t solve the disk encryption problem. Also, BTRFS is significantly more complex than F2FS, and desktop Linux apps ~never distrust persistent mutable state. Finally, the amount of resources required might be beyond what most phones can afford.

Adrian Vovk 22h ago

@alwayscurious @valpackett The good news is that there is work ongoing to improve this status quo

@valpackett Maybe have a service that the flatpak can use for whatever service level thing it needs (listening to notifications?) as allowing it to install a service breaks the whole idea with containerization layers like flatpak. Also it makes my spine tingle in a "run away" kind of way.

Val Packett 🧉1d ago

@aanee why would it break containerization? of course the service wouldn't be arbitrary whatever, it would only run things inside of the flatpak

@valpackett Then I missunderstood.

pabloyoyoista 1d ago

@valpackett for the kernel work related to the mobile data, maybe have a chat with @bshah :D

Adrian Vovk 22h ago

@valpackett part of the issue is trusting apps to keep their power usage low. Like sure, we could have dedicated low power backgroud services for apps. But how do we ensure that the app isn't going to be doing large amounts of work in the background for no reason (i.e. due to a bug)? CPU usage limits?

What we could do is allow apps to hand over a copy of the file descriptors to their open network connections. Then we can keep track of activity in the OS and wake the app when needed.

(1/2)

Adrian Vovk 22h ago

@valpackett Alternatively, we're already exploring OS-level p2p infrastructure over in @modal. We could have traditional style push notifications from the perspective of apps, but the backend could be a peer to peer service running outside the app sandbox.

The OS service would be responsible for transporting messages between apps (across devices!), so p2p-native apps will natively be going through the OS and we can make every p2p message optionally become a push notification

(2/2)

Val Packett 🧉21h ago

@AdrianVovk yes, everything from @modal is the reason I kinda ranted against focusing on server-based services :) Sure, actually having that handled in the session-level service would be nice.

I think background services are very necessary either way, for music playback, VPNs, whatever else. And yes the service manager can enforce limits, and do usage accounting that would be reported in gnome-control-center.

I have thought about the "pass file descriptor" thing, that's one of the first things that comes to mind but— what happens when the socket becomes readable? Just "wake the app" that still has the same socket open— well, this is just regular background activity with extra steps, just to be careful about apps' CPU time, which sounds kinda "meh" to me

Adrian Vovk 18h ago

@valpackett oh we totally need background services for other things. But those other usecases is kinda the reason I'd be hesitant to just let apps roam wild. We've found it helpful to stick to concrete purpose-built APIs than trying to address things with a general solution. An escape to systemd --user services might be in the cards but IMO that's only as a highly privileged sandbox escape. Not even because we can't sandbox these services, but because we don't know what it's being used for (1/2)

Adrian Vovk 18h ago

@valpackett As for FDs, it seems easier to set a one shot CPU limit than trying to keep track of usage over time. In other words, "the push notification adds N seconds to your app's execution time budget and once you're out you're out" is easier (both conceptually and in terms of implementation) than trying to tune some complex resource restriction that applies continuously. It also seems to fit better with a "Doze" like model where we wake apps periodically to let them do background work (2/2)