EmanuelMar 20
Matt Birchler

Increasingly, my number one feature hope for OS 27 is a larger context window for Apple's on-device model. There are so many features I am trying to build that I would love to run 100% locally, but I can't.

As a basic example, you literally can't use it as a meeting note generator because any meeting more than 5-10 minutes is going to exceed the window.

Mar 19 at 2:41pmIvory for iOS
Show threadMatt BirchlerMar 19

In my other apps, I've fallen back to using Gemini for this sort of thing. But now that means there's an external connection, and there is a usage-based cost that I'm dealing with and need to offset by charging customers

And because power users are going to generate exponentially more cost than the average user, I have to raise the subscription price for everyone to offset those users. Even though most people could be well profitable for me at a lower cost.

Show threadAndyMar 19
@matt_birchler it’s not a great solution but could you have it setup so that after X basic usage the power user has to input their own AI tool API key to use their own credits? Or does Apple then consider than locked functionality, or whatever.
Show threadMatt BirchlerMar 19
@andyn Yeah, I could do that. Apple would have no issue with that sort of gating. Like you say, it's just a bit messy and adds complexity to the app. Something to consider, though!
Show threadYury MolodtsovMar 20
@matt_birchler @andyn Or you could sell a limited volume in the subscription and offer the rest as one-time payments?
Show threadGreg PierceMar 19
@matt_birchler I've started to think we're going to get something like iCloud Storage, at least for the gemini-integrated stuff. Everybody gets a minimal token limit free, but you can sub for higher limits.
Show threadJames Huff Mar 19
@matt_birchler https://apps.apple.com/us/app/locally-ai-local-ai-chat/id6741426692 will give you a massive window for the on-device model. Unfortunately, it will inevitably reveal how inept the model is too. 😅
Locally AI - Local AI Chat App - App Store

Download Locally AI - Local AI Chat by Adrien Grondin on the App Store. See screenshots, ratings and reviews, user tips, and more apps like Locally AI - Local…

App Store
Show threadMatt BirchlerMar 19
@macmanx Yeah, it does it through third party models that require a few GB extra to download and use. It could work, but man, Apple's model is already there and is fine, it should work for more use cases 😭
Show threadJames Huff Mar 19
@matt_birchler It supports Apple’s foundational model too, no extra download required, only if you want more.
Show threadMatt BirchlerMar 19
@macmanx For sure, I guess my my use cases (podcast and meeting recordins) you literally always need more 😅
Show threadMatt BirchlerMar 19
@macmanx I take it back…I installed the app and it fails at everything I'm trying to do as well, even when I pick the best models it has. Bummer for sure.