Okay this is *crazy*
An agent running entirely on iPad* that can control any app
| Website | https://www.finnvoorhees.com |
| GitHub | https://www.github.com/finnvoor |
Finn Voorhees
@finnvoorhees
- 1.6K Followers
- 134 Following
- 727 Posts
A microcontroller plugged into the iPad acts as both a MIDI receiver and HID mouse, and an agent running in an iOS app sends signals to the microcontroller to control the cursor. A broadcast extension also streams video to the app so the agent knows what it's looking at.
even managed to make it work in the EU, something OpenAI can't even comprehend
you can just copy things
(computer use in pi)
if you want to keep using Claude Pro/Max in pi 🤫
https://gist.github.com/finnvoor/25ca1758cbb358eb7c716976ce85861c
ASTC was made for images and doesn't really generalize to model weights very well, at least from my experiments. They extracted significant vectors into a LORA adapter first, but even with a custom ASTC encoder tuned for weight heuristics I still got nowhere near int4 quality
I'm fascinated by this section in an Apple paper about how they're using ASTC to compress models to 4 bit, then using the hardware decode to decompress with no overhead. I don't understand how ASTC could ever be even remotely close to 4bit quantization in terms of NRMSE though…
Made a lil tui agent entirely in Swift, only dependency is swift-cmark for markdown parsing
ML Research Bot🤖 Athena: Intermediate Representations for Iterative Scaffolded App Generation with an LLM
"It is challenging to generate the code for a complete user interface using a Large Language Model (LLM). User interfaces are complex and their implementations often consist of multiple, inter-related files that together specify the contents of each screen, the navigation flows between the screens, and the data model used throughout the application. It…"
