Steve Troughton-SmithTurns out the M1 supports enough of the hardware accelerated raytracing APIs that I get a massive speedup here, too?
Trying to parse Apple's Metal feature support tables is hurting my brain, so I'll just accept it and move on
Not me offloading raytracing workloads to my iPad because it's so much faster than my MacâŚ
I spent untold hours in Bryce3D as a kid making silly scenes just like this. It did this stuff with machines a thousand times less powerful than my Mac mini. Certainly puts things into perspectiveâŚ
It's real nice having a complex app like this on iPad, that's for sure. This is by far the most complex and impressive thing I've vibecoded with Codex 5.3. I haven't seen or touched a single line of code, I didn't have a detailed plan to work from â this isn't at all like the structured ports I detailed previously
I thought my water should have some translucency and falloff, so I can put things underneath it
Recap: I vibecoded (code unseen, no plan) a 3D editor/renderer that has a scene graph, editing controls, primitives and gizmos, materials, procedural terrain and water, and hardware-accelerated Metal raytracing with soft shadows, clouds and bounce lighting, that runs on Mac and iPad.
Tool: Codex 5.3 Medium
Time: About a day's worth of work has gone into it
The thing about material shaders is I don't quite know the right questions to ask â the unknown unknowns. Am I taking into account the right refraction, internal reflection, attenuated shadows, etc. Why is this too bright, why is this too dark. You can tell from these examples where things are obviously wrong, and it takes quite a bit of iteration
Maybe I should have bought a faster Mac before trying to write a raytracerâŚ
Trying not to fry my GPU with caustics, but Metal isn't happy
I was sitting through hour-long renders (!) on my iPad yesterday, so I did an optimization pass on the hardware acceleration and it's much, much improved for simpler scenes, even on an M1
While this raytracer may never become a finished app, there are certainly elements from it I intend to yoink for future projects â like the really neat toolbars that go around all the screen edges, they would fit into a complex pro app very nicely
I made my control groups collapsible, with a priority system. Honestly they're my favorite part of this prototype
The old viewport gizmo was faked in 2D, so I had it rewrite it in Metal and with a different projection, and now it's much better. I also added exponential decay to the orbit gesture so you can fling the camera around
Ha, cute, you can even fling the raytracer around đ¤Ł
Also I added an expanded progress indicator
Hadn't tried the visionOS build, but it works too.
With a caveat.
visionOS is far more fragile to anything like 3D rendering. Saturating the GPU like this slows the compositor to a slideshow, and even got to a point where Metal was leaking out of the window into the OS and I was seeing squares of corrupted video memory in front of me until I got the equivalent to a SpringBoard crash.
Functionally, this could be a visionOS app.
Practically, no.
You can see here that the moment I invoke the raytracer, everything goes to shit on visionOS. The userspace went down right at the end of the video, where it cut
Pretty much everything I've worked on with Codex up to now has been stuff I could have built myself, within my area of expertise (or learnable), it just would have taken weeks or months.
This 3D scene app is something I never would have been able to build myself. I would have needed a team of rendering experts with domain-specific knowledge and human-years of research
I love how visionOS, uniquely, *explodes* when rendering goes wrong.
Hello [MacBook] Neo.
So now that visionOS 26 lets you spawn immersive scenes from UIKit apps, I had Codex implement me an immersive scene using Metal and CompositorServices that mirrors the in-window viewport and lets you live in your scene đ
It's real frickin cool.
The raytracer might be off limits for visionOS, but there's a lot of interesting stuff to do in other areas
I figured why not use RealityKit for the material previews, so now they are actual spheres.
Miraculously, it all still works â the Metal viewport, the Metal immersive scene, and the RealityKit UI elements, but it's very clear the Vision Pro (M2) doesn't have much headroom to build an actual app around this stuff
The raytracer is, for now, a no-go on visionOS. It's possible I could throttle it and stay within visionOS' systemwide render budget. But it's probably worth improving the RT performance a bunch on its own first before I come back and try it here. I might run out of steam on this prototype before then.
This entire app project is still in my 'Temp' folder, where throwaway projects live đ
This project, which runs on iPhone, iPad, Mac, and Vision Pro (with Immersive Space), is now 16.5K lines of code
I thought it was finally time to add vertex editing and subdivision. Now it's a 3D modeling tool and not just a raytracer
Some more things to show off here on this iPad mini 6!
⢠Longpress band gesture
⢠Multi-select
⢠Vertex editing
⢠Subdividing
⢠My 'generate a Cornell Box' button
⢠(And the raytracer, of course)
There is a lot of really neat stuff in this app. Still using Codex 5.3 Medium, still haven't touched a line of code myself
The touch gestures all work on visionOS too, but on all platforms it has keyboard and mouse support for all your precise selection and modifier key needs
Boolean operations seem pretty complex, but I made a start at it
Playing a bit of musical chairs with the floating controls in the toolbars now that I'm starting to run out of space for new UI
I made sure all my interactions work right with the Logitech Muse (i.e. stylus won't orbit the viewport, will modally lock to highlighted gizmo axes, etc), so now I can do a bit of vertex editing on the Apple Vision Pro, channeling @Dreamwieber
I never really thought about it before, but multitouch is actually legit for 3D modeling tools, maybe even better than a desktop. On a Mac, you need to hold modifier keys (or buy a multi-button mouse) to do everything you want with the viewport, but on touch you can orbit, pan, zoom, and multi-select very easily. If you special-case the stylus too, like I am, it feels very powerful.
Almost all of this extends to spatial computing, though visionOS struggles a bit with two-hand gestures
The raytrace operation will now be dispatched into the background on iOS, allowing for long-running background tasks
iPadOS has never been better for rich, complex, desktop-class apps. Almost all of the old barriers and blockers are gone.
Sadly, Apple waited until most developers had run out of patience with the platform.
If this thing had Xcode, a real Xcode, it would be effectively complete
(Maybe I should vibe code Xcode, next.)
Even my iPhone 12 Pro Max can raytrace!
Which I guess is not all that surprising, considering the A14 chip is the same generation as the M1
Thatâs a whole lot of raytracing from a little iPad. Biggest render to date, at 5120x2880 â took about an hour to get to the final pass.
Crashed right at the end, so I didnât get to take a picture of the final output đĽ˛
The raytracer still needs a bunch of work, but itâs more and more capable day by day
I was tired of running into raytracer resource limits, so now it uses wavefronts and a scheduler with a 4+ Kloc rewrite. Seems a lot better for complex scenes and lots of glass. All that work spent optimizing the previous renderer's performance ramp and failure recovery pays off now that the new renderer is so much more robust
Also I've switched my Codex model to GPT 5.4, OpenAI says it outperforms gpt-5.3-codex and it has over double the context window, so I figured a renderer rewrite was the right time to step up a level
I feel like I need a bunch of Apple's glass spheres on my desk to try to fine-tune my raytracer đ
The raytracer rewrite bears fruit â it can now make it through a long-running raytrace on visionOS without bringing the userspace down. It still chokes on more-complex shaders like glass, but it's a big improvement
This 3D modeling app and the switch to the more-expensive GPT 5.4 model might finally be the thing that burns through my weekly usage budget. It's now 25K lines of code; the more-complex renderer really ballooned things a bit, but it's worth it.
I finally moved it out of my temp folder and into my projects folder, and handed it off to a fresh GPT 5.4 session with a bunch of documented context about the intricacies of the project. Hopefully that's enough to jump-start it
I gave it an easy task to start with: a 'Frame Selection'/'center camera' button. Select an object or a group of objects and reposition the camera to fit
TheEJJ đ¨đŚ@stroughtonsmith I declare myself the oracle:
Timo Hetzel@stroughtonsmith 5.4 is more token efficient, so yes itâs more expensive per token but not clear how much more usage on the $20 plan it really uses. Looks to be about the same for me. Setting to high or xhigh really burns through the quota though. I use https://codexbar.app/ to keep an eye on that.
Brandon@stroughtonsmith you know it's funny but I can't help but ponder at the global combined number of developer hours and GPU hours wasted making glass đ¤Ş
Dr. Juande Santander-Vela@stroughtonsmith @alexr for true retro you need a disco ball!
Alex Rosenberg@stroughtonsmith Chefâs kiss (and why canât Genmoji do the obvious here)
Sim@stroughtonsmith App Store screenshot
Gabriel N@stroughtonsmith the guy from the design team that left cleaned his desk and took all that glass crap away.
Philipp@stroughtonsmith do you think codex is better than Claude Code?
@eierund I've never used Claude and I haven't found a reason to yet
George@stroughtonsmith context starts to fall off at around 260k iirc, itâs very easy to pollute so be mindful of that if you do see some issues.
5.4 by all accounts is a huge step up
5.4 by all accounts is a huge step up
brandon@stroughtonsmith Have you peeked under the hood at much of the code for this project? I saw you mention that you were completely hands off with the code for this one, but I'd definitely be curious to get your take on the overall code quality you're seeing from these models.
@brndnsh people keep asking me that. It writes code to the standard you ask it to write to. It's been following my style guide, so all of its code looks like mine.
Except for the Metal renderer, that's voodoo.
Andrew Burwell@stroughtonsmith It looks like you're quickly progressing towards an eventual release. I think you're doing more than playing around at this point. haha
Metin Seven đ¨
caseyneiba đŚŁ@stroughtonsmith Petition to rename all A Series SoC as M Series "Mini".
A14 -> M1 Mini
A18 Pro -> M4 Mini
The benchmarks don't lie.
Greg Gardner@stroughtonsmith only available on M3+ iPads, though đŠ
@gregggreg oh no this runs on devices before that. I think it needs an M1/A14-class device or higher
@stroughtonsmith using GPU in background tasks requires M3+ iPad I thought?
@gregggreg do you have a source for that? I havenât seen that documented
@stroughtonsmith https://developer.apple.com/forums/thread/797538?answerId=854825022#854825022
"...which is that the iPhone 16 Pro does not support background GPU. I don't know of anywhere we formally state exactly which devices support it, but I believe it's only support on iPad's with an M3 or better (and not supported on any iPhone)."
Paul-Vincent Roll (he/him)@stroughtonsmith I mostly use Shapr3D on my iPad for CAD stuff I am so much faster than on my Mac with the same app.
@stroughtonsmith itâs great to use with the pencil and as you said touch for navigation.
@stroughtonsmith Youâre going to have Blender on iPad before Blender gets Blender on iPad.
Iâm not as familiar with the crawling abilities of these agents, but Blender is open source. Could you not just say âHey, grab the array tool from Blender and add it here?â
@TheEjj in theory, yes you could hand it the Blender source code. But you would be deep in uncharted legal territory with Blenderâs license. Granted, people already believe that about AI anyway
@stroughtonsmith You wouldnât catch me doing it! But I know the Blender on iPad project was garnering a lot of enthusiasm, and it seems like some adventurous enthusiast could get it going similar to how you have your own 3D tool here.
Kevin Conner@stroughtonsmith If you're interested in avoiding the unstable appearance of two coplanar polygons, like the center square in this overlap, a term you can prompt with is Z-fighting, as in, "I see Z-fighting where the two front sides overlap." The term always made me think of Power Rangers.
@easeout I think z-fighting is appropriate here when overlapping objects, Blender does the same. It probably also helps rather than hinders when lining up a boolean operation?
@stroughtonsmith Hmm! Yes maybe this is a Worse is Better situation
@stroughtonsmith Itâs crazy to see how something like this can evolve. Do you find it weirdly addictive to see what the GPUs will cook up next?
@brndnsh it's just a string of dopamine hits
@stroughtonsmith So many âone more promptâ momentsâŚ
Max Mellman@stroughtonsmith âtsWork â26â
Arno Appenzeller@stroughtonsmith out of curiosity. Do you use Codex from inside Xcode or do you use the CLI?
My results in generating functional iOS projects with the CLI have been very mixed
My results in generating functional iOS projects with the CLI have been very mixed
@arno_app this is all using the Codex app, so 'neither' I guess?
@stroughtonsmith oh wow I was avoiding it since it felt like overkill first. Now you got me curios!
@arno_app I avoided it too until I started doing things that were more filesystem based, and I haven't looked back â it feels so much more capable even with the same model
@stroughtonsmith I found the CLI to be a suboptimal startpoint to create new projects so I went back to the built in Xcode agents. However they feel a bit more limited (even though I don't know if they are actually) but I on the other hand they benefit from the built in docs and so on.
After seeing what you built with it I will try the app as well.
@arno_app so what I do is I create a new project, with my custom Xcode template for UIKit apps (https://github.com/steventroughtonsmith/appleuniversal-xctemplates)
I also drop in a markdown file with my style guide.
Then I give that folder to Codex, and from then on Xcode is basically just a fancy build and run button
@stroughtonsmith Thanks a lot thatâs a helpful starting point.
Seems Xcode project structure is still a hard enough task so that an agent can't solve it completely đ