What we’ve seen of visionOS is really impressive. But thinking about the bits that are currently missing…

HUDs aren’t shown. Maybe they look weird when they aren’t anchored to your space. Maybe their applications are no good if other people in the space can’t see them.

But I wanna see more AR applications for drivers and pilots and surgeons.

Likewise we never saw any in-person shared experiences.

Whether multiple Vision devices would need to coordinate wirelessly (and know their relative position in meatspace), or whether it’d just be a FaceTime call with someone in the same room, you have to assume Apple’s own industrial designers and mechanicals designers want to collaborate on 3D models.

We also only ever saw indoor AR.

Easy to imagine R1 getting overwhelmed when the environment is changing, at speed or with environmental noise like leaves on the wind, or snowfall, but you have to imagine SlopesVision by @parrots would be *incredible*

TLDR outdoor mobile social AR is what everyone *sees* is the ideal use case (hovering nametags at a party, etc) but the state of the art is indoor standstill solo AR

Still really want one though…

Will be really interesting to watch the hardware platform and the software platform evolve to meet the use cases that are still out of reach.

@clarko I don't think this is the hardware for outdoor or industrial use, but really the only thing stopping multi person/same room is software, and you gotta imagine they're working on it.

The interesting thing to me is that IMO no other apple device has had such a huge post-launch change in how it was designed to be used. It's hard to even think of an example, maybe like if the iPad could become a table where four people could work, but only with a software update. Absurd!

@Soroush this *specific* hardware no, of course. But we’re all wishcasting in multiple directions. I guess the $6000 Vision Ultra is the one for extreme sports.

I think iPad’s transformation from “big iPod touch” to “Wacom all-in-one” to “touchscreen laptop” is a pretty huge set of changes. But that’s all opt-in with additional hardware. The base case is unchanged.

I hate to think how quickly this hardware will evolve, and how much money we’ll drop on upgrades.

@clarko agreed. I’m basically ready to buy one, but I am not ready to make this yearly expense lol
@Soroush @clarko I’ll point out that WatchOS 1.0 was centered around Digital Touch messages. One of the hardware buttons was dedicated to the circular contacts list. All that went away.
@clarko @parrots 3.5k is quite the motivating to not crash while skiing
@clemens @parrots On the other hand “it looks like you had a fall and hit your head hard, calling mountain rescue in 5…”
@clarko @parrots what happens if you run out of battery while going down a slope?
@fernandroid @parrots One more reason outdoor AR seems better suited to future hardware

More on this: proximity-based SharePlay is needed just to solve the case of your boss walking up to your desk and asking for a demo. Or swiveling around and asking a coworker for some help on a problem.

Otherwise you gotta go to different rooms and set up a meeting to present your work and talk through it.

Seems like the kind of feature that’s *so* important for office work that you can’t even imagine Apple putting headsets on employees’ desks until it’s solved.

Back to WFH everybody!

@clarko that reminds me of one other thought i've been having: how do i share a screen (or an item in space) with one person, but keep other stuff private? if i have messages open as well as the xcode window i'm pairing with another dev on, how does the ui assure me that they can see some stuff i can see but not all of it?
@Soroush Big overlap with the existing Mac screencasting API
@Soroush @clarko the Design Spatial SharePlay Experiences session shows that windows that are being shared are clearly marked with an ornament. Also mentioned that there is a limitation of one shared window or full volumetric space at a time.
@clarko biggest thing I missed was pinning things on walls. Like, the calendar lives in the kitchen, the tv is always above the fireplace.
@bradellis yeah that’s on my wishlist but it seems like that’ll be part of some eventual WidgetKit update. Everything in 1.0 is an app that you open and close.

@clarko @bradellis along the lines of a calendar on the wall: spatial HomeKit would be awesome! 🤩 Look at a light and pinch to turn it on/off. Flick to open the blinds.

(Like this but better https://apps.apple.com/app/devices-control-for-homekit/id966877433)

‎Devices – Control for HomeKit

‎Devices is the Apple HomeKit app that lets you manage and control smart home devices quickly and intuitively. Each device is represented by a specific icon. There is no complicated menu structure – all devices are shown in an overview and can be sorted by room or device type. In addition, the app bo…

App Store
@lexpostma @clarko @bradellis We created this quick prototype for HomeKit control using Pushcut on the Magic Leap 2 headset https://www.linkedin.com/feed/update/urn:li:activity:7072400395494424576/
Oliver Weidlich on LinkedIn: Light and Music Spatial Computing Prototype

Light and Music Spatial Computing Prototype

@clarko and I think they already did too much for 1.0!
@bradellis @clarko Here’s a prototype we did for pinning reminders to the door with data feeds etc before you head out. We were exploring proxemics for reducing the visual clutter until it’s relevant. Lot’s more refinement to play with content and preview states etc. Magic Leap 2 again :-) Can’t wait to get our hands on Vision Pro! https://www.contxtu.al//digital-in-reality/leaving-reminders
Leaving Reminders

3

Contxtual Website
@bradellis @clarko That’s an interesting idea, something I guess that HoloLens does. But if you pin your calendar to your office wall then go get a coffee in the kitchen, is your calendar trapped in the office? I suppose it could know you’ve left the office and have it accessible in a stack of pinned apps, then put them back to their pinned locations when you’re back inside that space? I’m not surprised that’s not a 1.0 thing. Sounds like a tricky thing to pull off for something we don’t know if most people will care about.
@clarko We’ve got one video prototype we explored a HUD for Focus Mode switches a while ago (but based on walking around with headset on) https://www.contxtu.al/apple-ar/apple's-focus-mode
Apple's Focus Mode

Contxtual Website

@clarko I think it’s very interesting that apps are (to some extent) still confined to the window borders. There’s clearly some iteration to be done on the software development story.

But this is very early days.

@clarko The category that as far as I can tell won't be supported in v1 but I'm excited about are the “let you add healthy nudges to the environment" ones I wrote about a few years back (https://allenpike.com/2019/the-supremacy-of-ar). Things like “make the recycling bin glow the evening before recycling pickup” and "show our list of dinners we bought ingredients for on the fridge at dinner time”. Will be more useful when the device is better suited to wearing throughout the day.
The Coming Supremacy of AR

How augmented reality is far more than a phone on your face.

Allen Pike

@apike Oh yeah we’re gonna need some object permanence before we get that.

Same as pinning things to the walls — I’m guessing visionOS doesn’t know or care that the objects in my home are meaningfully different to me than the ones in my hotel room.

@clarko Yes, though Andy Matuschak pointed out that within a given app, ARKit should let you pin a thing to your home wall, and it won't be there in your hotel but the app could restore it when you return home. The problem is, you'd need to launch the "grocery list on fridge" app to make this happen. So there needs to be a bit more of a visionOS-level plugin system to manage these various persistent things, and the resulting battery impact. https://developer.apple.com/videos/play/wwdc2023/10082/?time=567
Meet ARKit for spatial computing - WWDC23 - Videos - Apple Developer

Discover how you can use ARKit's tracking and scene understanding features to develop a whole new universe of immersive apps and games...

Apple Developer
@apike yeah I was saying to Brad that’s probably a WidgetKit thing for another year