Zhuowei ZhangJul 7, 2023
What's the best way to encode a video with an alpha channel?
Show threadZhuowei Zhang
The Internet says the best way is to encode two videos, one video holding YUV and one video holding the alpha channel, then during decoding time recombine. Not sure how to interleave them in .
(Or I could just key out one colour, but visionOS's e.g. app icon grid probably won't look good without full alpha...)
Jul 7, 2023 at 8:27amWeb
Show threadAttila TörökJul 7, 2023
@zhuowei That's exactly how "VP6 with alpha" in Flash works. The color is premultiplied, and alpha is a separate greyscale stream. There's a constraint there that any frame in the color stream that is a keyframe (I frame) must also be a keyframe in the alpha stream. Other than that VP8/VP9 in WebM can handle alpha natively it seems...?
Show threadAttila TörökJul 7, 2023
@zhuowei This is how it works in SWF/FLV:
Show threadKeith AhernJul 7, 2023
@zhuowei I played around with this using OBS and chroma keying out colours while looking at blocks of colour in the sim. Due to the inherent alpha blending of the app icon grid and frosted glass panels it didn't look great. Are you able to isolate the layers pre-composition?
Show threadKeith AhernJul 7, 2023
@zhuowei OBS made the chroma keyed stream available as a virtual camera device, I then streamed that over webRTC to the Magic Leap which was also sending gyro events. It was laggy and I haven't gone back to optimise.
Show threadZhuowei ZhangJul 8, 2023
@keithahern Figured it out: I can just hook RealityKit to prevent it from adding composeSyntheticEnvironment.rerendergraph, and I get a PNG of the visionOS UI without the background scene.. https://notnow.dev/notice/AXTQmrNkveomxqdIrw
Zhuowei Zhang: “My visionOS stereo screenshot library can now take screenshots with a transparent background: https://github.com/zhuowei/VisionOSStereoScreenshots/tree/transparent-background Next: stream this t...”

Zhuowei Zhang (@[email protected]): “My visionOS stereo screenshot library can now take screenshots with a transparent background: https://github.com/zhuowei/VisionOSStereoScreenshots/tree/transparent-background Next: stream this t...”

Show threadKeith AhernJul 8, 2023
@zhuowei that’s fantastic, did it also free up some CPU/GPU time?
Show threadZhuowei ZhangJul 7, 2023
@keithahern Not sure yet: I'll have to take a look. I remember visionOS supports rendering environment to a separate layer (see createPatchAndUpdateWithFrameDescription:(etc)), so might be able to use that.