Posts under Spatial Computing topic

Post

Replies

Boosts

Views

Activity

ARKit Face Tracking works in total darkness?
I’ve seen, mainly in discussions with AIs, that ARFaceTrackingConfiguration uses the same technology as Face ID and therefore should work in complete darkness. However, I haven’t been able to achieve this. Does anyone know if this is actually true? I'm using an iPhone 16 to test, and the Face ID works well in darkness.
1
1
277
3d
Is a volumetric window recommended for menus in 3D-focused visionOS apps?
For a visionOS app whose main content is 3D, such as a game or immersive experience, is it recommended to use a volumetric window as the main menu or in-game menu? I’d like to understand the intended use cases for menus built as volumetric windows. Specifically: What are the advantages of using a volumetric window for menu UI compared with a regular 2D window or an in-scene SwiftUI attachment? What limitations should developers be aware of, such as fixed size, placement behavior, lighting separation, interaction comfort, or window management? For a 3D game-like app, is a volumetric menu generally considered a good visionOS design pattern, or should volumetric windows be reserved for 3D content rather than menu-heavy UI? Any guidance on the recommended design approach would be appreciated.
4
1
100
3d
Tracking with a single rear camera
M4 iPad Pros only have a single camera rear camera, are the older iPad Pro models assisting the tracking with the ultra-wide camera? We noticed that a AR Session recording records all rear and front cameras, but covering the ultra-wide doesn't impact the tracking in our testing. Would a partial cover (at the corners not the center) of the ultra-wide lens POV impact the tracking as well?
1
0
43
3d
Lack of documentation on RealityKit
SceneKit was perfect to import 3D assets directly into our app scene. Since it's soft deprecation and the coming out of RealityKit, tutorials or documentation on how to bridge old possibilities are missing, everything seems to be about AR. More specifically, I am not able to find any resources on how to create a scene, load 3D assets, add a light and a camera, and everything within the app itself. No augmented reality. Is it intended to add some support about this? Any guidance would be appreciated.
2
1
91
3d
What is the latest recommendations and best practices for using 3-D content creation, tools, and importing 3-D objects into reality composer pro 3
Currently, I have a very basic entity in RCP that I call in my app's immersive view based on an included Package. I'm trying to get a much better 3d asset to use in my app instead. I've been afraid to put much effort into learning Blender to create and export an USDx asset for import into RCP3 (it's a whole world of learning itself...). I may be able to start using agents like Codex tied into Blender to see if I can more quickly create a 3d asset. I tried using the new AI assistant in RCP3 and it seems like I could create a 3d entity, but I'm not sure it will be the level of detail I want. Any high-level advice for the workflow (nowadays with AI agents) to create a desirable 3d asset, get it into RCP3, and then into Xcode?
3
0
106
3d
Why are polygon count limits still important in ImmersiveSpace with foveated rendering?
I have a question about rendering performance guidelines for visionOS ImmersiveSpace. Since visionOS uses foveated rendering, why are polygon count and scene complexity still treated as strict performance constraints for immersive content? My understanding is that foveated rendering reduces rendering cost outside the user’s central field of view. If so, should polygon count mainly matter near the gaze/foveal region, while objects in the peripheral area are much cheaper to render? Specifically: Does foveated rendering reduce only pixel shading cost, or does it also significantly reduce geometry processing cost? Are polygons outside the foveal region still submitted, culled, transformed, and rasterized in a way that affects CPU/GPU performance? For large immersive environments, should developers still optimize total scene polygon count, or focus mainly on what appears near the user’s gaze? Are there recommended guidelines for LOD, culling, and polygon budgets in visionOS immersive spaces, even when foveated rendering is enabled? I’d like to better understand how foveated rendering affects geometry budgets, and how developers should think about scene complexity for immersive visionOS apps. Any guidance would be appreciated.
2
0
61
3d
Motion Blur during ARKit Sessions
When not sunny in outdoor environments during an ARKit session, captured frames often suffer from motion blur due to automatically selected longer shutter speeds. What are the best practices for prioritizing faster shutter speeds to reduce motion blur? Are there specific APIs or recommended configurations to safely balance the low-light/shutter speed trade-off without disrupting tracking?
1
0
61
3d
Production LiDAR capture app affected by ARKit world-tracking drift on iOS 26.4+ (re: thread 827240) mitigation guidance and fix timeline?
Hello, and thank you for taking questions this week. We develop DeepWalk, a production App Store application that captures LiDAR depth, RGB, and ARKit camera poses while field technicians walk continuous outdoor routes (sidewalk and infrastructure assessment). Recorded ARCamera.transform values are used downstream for 3D reconstruction, so world-tracking accuracy directly determines our output quality. Problem: Beginning in early April 2026 and coinciding with the iOS 26.4 rollout across our device fleet, we observe directional, accumulating pose drift in recorded trajectories during normal walking motion. The behavior matches the regression reported in forum thread 827240 (world-tracking drift on LiDAR devices on iOS 26.4+, absent on non-LiDAR devices running identical code). Link to existing post here: https://developer.apple.com/forums/thread/827240 Our configuration is minimal: a default ARWorldTrackingConfiguration with frameSemantics = [.sceneDepth, .smoothedSceneDepth] and no other options set, running on a bare ARSession with a Metal renderer (no SceneKit/RealityKit). We measure drift directly from per-frame ARCamera.transform values, so the displacement appears in ARKit's pose output itself rather than in any rendering layer. The drift accumulates with walking distance and does not occur when the device is stationary. We've observed this across a large variety of devices (iPhone 15/16/17 Pro/Pro Max) on all iOS versions 26.4 or greater. We've also observed this on iPad Pros with updated firmware. Thread 827240 additionally reports that disabling autofocus, HDR, and plane detection and varying scene reconstruction and world alignment did not resolve it. Question Because depth capture is our core function, we cannot avoid engaging the LiDAR scanner, and we're not aware of a public API that adjusts its contribution to world tracking. We'd be grateful for guidance on three questions: Is this a known issue in iOS 26.4 and later? Did the LiDAR contribution to world tracking change in that release? For apps that require sceneDepth, is there any recommended session configuration, video format, or capture-side practice that reduces the drift in the interim? Is a fix planned, and if so, should we expect it in an iOS 26.x update or the iOS 27 cycle? This affects how we manage our deployed device fleet. We're happy to provide trajectory datasets, additional sysdiagnoses, or anything else useful. We have quantified drift-per-distance data across OS versions.
2
0
152
3d
Reality Composer 3 iOS versions support
I'm building an iOS game with RealityKit + Reality Composer Pro and was hoping to migrate to Reality Composer Pro 3. I created a new project in RCP and used the "link a new xcode project" flow. It built and rendered correctly on iOS 27. However, I encountered two issues when trying to build for iOS 26. The new RealityKitScripting package threw an error indicating <iOS 27 is not supported. I removed the related code, as I do not intend on using any Script Graphs. The iOS 27 scene still loaded correctly after removing this package. Are there any other consequences to removing this package other than not being able to use Script Graphs? Once the scene loading was working on iOS 26, I noticed that no materials built with the Shader Graph would render. Unlit and Physically Based materials render correctly, but Shader Graph materials do not. Even a simple Shader Graph with one Unlit material node input would not work. Is this expected behavior? My main question is, should I not expect to be able to use Reality Composer Pro 3 if I'm going to target an OS version <27?
0
0
84
3d
Automated testing via session replay
Is there a supported route to automated regression testing of ARKit apps? Reality Composer's "Record AR Session" plus Xcode's "ARKit Replay data" scheme option work well for manual debugging, but replay isn't wired into XCUITest, doesn't run in CI, and ARKit doesn't exist in the simulator — so every AR regression today needs to be run by a human holding a device. Even replay-driven ARSession in the simulator, or an XCTest hook for selecting replay files, would unlock some automated coverage.
2
0
106
4d
website environment and visionOS 27 beta
Is the website environment feature available on this first visionOS 27 beta? I am having troubles converting a web page that used the visionOS 26 website environment approach to the new visionOS 27 approach. Before I keep working on it, I would like confirmation that the feature is working. Also, does Apple have any web pages featuring a website environment that I can inspect?
3
0
78
4d
SharePlay connection issues between gen 1 and gen 2 vision pro's
SharePlay Connectivity Issue: Asymmetric Connection Failures Between Vision Pro Generations Environment: Device 1: 2nd generation Vision Pro Device 2: 1st generation Vision Pro OS Version: visionOS 6 build 23O471 Custom App Version: Both devices running identical version Both devices fully updated to the same OS and app versions Issue Description: SharePlay connections exhibit asymmetric behavior based on invitation direction, despite successful FaceTime connectivity between devices: 2nd gen → 1st gen: SharePlay invitation fails consistently 1st gen → 2nd gen: SharePlay invitation succeeds consistently. FaceTime calls: Successful in both directions Impact: Users cannot reliably initiate SharePlay sessions from newer generation devices to older generation devices. The issue is isolated to SharePlay functionality, as standard FaceTime connectivity works bidirectionally and both devices run identical versions of the custom app.
1
2
576
4d
LowLevelRenderer: visionOS custom shaders, composition with RealityView, custom surface shader ABI
A few questions about the intended usage of LowLevelRenderer / LowLevelRenderContextStandalone for custom rendering (custom Metal materials, dynamic meshes, custom splat compositing): visionOS custom shaders in immersive space LowLevelRenderer is listed as available on visionOS 27. Does that include user-authored Metal surface shaders via makeSurfaceShader when rendering in an immersive space, or are custom fragment shaders restricted there the way CustomMaterial was? Any foveation / Compositor Services constraints we should know about? Composition with RealityView Is LowLevelRenderer (via LowLevelRenderContextStandalone) intended only for fully standalone/offscreen rendering where the caller composites the output, or can it inject draws into an existing RealityView's managed render pass? On visionOS specifically, what's the supported way to get its output onto the drawable alongside RealityKit-rendered content with correct depth interaction (mutual occlusion between custom draws and normal entities)? Custom surface shader ABI For makeSurfaceShader with a user-authored Metal function: what's the function signature contract? Specifically — which [[buffer(n)]] / [[texture(n)]] slots are reserved by the renderer vs. available to the caller, how do camera and per-instance uniforms arrive in the shader, and how do LowLevelArgumentTable bindings map to argument indices? A pointer to sample code or header documentation for the shader-authoring contract would be ideal. Hot reload / resource lifetime For an authoring tool that recompiles shaders frequently: can LowLevelRenderContextStandalone.Resources be rebuilt incrementally, and what's the cost of re-init via the synchronous init(resources:) path? Any guidance on reusing resources across context instances? Relationship to GaussianSplatComponent Is GaussianSplatComponent built on LowLevelRenderer internally? For 4D / animated splats and counts above the component's internal limit, is implementing custom splat rendering directly on LowLevelRenderer a path you'd recommend, or discourage? Thank you!
3
3
122
4d
Cloth on skinned characters
Can the new cloth sim be driven on a skinned character mesh — a cape pinned to the shoulders following skeletal animation — and does it run at interactive rates on visionOS, or is it aimed at non-real-time/hero content? Can a sim be cached/baked for deterministic playback?
2
0
104
4d
ARKit Face Tracking works in total darkness?
I’ve seen, mainly in discussions with AIs, that ARFaceTrackingConfiguration uses the same technology as Face ID and therefore should work in complete darkness. However, I haven’t been able to achieve this. Does anyone know if this is actually true? I'm using an iPhone 16 to test, and the Face ID works well in darkness.
Replies
1
Boosts
1
Views
277
Activity
3d
Is a volumetric window recommended for menus in 3D-focused visionOS apps?
For a visionOS app whose main content is 3D, such as a game or immersive experience, is it recommended to use a volumetric window as the main menu or in-game menu? I’d like to understand the intended use cases for menus built as volumetric windows. Specifically: What are the advantages of using a volumetric window for menu UI compared with a regular 2D window or an in-scene SwiftUI attachment? What limitations should developers be aware of, such as fixed size, placement behavior, lighting separation, interaction comfort, or window management? For a 3D game-like app, is a volumetric menu generally considered a good visionOS design pattern, or should volumetric windows be reserved for 3D content rather than menu-heavy UI? Any guidance on the recommended design approach would be appreciated.
Replies
4
Boosts
1
Views
100
Activity
3d
Tracking with a single rear camera
M4 iPad Pros only have a single camera rear camera, are the older iPad Pro models assisting the tracking with the ultra-wide camera? We noticed that a AR Session recording records all rear and front cameras, but covering the ultra-wide doesn't impact the tracking in our testing. Would a partial cover (at the corners not the center) of the ultra-wide lens POV impact the tracking as well?
Replies
1
Boosts
0
Views
43
Activity
3d
Import Blender material to Reality Composer Pro
What are the best practices for import from Blender? I have a material that looks beautiful in Blender, but my attempts at migrating it to Reality Composer Pro had very poor results. An image of the nodes making up the material is attached, along with a rendering from Blender and RCP.
Replies
1
Boosts
2
Views
51
Activity
3d
Lack of documentation on RealityKit
SceneKit was perfect to import 3D assets directly into our app scene. Since it's soft deprecation and the coming out of RealityKit, tutorials or documentation on how to bridge old possibilities are missing, everything seems to be about AR. More specifically, I am not able to find any resources on how to create a scene, load 3D assets, add a light and a camera, and everything within the app itself. No augmented reality. Is it intended to add some support about this? Any guidance would be appreciated.
Replies
2
Boosts
1
Views
91
Activity
3d
What is the latest recommendations and best practices for using 3-D content creation, tools, and importing 3-D objects into reality composer pro 3
Currently, I have a very basic entity in RCP that I call in my app's immersive view based on an included Package. I'm trying to get a much better 3d asset to use in my app instead. I've been afraid to put much effort into learning Blender to create and export an USDx asset for import into RCP3 (it's a whole world of learning itself...). I may be able to start using agents like Codex tied into Blender to see if I can more quickly create a 3d asset. I tried using the new AI assistant in RCP3 and it seems like I could create a 3d entity, but I'm not sure it will be the level of detail I want. Any high-level advice for the workflow (nowadays with AI agents) to create a desirable 3d asset, get it into RCP3, and then into Xcode?
Replies
3
Boosts
0
Views
106
Activity
3d
Questions about VD B and some best implementation suggestions for simulations
Hello, I would like to inquire about recommendations regarding the performance overhead of VDB simulation on the M5 Vision Pro. Additionally, I would like suggestions for the best implementation for simulating rivers, waterfalls, snowflakes, and flames.
Replies
1
Boosts
0
Views
48
Activity
3d
Why are polygon count limits still important in ImmersiveSpace with foveated rendering?
I have a question about rendering performance guidelines for visionOS ImmersiveSpace. Since visionOS uses foveated rendering, why are polygon count and scene complexity still treated as strict performance constraints for immersive content? My understanding is that foveated rendering reduces rendering cost outside the user’s central field of view. If so, should polygon count mainly matter near the gaze/foveal region, while objects in the peripheral area are much cheaper to render? Specifically: Does foveated rendering reduce only pixel shading cost, or does it also significantly reduce geometry processing cost? Are polygons outside the foveal region still submitted, culled, transformed, and rasterized in a way that affects CPU/GPU performance? For large immersive environments, should developers still optimize total scene polygon count, or focus mainly on what appears near the user’s gaze? Are there recommended guidelines for LOD, culling, and polygon budgets in visionOS immersive spaces, even when foveated rendering is enabled? I’d like to better understand how foveated rendering affects geometry budgets, and how developers should think about scene complexity for immersive visionOS apps. Any guidance would be appreciated.
Replies
2
Boosts
0
Views
61
Activity
3d
Motion Blur during ARKit Sessions
When not sunny in outdoor environments during an ARKit session, captured frames often suffer from motion blur due to automatically selected longer shutter speeds. What are the best practices for prioritizing faster shutter speeds to reduce motion blur? Are there specific APIs or recommended configurations to safely balance the low-light/shutter speed trade-off without disrupting tracking?
Replies
1
Boosts
0
Views
61
Activity
3d
Production LiDAR capture app affected by ARKit world-tracking drift on iOS 26.4+ (re: thread 827240) mitigation guidance and fix timeline?
Hello, and thank you for taking questions this week. We develop DeepWalk, a production App Store application that captures LiDAR depth, RGB, and ARKit camera poses while field technicians walk continuous outdoor routes (sidewalk and infrastructure assessment). Recorded ARCamera.transform values are used downstream for 3D reconstruction, so world-tracking accuracy directly determines our output quality. Problem: Beginning in early April 2026 and coinciding with the iOS 26.4 rollout across our device fleet, we observe directional, accumulating pose drift in recorded trajectories during normal walking motion. The behavior matches the regression reported in forum thread 827240 (world-tracking drift on LiDAR devices on iOS 26.4+, absent on non-LiDAR devices running identical code). Link to existing post here: https://developer.apple.com/forums/thread/827240 Our configuration is minimal: a default ARWorldTrackingConfiguration with frameSemantics = [.sceneDepth, .smoothedSceneDepth] and no other options set, running on a bare ARSession with a Metal renderer (no SceneKit/RealityKit). We measure drift directly from per-frame ARCamera.transform values, so the displacement appears in ARKit's pose output itself rather than in any rendering layer. The drift accumulates with walking distance and does not occur when the device is stationary. We've observed this across a large variety of devices (iPhone 15/16/17 Pro/Pro Max) on all iOS versions 26.4 or greater. We've also observed this on iPad Pros with updated firmware. Thread 827240 additionally reports that disabling autofocus, HDR, and plane detection and varying scene reconstruction and world alignment did not resolve it. Question Because depth capture is our core function, we cannot avoid engaging the LiDAR scanner, and we're not aware of a public API that adjusts its contribution to world tracking. We'd be grateful for guidance on three questions: Is this a known issue in iOS 26.4 and later? Did the LiDAR contribution to world tracking change in that release? For apps that require sceneDepth, is there any recommended session configuration, video format, or capture-side practice that reduces the drift in the interim? Is a fix planned, and if so, should we expect it in an iOS 26.x update or the iOS 27 cycle? This affects how we manage our deployed device fleet. We're happy to provide trajectory datasets, additional sysdiagnoses, or anything else useful. We have quantified drift-per-distance data across OS versions.
Replies
2
Boosts
0
Views
152
Activity
3d
Reality Composer 3 iOS versions support
I'm building an iOS game with RealityKit + Reality Composer Pro and was hoping to migrate to Reality Composer Pro 3. I created a new project in RCP and used the "link a new xcode project" flow. It built and rendered correctly on iOS 27. However, I encountered two issues when trying to build for iOS 26. The new RealityKitScripting package threw an error indicating <iOS 27 is not supported. I removed the related code, as I do not intend on using any Script Graphs. The iOS 27 scene still loaded correctly after removing this package. Are there any other consequences to removing this package other than not being able to use Script Graphs? Once the scene loading was working on iOS 26, I noticed that no materials built with the Shader Graph would render. Unlit and Physically Based materials render correctly, but Shader Graph materials do not. Even a simple Shader Graph with one Unlit material node input would not work. Is this expected behavior? My main question is, should I not expect to be able to use Reality Composer Pro 3 if I'm going to target an OS version <27?
Replies
0
Boosts
0
Views
84
Activity
3d
Automated testing via session replay
Is there a supported route to automated regression testing of ARKit apps? Reality Composer's "Record AR Session" plus Xcode's "ARKit Replay data" scheme option work well for manual debugging, but replay isn't wired into XCUITest, doesn't run in CI, and ARKit doesn't exist in the simulator — so every AR regression today needs to be run by a human holding a device. Even replay-driven ARSession in the simulator, or an XCTest hook for selecting replay files, would unlock some automated coverage.
Replies
2
Boosts
0
Views
106
Activity
4d
website environment and visionOS 27 beta
Is the website environment feature available on this first visionOS 27 beta? I am having troubles converting a web page that used the visionOS 26 website environment approach to the new visionOS 27 approach. Before I keep working on it, I would like confirmation that the feature is working. Also, does Apple have any web pages featuring a website environment that I can inspect?
Replies
3
Boosts
0
Views
78
Activity
4d
SharePlay connection issues between gen 1 and gen 2 vision pro's
SharePlay Connectivity Issue: Asymmetric Connection Failures Between Vision Pro Generations Environment: Device 1: 2nd generation Vision Pro Device 2: 1st generation Vision Pro OS Version: visionOS 6 build 23O471 Custom App Version: Both devices running identical version Both devices fully updated to the same OS and app versions Issue Description: SharePlay connections exhibit asymmetric behavior based on invitation direction, despite successful FaceTime connectivity between devices: 2nd gen → 1st gen: SharePlay invitation fails consistently 1st gen → 2nd gen: SharePlay invitation succeeds consistently. FaceTime calls: Successful in both directions Impact: Users cannot reliably initiate SharePlay sessions from newer generation devices to older generation devices. The issue is isolated to SharePlay functionality, as standard FaceTime connectivity works bidirectionally and both devices run identical versions of the custom app.
Replies
1
Boosts
2
Views
576
Activity
4d
LowLevelRenderer: visionOS custom shaders, composition with RealityView, custom surface shader ABI
A few questions about the intended usage of LowLevelRenderer / LowLevelRenderContextStandalone for custom rendering (custom Metal materials, dynamic meshes, custom splat compositing): visionOS custom shaders in immersive space LowLevelRenderer is listed as available on visionOS 27. Does that include user-authored Metal surface shaders via makeSurfaceShader when rendering in an immersive space, or are custom fragment shaders restricted there the way CustomMaterial was? Any foveation / Compositor Services constraints we should know about? Composition with RealityView Is LowLevelRenderer (via LowLevelRenderContextStandalone) intended only for fully standalone/offscreen rendering where the caller composites the output, or can it inject draws into an existing RealityView's managed render pass? On visionOS specifically, what's the supported way to get its output onto the drawable alongside RealityKit-rendered content with correct depth interaction (mutual occlusion between custom draws and normal entities)? Custom surface shader ABI For makeSurfaceShader with a user-authored Metal function: what's the function signature contract? Specifically — which [[buffer(n)]] / [[texture(n)]] slots are reserved by the renderer vs. available to the caller, how do camera and per-instance uniforms arrive in the shader, and how do LowLevelArgumentTable bindings map to argument indices? A pointer to sample code or header documentation for the shader-authoring contract would be ideal. Hot reload / resource lifetime For an authoring tool that recompiles shaders frequently: can LowLevelRenderContextStandalone.Resources be rebuilt incrementally, and what's the cost of re-init via the synchronous init(resources:) path? Any guidance on reusing resources across context instances? Relationship to GaussianSplatComponent Is GaussianSplatComponent built on LowLevelRenderer internally? For 4D / animated splats and counts above the component's internal limit, is implementing custom splat rendering directly on LowLevelRenderer a path you'd recommend, or discourage? Thank you!
Replies
3
Boosts
3
Views
122
Activity
4d
Curved windows in visionOS 27
Safari, Freeform and some other apps now support curved windows. It this now available to developers as well? I couldn't find any information about it. Thanks!
Replies
6
Boosts
1
Views
177
Activity
4d
I can’t complete assistant setup up in Reality Composer Pro 3 beta
I can’t complete assistant setup up in Reality Composer Pro 3 beta. I filled in the set up fields for OpenAI and clicked done. However, when I click connect nothing happens.
Replies
12
Boosts
1
Views
181
Activity
4d
WindowGrop How to customize bending styles?
How should I set the window of WindowGrop to resemble a curved screen style?
Replies
2
Boosts
0
Views
658
Activity
4d
Cloth on skinned characters
Can the new cloth sim be driven on a skinned character mesh — a cape pinned to the shoulders following skeletal animation — and does it run at interactive rates on visionOS, or is it aimed at non-real-time/hero content? Can a sim be cached/baked for deterministic playback?
Replies
2
Boosts
0
Views
104
Activity
4d
Gaussian Splats, Ingestion + Mesh Interop
The new GaussianSplatComponent: what's the supported ingestion path — standard .ply splats, .spz, or a conversion step inside RCP? And can a splat coexist in a scene with mesh geometry and depth-sort/occlude correctly against it, or are splats a separate pass with no depth interaction with meshes?
Replies
2
Boosts
1
Views
73
Activity
4d