My cousin implanted an itch I'd like to scratch. She does some academic psycho-neurological research that I don't fully understand. But in talking it sounds like mixed reality in the AVP might facilitate a type of research that she's pursuing. I just don't know if such a thing is currently possible with the AVP given how it treats physically moving frames of reference. The idea is to have subjects in a motion vehicle simulator rig, able to operate it and see the interior of the vehicle/gauges/etc. But at times and in specific ways, obscure the views out of the "windows" of the vehicle. The obvious problem here is that the AVP doesn't like being in a dynamically moving frame of reference. This seems to be beyond what "travel mode" is intended for. They need to bump and turn the subjects to get the responses they're studying, so not a smooth ride like on a plane or train. The interior of the simulator rig is a "known object" and can be modeled. I just watched the new video about training/tracking a hand-held object via Create ML. Could a similar approach from a hand-held object be applied to the user's surroundings - the mocked up vehicle interior and its window frames? We'd then apply an obscuring blur or even just an opaque polygon to that "window" while the simulator rig (and thus the user) is in motion? The alignment of the blur/polygon doesn't need to have perfect tracking and registration. Update rate does not need to be low millisecond, but full second updates might make motion sickness worse. Also, am I correct in inferring that object tracking should be free from major drift over the course of tens of minutes? A stretch: would it be possible to counteract the illumination from room light sources so that the movement of light/dark is reduced on the interior of the "vehicle" while it is moving? I also noticed in the new "Explore enhancements to visionOS object tracking" video that some of the effect of overlaying MR elements onto something that is visually passed through is being demonstrated on an iPhone. Could a proof of concept of this app(?) be "mocked up" using an iPhone as a basis to justify the expense of buying the AVP? Hold the phone while the rig is moving, track the interior and blur/"open" the windows as seen on the phone screen?

With the recent announcement that VisionOS 27 would support custom actively tracked tools, via arrays of embedded LEDs, I would just like to confirm if the demo given in this video here: https://developer.apple.com/videos/play/wwdc2026/283/ at minute 1:00 is indeed an actively tracked tool, or if it is passive IR. if it's passive IR (reflectance), are there any examples of how this could be achieved, i.e. can we access the onboard IR cameras directly?

Hello, we are evaluating Apple Vision Pro for an AR calibration experiment. Can BarcodeDetectionProvider on visionOS 2.0+ detect a QR code/barcode and provide its 3D position and orientation, such as a transform, so that we can align virtual content with a real-world marker? Does the BarcodeAnchor update continuously when the headset or the barcode moves? Thank you.

Hello Apple Developer Team, We are currently developing an enterprise medical navigation application for Apple Vision Pro and would like to request clarification regarding the currently available visionOS Enterprise APIs related to camera access. Our application scenario involves real-time medical/surgical navigation and instrument tracking in a professional enterprise environment. We would like to better understand the following: How many cameras on Apple Vision Pro are currently accessible through the Enterprise APIs? Which specific cameras are accessible? For example: Main RGB cameras Passthrough cameras Tracking cameras Front-facing cameras Depth sensors LiDAR or structured-light related sensors Are simultaneous multi-camera streams supported? Does the Enterprise API provide: Real-time image frames Camera intrinsic/extrinsic parameters Stereo camera data Depth information Low-latency tracking-related data Are there any restrictions regarding the use of Vision Pro cameras for: Medical navigation Surgical guidance Instrument tracking Enterprise healthcare software Is Apple Vision Pro currently permitted or recommended for medical enterprise spatial-navigation workflows under the Enterprise APIs? We would greatly appreciate any official clarification regarding the current capabilities and limitations of camera access on Apple Vision Pro for enterprise medical applications.

Hello, I am doing a personal research project around ARKit camera frames and camera calibration data. For each accepted ARFrame, I export: ARFrame.timestamp ARFrame.camera.imageResolution ARFrame.camera.intrinsics ARFrame.capturedImage The target video mode is 4K / 30 FPS. In my test captures, I noticed two things: ARFrame.camera.intrinsics is not constant across the recording. Some neighboring frames from ARFrame.capturedImage have noticeably different sharpness: one frame can look sharp, while the next frame can look blurred. Regarding intrinsics, I observe small per-frame changes in fx/fy/cx/cy. For example, in one session: fx/fy changed by approximately 36 px cx changed by approximately 1.5 px cy changed by approximately 2.3 px I understand the standard meaning of the intrinsic matrix: fx / fy = focal length in pixels cx / cy = principal point in pixels relative to the image/reference frame My main question is not about the definition of these values, but about how to correctly interpret their changes over time. In particular, I am interested in cx/cy, because the principal point is important for measurements. I would like to understand whether a change in cx/cy corresponds to a meaningful update of the camera model for the current frame, and what physical or camera-pipeline process may cause this change. Questions about ARFrame.camera.intrinsics: Are ARFrame.camera.intrinsics values expected to change between frames during a single ARWorldTrackingConfiguration session? If fx/fy changes over time, can this reflect autofocus / focus breathing, internal camera calibration updates, digital crop/scaling, stabilization-related mapping, or other camera pipeline changes? If cx/cy changes over time, should I interpret this as an updated principal point for the current ARFrame image/reference frame? Does ARKit update cx/cy to account for any internal crop, scaling, stabilization, virtual camera behavior, lens movement, or calibration changes? Is there a known physical interpretation for small cx/cy changes reported by ARFrame.camera.intrinsics, or should these values simply be treated as ARKit’s best current camera model for that frame? In an ARKit-first pipeline where focus/exposure/stabilization are managed by ARKit/camera pipeline, is the recommended approach to store ARFrame.camera.intrinsics per frame together with ARFrame.timestamp and ARFrame.camera.imageResolution? Is there any public API that exposes the reason for intrinsics changes, such as focus position, stabilization transform, crop transform, active camera constituent, lens movement, or per-frame calibration update reason? I also have a related question about ARFrame.capturedImage frame quality. In some recordings, neighboring frames can have noticeably different sharpness. For example, one frame can look sharp, while the next frame can be visibly blurred, even though the scene and camera movement are continuous. Questions about frame sharpness / camera pipeline behavior: Is frame-to-frame sharpness variation expected when using ARFrame.capturedImage as the video source? Can ARKit change focus, exposure duration, ISO, white balance, crop/scaling, or other camera pipeline parameters between frames during an ARWorldTrackingConfiguration session? Is there any public API to inspect per-frame exposure duration, ISO, focus position, lens position, stabilization state, crop/scaling transform, or other capture parameters for ARFrame.capturedImage? If some frames are sharp and neighboring frames are blurred, should this usually be interpreted as motion blur / exposure behavior / autofocus behavior, or can ARKit internal processing also affect this? Is there any recommended way in an ARKit-first pipeline to reduce frame-to-frame sharpness inconsistency, or is the only reliable approach to use an AVFoundation-first capture pipeline with locked focus/exposure/ISO/white balance? The main goal of my research is to understand how to correctly interpret ARKit-provided intrinsics over time and how much of the camera pipeline behavior is observable through public ARKit APIs. Since fx/fy/cx/cy are important for measurements, I want to know whether treating ARFrame.camera.intrinsics as the Apple-delivered per-frame camera model is the safest approach. Thank you.

Hi everyone, I’m currently using the RoomPlan API, which has been working reliably until recently. However, I’ve started encountering an intermittent error and I’m trying to understand what might be causing it. The error is triggered in the ARSession observer method: session(_ session: ARSession, didFailWithError error: Error) It has occurred on at least two devices: iPhone 14 Pro iPhone 17 Pro Here’s the full error message: ARSession failed domain=com.apple.arkit.error code=102 desc=Required sensor failed. userInfo=["NSLocalizedFailureReason": A sensor failed to deliver the required input., "NSUnderlyingError": Error Domain=AVFoundationErrorDomain Code=-11819 "Cannot Complete Action" UserInfo={NSLocalizedDescription=Cannot Complete Action, NSLocalizedRecoverySuggestion=Try again later.}, "NSLocalizedDescription": Required sensor failed.] This seems to indicate that a required sensor (likely LiDAR or camera) failed to provide input, but I’m not sure what’s causing it or why it happens only occasionally. Has anyone experienced something similar or has insight into possible causes or fixes? Thanks in advance!

ARCamera.TrackingState questions: Did the threshold or sensitivity for transitioning ARCamera.TrackingState from .normal to .limited(.excessiveMotion) or .limited(.insufficientFeatures) change between iOS 26.3 and iOS 26.4? What does "ARWorldTrackingTechnique: resource constraints [33]" mean, and is it new in iOS 26.4? Does it correspond to a tracking state degradation? Is there a way for the client to detect or respond to ARKit entering a resource-constrained mode short of the full tracking state transition — for example, a lower-level notification or a flag on ARFrame — so that apps can take protective action without interpreting it as a full tracking failure? ARCamera.unprojectPoint questions: Did the behavior of ARCamera.unprojectPoint(_:ontoPlane:orientation:viewportSize:) change between iOS 26.3 and iOS 26.4 for near-parallel geometry? Specifically, on iOS 26.3 this method returns nil when the camera ray is nearly parallel to the target plane (denominator of the ray-plane intersection → 0 at ~90° of camera rotation). On iOS 26.4, with identical code and environment, it returns a large finite value instead — we observed z = −12.27m. Since the method's optional return type implies nil is the documented signal for no valid intersection, this reads as a behavioral regression rather than an intentional change. If returning the computed value for near-parallel geometry is now the intended behavior, is there a recommended way for the caller to guard against it? For example, should we check abs(dot(rayDirection, planeNormal)) against a threshold before calling, and if so, is there a documented epsilon Apple uses internally? Alternatively, is there a newer API we should prefer over unprojectPoint(:ontoPlane:) for this use case that handles degenerate geometry more gracefully — such as ARSession.raycast(:)? Are there any other ARKit API adjustments between OS 26.3 and 26.4? We are using the same codebase, but it behaves differently in between these 2 OS versions now. Thanks!

I have two RealityView: ParentView and When click the button in ParentView, ChildView will be shown as full screen cover, but the camera feed in ChildView will not be shown, only black screen. If I show ChildView directly, it works with camera feed. Please help me on this issue? Thanks. import RealityKit import SwiftUI struct ParentView: View{ @State private var showIt = false var body: some View{ ZStack{ RealityView{content in content.camera = .virtual let box = ModelEntity(mesh: MeshResource.generateSphere(radius: 0.2),materials: [createSimpleMaterial(color: .red)]) content.add(box) } Button("Click here"){ showIt = true } } .fullScreenCover(isPresented: $showIt){ ChildView() .overlay( Button("Close"){ showIt = false }.padding(20), alignment: .bottomLeading ) } .ignoresSafeArea(.all) } } import ARKit import RealityKit import SwiftUI struct ChildView: View{ var body: some View{ RealityView{content in content.camera = .spatialTracking } } }

I have an iOS app that uses RealityView to display some models and interact with them, and the app uses regular iOS app navigations, then a challenge I'm facing is how to maintain multiple RealityView on multiplescreens. For example Screen A has a RealityView, and then I navigate to Screen B (also has a RealityView) using stack based navigation, when I do so I got a crash -[MTLDebugRenderCommandEncoder validateCommonDrawErrors:]:5970: failed assertion `Draw Errors Validation Fragment Function(fsRealityPbr): argument envProbeTable[0] from Buffer(7) with offset(0) and length(16) has space for 16 bytes, but argument has a length(864). Fragment Function(fsRealityPbr): incorrect type of texture (MTLTextureType2D) bound at Texture binding at index 20 (expect MTLTextureTypeCubeArray) for envProbeDiffuseArray[0]. Interestingly this crash only happens when debugging with Xcode, not happens when the app runs on its own. I'm not sure what I'm doing is anti-pattern or it's some Xcode debugging limitation.

I use ARKit for motion tracking. I get the skeleton joint coordinates and use them for animation. I didn't make any changes to the code, but I updated the iOS version from 18 to 26, and modelTransform now always returns nil. https://developer.apple.com/documentation/arkit/arskeleton3d/modeltransform(for:) For example bodyAnchor.skeleton.modelTransform(for: .init(rawValue: "head_joint")) bodyAnchor is ARBodyAnchor. I see the default skeleton on the screen, but now I can't get the coordinates out of it. I'm using an example from Apple's WWDC presentation. https://developer.apple.com/documentation/arkit/capturing-body-motion-in-3d Are there any changes in the API? Or just bug?

Hi, is there a way to track feet in a visionOS app in an immersive space? I want the whole body to be visible in VR, and I want to know if the user touches an object with their foot.

I have already add liscene and entitlement for Passthrough in screen capture, but still get black background instaed of real world. Do you have any idea ? I've been stuck with this question for a long time 😢

I am using AccessoryTrackingProvider from ARKit to get the transform of the PSVR2 controller via originFromAnchorTransform of the AccessoryAnchor. I also am trying to use AnchorEntity on the controller using RealityKit However, none of the three options for Accessory.LocationName, which should be used to define the AnchorEntity target, seem to match the position on the controller which is being sent from ARKit. The picture attached is showing two transforms: RealityKit - using .gripSurface to define the AnchoringComponent.Target.accesssory location. ARKit - using originFromAnchorTransform for AccessoryTrackingProvider. They are not aligned at the same point. As for the other options of Accessory.LocationName, using .aim is located at the tip of the controller and .grip is the same position as .gripSurface but with a different orientation. I am wondering why there is not an option for Accessory.LocationName that actually matches the transform captured by ARKit?

I have a ModelEntity with GroundingShadowComponent entity.enumerateHierarchy { child, stop in child.components.set(GroundingShadowComponent(castsShadow: true)) } When I set it on the table, I can see the shadow on the table, even if I disable plane detection. However, when I enable plane detection, and the plane's material is OcclusionMaterial. I can not see the shadow on the table. As far as I know, receivesDynamicLighting is not usable in VisionOS. So how can I cast shadow on OcclusionMaterial in VisionOS? Or rather, is it possible to have the shadow properly displayed on the tabletop while ensuring that I cannot see objects beneath the table through it?

Dear Apple Team, I’m a high school student (vocational upper secondary school) working on my final research project about LiDAR sensors in smartphones, specifically Apple’s iPhone implementation. My current understanding (for context): I understand Apple’s LiDAR uses dToF with SPAD detectors: A VCSEL laser emits pulses, a DOE splits the beam into a dot pattern, and each spot’s return time is measured separately → point cloud generation. My specific questions: How many active projection dots does the LiDAR projector have in the iPhone 15 Pro vs. iPhone 12 Pro? Are the dots static or do they shift/move over time? How many depth measurement points does the system deliver internally (after processing)? What is the ranging accuracy (cm-level precision) of each measurement point? Experimental background: Using an IR night vision camera, I counted approximately 111 dots on the 15 Pro vs. 576 dots on the 12 Pro. Do these match the internal specifications? Photos of my measurements are available if helpful. Contact request: I would be very grateful if you could connect me with an Apple engineer or ARKit specialist who works with LiDAR technology. I would love to ask follow-up questions directly and would be happy to provide my contact details for this purpose. These specifications would be essential for my research paper. Thank you very much in advance! Best regards, Max! Vocational Upper Secondary School Hans-Leipelt-Schule Donauwörth Research Project: “LiDAR Sensor Technology in Smartphones”

ARSession provides video stream from the wide angle camera. If ARSession uses the ultra wide camera at the same time, ARSession may provide video stream from that camera, otherwise AVCaptureSession with an ultra wide camera should be allowed to launch. It would be very very useful if we can access different cameras while ARSession is running. We'd like to cooperate with you if possible. Steps to reproduce: run an AVCaptureSession and then run an ARSession. The AVCaptureSession stops.

Hardware Specifications Regarding the LiDAR scanner in the iPhone 13/14/15/16/17 Pro series, could you please provide the following technical details for academic verification: Point Cloud Density / Resolution: The effective resolution of the depth map. Sampling Frequency: The sensor's refresh rate. Accuracy Metrics: Official tolerance levels regarding depth accuracy relative to distance (specifically within 0.5m – 2m range). Data Acquisition Methodology For a scientific thesis requiring high data integrity: Does Apple recommend a custom ARKit implementation over third-party applications (e.g., Polycam) to access raw depth data? I need to confirm if third-party apps typically apply smoothing or post-processing that would obscure the sensor's native performance, which must be avoided for my error analysis.

https://developer.apple.com/documentation/arkit/arpointcloud https://developer.apple.com/documentation/arkit/arframe/rawfeaturepoints The point cloud (collection of points/features) main intention is a debug visualization to what the underlying tracking algorithm processes and is not designed for additional algorithms on top of that. But, we are utilizing information contained in the points/features collected by ARKit. Currently, the range of rawfeaturepoints is limited to about 10 meter from the device. We see a great chance if the range is unlock. The global localization will be more robust and accurate. ARPointCloud - Apple ARKit - FindSurface YouTube SIdQRiLj2jY

The samples shown in volumetric work great but moving to an immersive experience the pen physical buttons don't work when you're focusing to an entity with a collision.

Error: RoomCaptureSession.CaptureError.exceedSceneSizeLimit Apple Documentation Explanation: An error that indicates when the scene size grows past the framework’s limitations. Issue: This error is popping up in my iPhone 14 Pro (128 GB) after a few roomplan scans are done. This error shows up even if the room size is small. It occurs immediately after I start the RoomCaptureSession after the relocalisation of previous AR session (in world tracking configuration). I am having trouble understanding exactly why this error shows and how to debug/solve it. Does anyone have any idea on how to approach to this issue?