We were having an issue wrb the system rotate and scale gestures (two-handed gestures / RotateGesture3D and MagnifyGesture) were extremely difficult to register (make work) in the visionOS simulator.
The solution we found was to:
Launch your app in the simulator
Move the pointer on top of the 3D object for which you are testing rotation and scaling gestures.
Press and hold the Option key to display touch points (ie: the two-handed gesture points).
While maintaining the option key pressed, release the pointer and re-enable it again. I am using a track pad with tap-to-click enabled and three-finger to drag enabled in accessibility, so "release the pointer and re-enable it again" translates simply to removing the three finger and placing them again on the trackpad.
If you have maintained the option key pressed, then you should now be able to rotate and scale the 3D object.
Context if you are interested:
Our issue was also occurring in Apple's own sample project relating to gestures "Transforming RealityKit entities using gestures", at below link.
On Apple's article "Interacting with your app in the visionOS simulator" at the below link, for two-handed gestures it states "Press and hold the Option key to display touch points. Move the pointer while pressing the Option key to change the distance between the touch points. Move the pointer and hold the Shift and Option keys to reposition the touch points."
This simply did not work anymore for rotation and scaling gestures.
These gestures used to be a lot more responsive in Sonoma. Either the article should be updated to what I described above, or there is an issue. Our colleague who is using macOS Sonoma 14.6.1 with the latest release of Xcode is not having these issues.
Here is the list of configurations (troubleshooting we tried!) where it is difficult to achieve rotation and scaling gestures in the visionOS simulator:
macOS Sequoia 16.1 Beta, Xcode 16.1 RC w visionOS 2.1
macOS Sequoia 16.1 Beta, Xcode 16.1 RC w visionOS 2.0
macOS Sequoia 16.1 Beta, Xcode 16.2 Beta 1 w visionOS 2.1
macOS Sequoia 16.1 Beta, Xcode 16.2 Beta 1 w visionOS 2.0
macOS Sequoia 16.1 Beta, remove all Xcodes and installed the build from AppStore (Xcode 16.1)
macOS Sequoia 16.1 Beta, Xcode 16.0 w visionOS 2.0
completely wiped out, and reset entire development machine, re-installed latest releases of sequoia (15.1) and xcode (15.1))
Throughout these troubleshooting I often:
restarted both xcode and sim
erased all derived data
erased all contents and settings from sims
performed fresh git clones
None of the above worked, only the workaround described above works atm. As you can maybe deduce, it was very time consuming to find the workaround, we also wasted some development effort thinking our gesture development was no-good.
Hopefully this will help other devs.
Article Link:
https://developer.apple.com/documentation/xcode/interacting-with-your-app-in-the-visionos-simulator
Gesture sample project link:
https://developer.apple.com/documentation/realitykit/transforming-realitykit-entities-with-gestures
Discuss spatial computing on Apple platforms and how to design and build an entirely new universe of apps and games for Apple Vision Pro.
Selecting any option will automatically load the page
Post
Replies
Boosts
Views
Activity
I've been experimenting with the Muse pen and understand that it can be accessed by my app through a SpatialTrackingSession, but is there any current or planned support for devices like this as for general UI input like game controllers are? For example, using the button as a tap analogue for SwiftUI views.
Topic:
Spatial Computing
SubTopic:
General
I am building a 360 photo viewer in VisionOS 26. Which allows the user to choose a 2 by 1 jpg and then renders it with a sphere mesh entity. And I use: TextureResource(contentsOf: url, options: options).
I noticed two situations here in terms of mipmaps options.
When setting "mipmapsMode: .none":
The graphic quality within the "gaze area" looks sharp and clear
The two poles (top and bottom) are perfectly rendered
Massive shimmer around the "gaze area"
When setting "mipmapsMode: .allocateAndGenerateAll":
The graphic looks slightly blurrier than in ".none" within the "gaze area"
The two poles are very blurry and hard to recognize the texture
Much less shimmer around the "gaze area"
My question would be: Is there a way to have the perfect graphic quality in ".none" without the massive shimmer?
Thank you!
Screenshots:
mipmapsMode: .none
mipmapsMode: .allocateAndGenerateAll
Hello,
If you add a ManipulationComponent to a RealityKit entity and then continue to add instructions, sooner or later you will encounter a crash with the following error message:
Attempting to move entity “%s” (%p) under “%s” (%p), but the new parent entity is currently being removed. Changing the parent/child entities of an entity in an event handler while that entity is already being reassigned is not supported.
CoreSimulator 1048 – Device: Apple Vision Pro 4K (B87DD32A-E862-4791-8B71-92E50CE6EC06) – Runtime: visionOS 26.0 (23M336) – Device Type: Apple Vision Pro
The problem occurs precisely with this code:
ManipulationComponent.configureEntity(object)
I adapted Apple's ObjectPlacementExample and made the changes available via GitHub.
The desired behavior is that I add entities to ManipulationComponent and then Realitiykit runs stably and does not crash randomly.
GitHub Repo
Thanks
Andre
Hi everyone,
We’re developing a Unity project for Apple Vision Pro that connects PSVR2 Sense controllers for advanced interaction and input.
We’ve encountered a major limitation:
when the controller is not held close to the designated hand (e.g., resting on a table or held by the non designated hand), the Sense controller enters a low-power or reduced-update mode. This results in noticeably reduced tracking update frequency and responsiveness until the controller is held again.
For certain use cases, this behavior is undesirable. In our case, it prevents continuous real-time tracking of the controller even when it’s stationary or being tracked externally.
Request:
Please consider exposing an API flag or developer option in ARKit to disable and optionally delay the low-power mode when the app requires full-rate updates regardless of proximity or hand pose detection.
Hi, I have a hand model that is in FBX and I'm exporting it to USD in Blender. I get a skinned mesh and while I can track the whole hand how do I track each joint and assign it and animate the skinned mesh itself. All my attempts say this is not possible in RealityKit as of now. True?
Hi All,
We're a studio building an app and as part of a scene we have a 3D asset with a smoke particle emitter and a curved mesh that plays video. I notice that when the video alone is played or the particle effect alone is done then the scene works fine but the frame rate drops drastically when both are turned on.
How do I solve this because this is an important storytelling feature.
We are working on a world scale AR app that leverages the device location and heading to place objects in the streets, so that they are correctly and stably anchored to certain locations.
Since the geo-tracking imagery is only available in certain cities and areas, we are trying to figure out how to fallback when geo-tracking is not available as the device move away, to still retain good AR camera accuracy. We might need to come up with some algorithm using the device GPS, to line up the ARCamera with our objects.
Question: Does geo-tracking always provide greater than or equal to the accuracy of world tracking, for a GPS outdoor AR experience?
If so, we can simply use the ARGeoTrackingConfiguration for the entire time, and rely on the ARView keeping itself aligned. Otherwise, we need to switch between it and ARWorldTrackingConfiguration when geo-tracking is not available and/or its accuracy is low, then roll our own algorithm to keep the camera aligned.
Thanks.
I have read in the apple documentation and on forums that in order to access the camera and capture images on VisionPro, both an Entitlement and an Enterprise.license are required. I already have the Entitlement, but I don’t yet have the Enterprise.license. I would like to ask: is the Enterprise.license strictly required to gain camera access for capturing images? How can I obtain this file, and does it require an Enterprise account? Currently, my developer account is a regular Developer 99$, not an Enterprise account.
Topic:
Spatial Computing
SubTopic:
ARKit
Hello, I've pre-ordered the Logitech Muse with hopes of developing with it, but have yet to find any documentation relating to the capabilities it will have/any APIs that will be available to take advantage of the Muse. Is anyone aware of what might become available?
Thank you in advance.
Hi, I'm developing an app for Vision Pro using Xcode, while updating the latest update, things that worked in my app suddenly didn't.
in my app flow I'm tapping spheres to get their positions, from some reason I get an offset from where I tap to where a marker on that position is showing up.
here's the part of code that does that, and a part that is responsible for an alignment that happens afterwards:
func loadMainScene(at position: SIMD3) async {
guard let content = self.content else { return }
do {
let rootEntity = try await Entity(named: "surgery 16.09", in: realityKitContentBundle)
rootEntity.scale = SIMD3<Float>(repeating: 0.5)
rootEntity.generateCollisionShapes(recursive: true)
self.modelRootEntity = rootEntity
let bounds = rootEntity.visualBounds(relativeTo: nil)
print("📏 Model bounds: center=\(bounds.center), extents=\(bounds.extents)")
let pivotEntity = Entity()
pivotEntity.addChild(rootEntity)
self.pivotEntity = pivotEntity
let modelAnchor = AnchorEntity(world: [1, 1.3, -0.8])
modelAnchor.addChild(pivotEntity)
content.add(modelAnchor)
updateModelOpacity(0.5)
self.modelAnchor = modelAnchor
rootEntity.visit { entity in
print("👀 Entity in model: \(entity.name)")
if entity.name.lowercased().hasPrefix("focus") {
entity.generateCollisionShapes(recursive: true)
entity.components.set(InputTargetComponent())
print("🎯 Made tappable: \(entity.name)")
}
}
print("✅ Model loaded with collisions")
guard let sphere = placementSphere else { return }
let sphereWorldXform = sphere.transformMatrix(relativeTo: nil)
var newXform = sphereWorldXform
newXform.columns.3.y += 0.1 // move up by 20 cm
let gridAnchor = AnchorEntity(world: newXform)
self.gridAnchor = gridAnchor
content.add(gridAnchor)
let baseScene = try await Entity(named: "Scene", in: realityKitContentBundle)
let gridSizeX = 18
let gridSizeY = 10
let gridSizeZ = 10
let spacing: Float = 0.05
let startX: Float = -Float(gridSizeX - 1) * spacing * 0.5 + 0.3
let startY: Float = -Float(gridSizeY - 1) * spacing * 0.5 - 0.1
let startZ: Float = -Float(gridSizeZ - 1) * spacing * 0.5
for i in 0..<gridSizeX {
for j in 0..<gridSizeY {
for k in 0..<gridSizeZ {
if j < 2 || j > gridSizeY - 5 { continue } // remove 2 bottom, 4 top
let cell = baseScene.clone(recursive: true)
cell.name = "Sphere"
cell.scale = .one * 0.02
cell.position = [
startX + Float(i) * spacing,
startY + Float(j) * spacing,
startZ + Float(k) * spacing
]
cell.generateCollisionShapes(recursive: true)
gridCells.append(cell)
gridAnchor.addChild(cell)
}
}
}
content.add(gridAnchor)
print("✅ Grid added")
} catch {
print("❌ Failed to load: \(error)")
}
}
private func handleModelOrGridTap(_ tappedEntity: Entity) {
guard let modelRootEntity = modelRootEntity else { return }
let localPosition = tappedEntity.position(relativeTo: modelRootEntity)
let worldPosition = tappedEntity.position(relativeTo: nil)
switch tapStep {
case 0:
modelPointA = localPosition
modelAnchor?.addChild(createMarker(at: worldPosition, color: [1, 0, 0]))
print("📍 Model point A: \(localPosition)")
tapStep += 1
case 1:
modelPointB = localPosition
modelAnchor?.addChild(createMarker(at: worldPosition, color: [1, 0.5, 0]))
print("📍 Model point B: \(localPosition)")
tapStep += 1
case 2:
targetPointA = worldPosition
targetMarkerA = createMarker(at: worldPosition,color: [0, 1, 0])
modelAnchor?.addChild(targetMarkerA!)
print("✅ Target point A: \(worldPosition)")
tapStep += 1
case 3:
targetPointB = worldPosition
targetMarkerB = createMarker(at: worldPosition,color: [0, 0, 1])
modelAnchor?.addChild(targetMarkerB!)
print("✅ Target point B: \(worldPosition)")
alignmentReady = true
tapStep += 1
default:
print("⚠️ Unexpected tap on model helper at step \(tapStep)")
}
}
func alignModel2Points() {
guard let modelPointA = modelPointA,
let modelPointB = modelPointB,
let targetPointA = targetPointA,
let targetPointB = targetPointB,
let modelRootEntity = modelRootEntity,
let pivotEntity = pivotEntity,
let modelAnchor = modelAnchor else {
print("❌ Missing data for alignment")
return
}
let modelVec = modelPointB - modelPointA
let targetVec = targetPointB - targetPointA
let modelLength = length(modelVec)
let targetLength = length(targetVec)
let scale = targetLength / modelLength
let modelDir = normalize(modelVec)
let targetDir = normalize(targetVec)
var axis = cross(modelDir, targetDir)
let axisLength = length(axis)
var rotation = simd_quatf()
if axisLength < 1e-6 {
if dot(modelDir, targetDir) > 0 {
rotation = simd_quatf(angle: 0, axis: [0,1,0])
} else {
let up: SIMD3<Float> = [0,1,0]
axis = cross(modelDir, up)
if length(axis) < 1e-6 {
axis = cross(modelDir, [1,0,0])
}
rotation = simd_quatf(angle: .pi, axis: normalize(axis))
}
} else {
let dotProduct = dot(modelDir, targetDir)
let clampedDot = max(-1.0, min(dotProduct, 1.0))
let angle = acos(clampedDot)
rotation = simd_quatf(angle: angle, axis: normalize(axis))
}
modelRootEntity.scale = .one * scale
modelRootEntity.orientation = rotation
let transformedPointA = rotation.act(modelPointA * scale)
pivotEntity.position = -transformedPointA
modelAnchor.position = targetPointA
alignedModelPosition = modelAnchor.position
print("✅ Aligned with scale \(scale), rotation \(rotation)")
Topic:
Spatial Computing
SubTopic:
General
When scanning multiple rooms (10+) in a single structure using ARWorldMap for coordinate space consistency, RoomCaptureSession throws CaptureError.exceedSceneSizeLimit. The instructions here (https://developer.apple.com/documentation/roomplan/scanning-the-rooms-of-a-single-structure) provide exactly what I am doing to keep the underlying ARSession alive (by calling captureSession.stop(pause: false)) and save the results before a user moves to the next room. Scanning 11 or so rooms will cause the user to hit the exceedSceneSizeLimit error. The ARWorldMap is about 58 MB and always is around this size when hitting this issue. No anchors are present and all the data seems to be from tracking data.
On iPad devices (where I do not see this issue) the ARWorldMap grows as a significantly slower rate in size.
I save the ARWorldMap after each room is scanned and confirmed by the user. If I use the ARMap to initialize the ARSession (as described in the docs) the session will immediately error with "exceedSceneSizeLimit" once the captureSession.run() is executed. Occasionally it will allow me/the user to scan again, but either breaks mid scan or the following.
This has been working fine for the past 2 years and users have been able to scan dozens of rooms without issue. It seems only lately that it has been a problem.
I would expect the ARWorldMap to be allowed for much bigger sizes. At this point I can just about scan more area of my house with a single scan than I can when I use different captureSessions.
Few observations:
This happens on my iPhone 15 Pro Max, my iPhone 17 Pro, but not my iPad M4 (maybe memory related?). It is possible if scanning many more rooms it would happen on the iPad too.
I have tried things such as resetting the ARConfig on the underlying ARSession to reset some, but this doesn't work.
I have tried to create a new ARWorldMap and move the origin to the older map to clear out tracking data. This almost works but causes a mess of issues when a user moves at all due to the unshared coordinate space.
I believe there are three active issues regarding this: FB14454922, FB15035788, FB20642944
Could we get an update for this issue? It is a production issue and severely limits my user experience in my production application.
Hi, I am trying to load files from the Apple Vision Pro's storage into a Unity App (using Apple visionOS XR Plugin and not PolySpatial package). So far, I've tried using UnitySimpleFileBrowser and UnityStandaloneFileBrowser (both aren't made for the Vision Pro and don't work there), and then implemented my own naive file browser that at least allows me to view directories (that I can see from the App Sandbox). This is of course very limited:
Gray folders can't be accessed, the only 3 available ones don't contain anything where a user would put files through the "Files" app.
I know that an app can request access to these "Files & Folders":
So my question is: Is there a way to request this access for a Unity-built app at the moment? If yes, what do I need to do? I've looked into the generated Xcode project's "Capabilities", but did not find anything related to file access. Any help is appreciated!
I'm trying to add a feature to my app to allow a user to import items from other apps, like Safari, via the share sheet.
I've done this many times on iOS/iPadOS easily with a Share Extension. From what I can tell, Xcode tells me share extensions are not available on visionOS - though my experience on device tells me differently (It seems Reminders, Notes & more implement them somehow.) I was finally able to get it working on device only...but I can now no longer test in the simulator, and have not found a way to distribute this app.
When attempting to run on the simulator, I get this issue:
Please try again later. Appex bundle at /Users/jason/Library/Developer/CoreSimulator/Devices/09A70160-4F4F-4F5E-B679-F6F7D876D7EF/data/Library/Caches/com.apple.mobile.installd.staging/temp.6OAEZp/extracted/LaunchBar.app/PlugIns/LaunchBarShareExtension.appex with id co.swiftfox.LaunchBar.ShareExtension specifies a value (com.apple.share-services) for the NSExtensionPointIdentifier key in the NSExtension dictionary in its Info.plist that does not correspond to a known extension point.
When trying to archive an upload to test flight, I get this similar error:
Invalid Info.plist value. The value for the key 'DTPlatformName' in bundle LaunchBar.app/PlugIns/LaunchBarShareExtension.appex is invalid. (ID: 207610c7-b7e1-48be-959b-22a43cd32d16)
The app is for visionOS only - which I'm thinking might be the problem? The share extension is "Designed For iPhone" and requires me to include iPhone as a run destination. In the worst case I can build an iPhone UI for the app but I'd rather not, as it is very specific to visionOS.
Has anyone successfully launched a share extension on a visionOS only app? I have an iPad app with a share extension that shows up fine on visionOS, but the issue seems to be specifically with visionOS only apps.
Topic:
Spatial Computing
SubTopic:
General
The issue reproducible with empty project. When you run it and tap "Open immersive space" it takes a couple of minutes to respond. The issue only reproducible on real device with debugger attached. Reproducible other developers too (not specific to my environment). Issue doesn't exists in Xcode 16.
Afer initial long delay subsequent opens works fine.
Console logs:
nw_socket_copy_info [C1:2] getsockopt TCP_INFO failed [102: Operation not supported on socket]
nw_socket_copy_info getsockopt TCP_INFO failed [102: Operation not supported on socket]
Failed to set dependencies on asset 9303749952624825765 because NetworkAssetManager does not have an asset entity for that id.
void * _Nullable NSMapGet(NSMapTable * _Nonnull, const void * _Nullable): map table argument is NULL
PSO compilation completed for driver shader copyFromBufferToTexture so=0 sbpr=256 sbpi=16384 ss=(64, 64, 1) p=70 sc=1 ds=0 dl=0 do=(0, 0, 0) in 1997
XPC connection interrupted
<<<< FigAudioSession(AV) >>>> audioSessionAVAudioSession_CopyMXSessionProperty signalled err=-19224 (kFigAudioSessionError_UnsupportedOperation) (getMXSessionProperty unsupported) at FigAudioSession_AVAudioSession.m:606
Failed to load item AXCodeItem<0x14706f250> [Rank:6000] SpringBoardUIServices [AXBundle name:/System/Library/AccessibilityBundles/SpringBoardUIServices.axbundle/SpringBoardUIServices] [Platforms and Targets:{ iOS = SpringBoardUIServices; } Framework] [Excluded: (null)]. error: Error Domain=AXLoading Code=0 "URL does not exist: file:///System/Library/AccessibilityBundles/SpringBoardUIServices.axbundle" UserInfo={NSLocalizedDescription=URL does not exist: file:///System/Library/AccessibilityBundles/SpringBoardUIServices.axbundle}
Failed to load item AXCodeItem<0x14706f250> [Rank:6000] SpringBoardUIServices [AXBundle name:/System/Library/AccessibilityBundles/SpringBoardUIServices.axbundle/SpringBoardUIServices] [Platforms and Targets:{ iOS = SpringBoardUIServices; } Framework] [Excluded: (null)]. error: Error Domain=AXLoading Code=0 "URL does not exist: file:///System/Library/AccessibilityBundles/SpringBoardUIServices.axbundle" UserInfo={NSLocalizedDescription=URL does not exist: file:///System/Library/AccessibilityBundles/SpringBoardUIServices.axbundle}
[b30780-MRUIFeedbackTypeButtonWithBackgroundTouchDown] Playback timed out before completion (after 3111 ms)
Failed to set dependencies on asset 7089614247973236977 because NetworkAssetManager does not have an asset entity for that id.
If I trigger the apple rating modal in an Immersive space it appears on the ground in (0,0,0) I need it to be in front of the user like push notification perimssion does or other permissions requests.
Hi, would love for your help in that matter.
I try to get the position in space of two QR codes to make an alignment to their positions in space. The detection shows that the QR codes position is always 0,0,0 and I don't understand why. Here's my code:
import SwiftUI
import RealityKit
import RealityKitContent
struct AnchorView: View {
@ObservedObject var qrCoordinator: QRCoordinator
@ObservedObject var coordinator: ImmersiveCoordinator
let qrName: String
@Binding var startQRDetection: Bool
@State private var anchor: AnchorEntity? = nil
@State private var detectionTask: Task<Void, Never>? = nil
var body: some View {
RealityView { content in
// Add the QR anchor once (must exist before detection starts)
if anchor == nil {
let imageAnchor = AnchorEntity(.image(group: "QRs", name: qrName))
content.add(imageAnchor)
anchor = imageAnchor
print("📌 Created anchor for \(qrName)")
}
}
.onChange(of: startQRDetection) { enabled in
if enabled {
startDetection()
} else {
stopDetection()
}
}
.onDisappear {
stopDetection()
}
}
private func startDetection() {
guard detectionTask == nil, let anchor = anchor else { return }
detectionTask = Task {
var detected = false
while !Task.isCancelled && !detected {
print("🔎 Checking \(qrName)... isAnchored=\(anchor.isAnchored)")
if anchor.isAnchored {
// wait a short moment to let transform update
try? await Task.sleep(nanoseconds: 100_000_000)
let worldPos = anchor.position(relativeTo: nil)
if worldPos != .zero {
// relative to modelRootEntity if available
var posToSave = worldPos
if let modelEntity = coordinator.modelRootEntity {
posToSave = anchor.position(relativeTo: modelEntity)
print("converted to model position")
} else {
print("⚠️ modelRootEntity not available, using world position")
}
print("✅ \(qrName) detected at position: world=\(worldPos) saved=\(posToSave)")
if qrName == "reanchor1" {
qrCoordinator.qr1Position = posToSave
let marker = createMarker(color: [0,1,0])
marker.position = .zero // sits directly on QR
marker.position = SIMD3<Float>(0, 0.02, 0)
anchor.addChild(marker)
print("marker1 added")
} else if qrName == "reanchor2" {
qrCoordinator.qr2Position = posToSave
let marker = createMarker(color: [0,0,1])
marker.position = posToSave // sits directly on QR
marker.position = SIMD3<Float>(0, 0.02, 0)
anchor.addChild(marker)
print("marker2 added")
}
detected = true
} else {
print("⚠️ \(qrName) anchored but still at origin, retrying...")
}
}
try? await Task.sleep(nanoseconds: 500_000_000) // throttle loop
}
print("🛑 QR detection loop ended for \(qrName)")
detectionTask = nil
}
}
private func stopDetection() {
detectionTask?.cancel()
detectionTask = nil
}
private func createMarker(color: SIMD3<Float>) -> ModelEntity {
let sphere = MeshResource.generateSphere(radius: 0.05)
let material = SimpleMaterial(color: UIColor(
red: CGFloat(color.x),
green: CGFloat(color.y),
blue: CGFloat(color.z),
alpha: 1.0
), isMetallic: false)
let marker = ModelEntity(mesh: sphere, materials: [material])
marker.name = "marker"
return marker
}
}
Topic:
Spatial Computing
SubTopic:
General
Hi I have a monitoring app, that will take input video from uvc and process it using Metal, and eventually get a MTLTexture.
The problem I'm facing is I have to convert MTLTexture to CGImage then call TextureResource.replace, which is super slow. Metal processing speed is same as input frame rate(50pfs), but MTLTexture -> CGImage -> TextureResource only got 7fps...
Is there any way I can make it faster?
Topic:
Spatial Computing
SubTopic:
General
Tags:
Media Player
Frameworks
Media Accessibility
Core Media
We’re trying to build a custom player for Unity. For this, we’re using AVPlayer with AVPlayerItemVideoOutput to get textures. However, we noticed that playback is not smooth and the stream often freezes.
For testing, we used this 8K video:
https://deovr.com/nwfnq1
The video was played using the following code:
@objc public func playVideo(urlString: String)
{
guard let url = URL(string: urlString) else { return }
let pItem = AVPlayerItem(url: url)
playerItem = pItem
pItem.preferredForwardBufferDuration = 10.0
let pixelBufferAttributes: [String: Any] = [
kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange,
kCVPixelBufferMetalCompatibilityKey as String: true,
]
let output = AVPlayerItemVideoOutput( pixelBufferAttributes: pixelBufferAttributes )
pItem.add(output)
playerItemObserver = pItem.observe(\.status)
{
[weak self] pItem, _ in
guard pItem.status == .readyToPlay else { return }
self?.playerItemObserver = nil
self?.player.play()
}
player = AVPlayer(playerItem: pItem)
player.currentItem?.preferredPeakBitRate = 35_000_000
}
When AVPlayerItemVideoOutput is attached, the video stutters and the log looks like this:
🟢 Playback likely to keep up
🟡 Buffer ahead: 4.08s | buffer: 4.08s
🟡 Buffer ahead: 4.08s | buffer: 4.08s
🟡 Buffer ahead: -0.07s | buffer: 0.00s
🟡 Buffer ahead: 2.94s | buffer: 3.49s
🟡 Buffer ahead: 2.50s | buffer: 4.06s
🟡 Buffer ahead: 1.74s | buffer: 4.30s
🟡 Buffer ahead: 0.74s | buffer: 4.30s
🟠 Playback may stall
🛑 Buffer empty
🟡 Buffer ahead: 0.09s | buffer: 4.30s
🟠 Playback may stall
🟠 Playback may stall
🛑 Buffer empty
🟠 Playback may stall
🟣 Buffer full
🟡 Buffer ahead: 1.41s | buffer: 1.43s
🟡 Buffer ahead: 1.41s | buffer: 1.43s
🟡 Buffer ahead: 1.07s | buffer: 1.43s
🟣 Buffer full
🟡 Buffer ahead: 0.47s | buffer: 1.65s
🟠 Playback may stall
🛑 Buffer empty
🟡 Buffer ahead: 0.10s | buffer: 1.65s
🟠 Playback may stall
🟡 Buffer ahead: 1.99s | buffer: 2.03s
🟡 Buffer ahead: 1.99s | buffer: 2.03s
🟣 Buffer full
🟣 Buffer full
🟡 Buffer ahead: 1.41s | buffer: 2.00s
🟡 Buffer ahead: 0.68s | buffer: 2.27s
🟡 Buffer ahead: 0.09s | buffer: 2.27s
🟠 Playback may stall
🛑 Buffer empty
🟠 Playback may stall
When we remove AVPlayerItemVideoOutput from the player, the video plays smoothly, and the output looks like this:
🟢 Playback likely to keep up
🟡 Buffer ahead: 1.94s | buffer: 1.94s
🟡 Buffer ahead: 1.94s | buffer: 1.94s
🟡 Buffer ahead: 1.22s | buffer: 2.22s
🟡 Buffer ahead: 1.05s | buffer: 3.05s
🟡 Buffer ahead: 1.12s | buffer: 4.12s
🟡 Buffer ahead: 1.18s | buffer: 5.18s
🟡 Buffer ahead: 0.72s | buffer: 5.72s
🟡 Buffer ahead: 1.27s | buffer: 7.28s
🟡 Buffer ahead: 2.09s | buffer: 3.03s
🟡 Buffer ahead: 4.16s | buffer: 6.10s
🟡 Buffer ahead: 6.66s | buffer: 7.09s
🟡 Buffer ahead: 5.66s | buffer: 7.09s
🟡 Buffer ahead: 4.66s | buffer: 7.09s
🟡 Buffer ahead: 4.02s | buffer: 7.45s
🟡 Buffer ahead: 3.62s | buffer: 8.05s
🟡 Buffer ahead: 2.62s | buffer: 8.05s
🟡 Buffer ahead: 2.49s | buffer: 3.53s
🟡 Buffer ahead: 2.43s | buffer: 3.38s
🟡 Buffer ahead: 1.90s | buffer: 3.85s
We’ve tried different attribute settings for AVPlayerItemVideoOutput. We also removed all logic related to reading frame data, but the choppy playback still remained.
Can you advise whether this is a player issue or if we’re doing something wrong?
After writing the code, when debugging on VisionPro, the program will encounter a blocking situation when running from Xcode to VisionPro. It will take a long time for the execution information to appear on the Xcode console