Explore the integration of media technologies within your app. Discuss working with audio, video, camera, and other media functionalities.

All subtopics
Posts under Media Technologies topic

Post

Replies

Boosts

Views

Activity

Core Image intermediate rendering for full-extent lower-resolution kernel inputs
I render images using a tiled/LOD pipeline. The final render usually requests only the visible ROI at the current display scale, which works well for local filters. The difficult case is filters that need a lower-resolution full-extent or larger-extent input, such as guided filters, blurs, histograms/statistics, and reduce passes. If that dependency remains inside the same CI graph, ROI propagation can force larger-extent evaluation of the upstream image, defeating small-ROI rendering. My current approach is to render those intermediates in a separate lower-resolution CI render pass, cache the result, and feed the rendered intermediate back into the final graph. Is this the recommended architecture, or is there a better Core Image structure for expressing this kind of intermediate render/cache barrier without manually creating and rendering a separate intermediate graph? Thanks!
2
0
62
2d
CIRAWFilter isHighlightRecoveryEnabled
Could you provide a more detailed description of how isHighlightRecoveryEnabled affects the image, and how it differs from isGamutMappingEnabled? I’d like to explain these controls accurately to my users. Does highlight recovery run before or after linearSpaceFilter, and should it be disabled when generating scene-referred linear output? Thanks!
4
0
74
2d
CI_PRINT_TREE: debugging intermediate cache hits
In WWDC20 it was mentioned that missing compileTime/renderTime (or no intermediate image with the dump-intermediates option) can indicate a cached intermediate. I have also seen renderTime=cacheHit on passes that are clearly rendering intermediates, and an actual renderTime is only properly shown on the final pass which always seems to equal the final time render time. In compileTime it says cacheMiss. Which fields should be considered authoritative for tracking intermediate cache hits versus actual rendering? Thanks!
3
0
34
2d
CIRAWFilter isDraftModeEnabled
In my testing, enabling isDraftModeEnabled improves decode performance for very large non-RAW images. Does it also affect decoding performance or quality for RAW images? Are there non-RAW formats that benefit more than others, or not at all, with draft mode? For non-RAW images, is it generally recommended to load them through CIRAWFilter or CIImage? If CIImage is preferred, is there an equivalent to isDraftModeEnabled for CIImage initialization? Conversely, if CIRAWFilter is acceptable for non-RAW images, how can I get the expandToHDR behavior that is available through CIImage options? Thanks!
3
0
48
2d
RAW Boost filter / EDR
When processing RAW images with CIRAWFilter's extendedDynamicRange, does the boost filter’s behavior change? How does boost handle extended range values and does boost have a limit before clamping? Also, in the RAW EDR case, should we still use the linearSpace filter or is it now fine to run filters post-boost (assuming the filters are linear-dependent and can run post-boost)
2
0
26
2d
CIImage compression to JPEG
I am trying to compress a CIImage below a certain file size threshold when creating a JPEG. Currently, I am handling this in, potentially, multiple passes. In the first pass, I am essentially calling:
encodeJPEG(CIImage, quality: 0.85, colorSpace: CIImage.colorSpace ?? CGColorSpace(name: CGColorSpace.sRGB)!)
I am then checking the image size. If the image is still too big, I try to estimate how much I need to reduce it and try again until the image size is small enough. My question: Is there a more efficient way of doing this?
3
0
33
2d
Core Image processing for a photo overlaid text effect
The iPhone Lock Screen lets you display a small time and date at the top of the display that has a neat image processing effect where the text takes on the underlying colors with a blurred gradient while still maintaining legibility appearing largely white. In this screenshot you can see the white text features blue colors at the top left and transitions to a purple color on the right due to the underlying vibrant blue and purple blocks. Can you share how this text effect is achieved using Core Image?
1
0
22
2d
CIRAWFilter decoderVersion scope and RAW 9
I’m really excited about RAW 9, great work! Since RAW 9 is not enabled by default, are there any recommended use cases where it makes sense to select it by default, if supported? For compatibility handling, should I treat decoderVersion as covering all processing and adjustments performed by CIRAWFilter, or does it mostly affect the RAW decoding stage? When switching decoder versions, should I re-evaluate which CIRAWFilter adjustments are supported? Also, the macOS 26 headers appear to expose version9 and version9DNG. Is RAW 9 actually supported on macOS 26 runtimes? Thanks!
2
0
59
2d
CIRAWFilter despeckleAmount
CIRAWFilter documentation lists despeckleAmount as available on macOS 12.0+, but I cannot find this property in the macOS 26 headers. Could you provide a detailed description of how despeckleAmount affects the image so I can explain this adjustment accurately to my users? Does it run before or after linearSpaceFilter, and what should it be set to when generating scene-referred linear output? Thanks!
2
1
43
2d
RAW 9: High watermark for memory?
What is the high water mark for rendering an image in RAW 9 vs. RAW 8? I realize it depends on the megapixels of the render, but what can you share memory wise, especially compared to RAW 8, perhaps as a multiple? I'm mostly concerned about iOS in case that's relevant.
1
1
29
2d
Channel Mapping with AUHAL
Hi, our Mac app uses AUHAL for audio output. The data we’re playing has channels laid out in the WAVE default ordering (this is the same order as Core Audio’s AudioChannelBitmap). Playing this data usually results in the correct channel playing through the correct speaker, but there are cases where it does not: Outputting audio over HDMI usually results in the Center and LFE channels being swapped Or “Configure Speakers” in Audio MIDI Setup.app can be used to change the speaker<->channel mapping (i.e. even swapping L/R on a 2ch stereo setup), but this does not affect our app’s audio I’ve found that getting kAudioUnitProperty_AudioChannelLayout on an output unit returns the speaker<->channel mapping as set up in Audio MIDI Setup (i.e. it shows that Center/LFE swap, or if L/R are swapped). My question: is it possible to have Core Audio do the channel remapping between the WAVE default ordering, and the speaker<->channel mapping as returned by kAudioUnitProperty_AudioChannelLayout? I have tried setting kAudioUnitProperty_AudioChannelLayout on the input unit, and didn’t see a change. I’ve also tried to set kAudioOutputUnitProperty_ChannelMap on the output unit but cannot get it to work. Or do I need to remap the channels myself before giving samples to Core Audio?
4
0
123
2d
Supporting custom headers in CMCD
Today we are on-boarded to using CMCD data to allow us to get more diagnostic data of our streaming. We would like to have the ability to upload our own custom headers, this is supported on other players, but not supported on AVPlayer. The workaround today that works is adding in custom query parameters then querying based on that, but we would like support explicitly in the headers.
2
1
91
2d
PHAssetResource.originalFilename can be nil despite non-optional Swift API
In our PhotoKit backup/export code, we need to read PHAssetResource.originalFilename for each resource. The Swift API imports this property as non-optional: var originalFilename: String { get } However, we have customer reports from real Photos libraries where reading this property appears to crash because the underlying Objective-C value is nil at runtime. This affects actual users of our shipping app, not just a synthetic test case. The relevant code path is essentially this: import Photos func inspectAsset(_ asset: PHAsset) { let resources = PHAssetResource.assetResources(for: asset) for resource in resources { // This property is imported as non-optional String. // For some affected customer assets, reading it appears to trap/crash // because the underlying value is nil. let filename: String = resource.originalFilename print(filename) } } This makes the API difficult to use safely from Swift: because the property is non-optional, there is no normal Swift optional-handling path before the crash. We filed this as FB22589474. We also filed a related Feedback, FB22519412, for empty-string / otherwise unusable filename values. That is a separate issue, but it points in the same direction: apps that export, back up, or sync Photos resources cannot treat PHAssetResource.originalFilename as a guaranteed usable filename. For PHAssetResource.originalFilename, what is the intended contract?
3
0
82
2d
FairPlay multi key encryption in Safari on macOS
Is FairPlay multi key encryption supposed to work in Safari 26 on macOS? It stopped working for us some time last year. The very same content works on iOS in AVPlayer and single key encryption works fine. On macOS, the multi key content sometimes plays, sometimes not. It is as if there was some race condition on how fast the KSM responds with content keys. Very often, I can see just one KSM request in network console (instead of multiple for each video profile and audio) and the playback fails with Video error: code=3 msg=Media failed to decode. We've filed a Code Level Support ticket (19025185) and a bug report (FB22353469), but haven't really heard back. Just that engineering is actively investigating this issue, and was able to reproduce it.
1
1
52
2d
Updating downloaded manifest
Following up on: https://feedbackassistant.apple.com/feedback/22501602 We have a downloaded .movpkg (via AVAggregateAssetDownloadTask) where we need to update #EXT-X-DATERANGE tag URLs in the manifest Apple Developer Technical Support (FB15965814) confirmed "it is perfectly fine to update the m3u8 playlist inside the movpkg if that works well for you." Can you confirm this is safe to rely on long-term and won't break in future OS updates? Is there anything we should watch out for (e.g., movpkg integrity checks, asset download storage manager interactions)?
1
0
41
2d
AVAssetResourceLoaderDelegate and error handling
If we are using an AVAssetResourceLoaderDelegate, and performing our own playlist requests using a URLSession, what are some best practices for responding to the resource loading requests with errors? In other words, how best to populate the NSError domain, code, and userInfo values when calling finishLoadingWithError? SImilarly, are there any things to look out for when the URLSession has caching enabled, and we receive HTTP 304s for live playlists? The URLSession appears to transparently handle the 304, and return a 200 with the cached data in the URLSession callbacks. But could there be edge cases where the URLSession’s cache is becoming invalid just as we get a 304?
3
0
88
2d
How many concurrent VTCompressionSession / VTDecompressionSession can an app run, and can the limit be queried?
I'm batch-transcoding a library of clips into downscaled editing proxies. If I kick off two hardware transcodes at once, the encoder reliably falls over, so right now I run everything serially: each clip gets fully decoded, encoded, and muxed before the next one starts. It works, but it's slow, and the decode and encode hardware are mostly idle waiting on each other. A few things I can't pin down from the docs: Is there an actual ceiling on how many VTCompressionSession / VTDecompressionSession instances can be live at once, and does it depend on the device, the codec, or the resolution? Can I query that ceiling at runtime? I'd rather size my concurrency up front than find it by crashing. Decode and encode are separate hardware blocks, so can I safely run a decode session for one clip while the previous clip is still encoding, or does VideoToolbox serialize them anyway? When I do go over the limit, what should I be checking so I can back off cleanly? Right now I just get a crash instead of an error I can catch. Anything that gets me off the fully-serial pipeline would help. Thank you
5
0
215
2d
Core Image intermediate rendering for full-extent lower-resolution kernel inputs
I render images using a tiled/LOD pipeline. The final render usually requests only the visible ROI at the current display scale, which works well for local filters. The difficult case is filters that need a lower-resolution full-extent or larger-extent input, such as guided filters, blurs, histograms/statistics, and reduce passes. If that dependency remains inside the same CI graph, ROI propagation can force larger-extent evaluation of the upstream image, defeating small-ROI rendering. My current approach is to render those intermediates in a separate lower-resolution CI render pass, cache the result, and feed the rendered intermediate back into the final graph. Is this the recommended architecture, or is there a better Core Image structure for expressing this kind of intermediate render/cache barrier without manually creating and rendering a separate intermediate graph? Thanks!
Replies
2
Boosts
0
Views
62
Activity
2d
CIRAWFilter isHighlightRecoveryEnabled
Could you provide a more detailed description of how isHighlightRecoveryEnabled affects the image, and how it differs from isGamutMappingEnabled? I’d like to explain these controls accurately to my users. Does highlight recovery run before or after linearSpaceFilter, and should it be disabled when generating scene-referred linear output? Thanks!
Replies
4
Boosts
0
Views
74
Activity
2d
CI_PRINT_TREE: debugging intermediate cache hits
In WWDC20 it was mentioned that missing compileTime/renderTime (or no intermediate image with the dump-intermediates option) can indicate a cached intermediate. I have also seen renderTime=cacheHit on passes that are clearly rendering intermediates, and an actual renderTime is only properly shown on the final pass which always seems to equal the final time render time. In compileTime it says cacheMiss. Which fields should be considered authoritative for tracking intermediate cache hits versus actual rendering? Thanks!
Replies
3
Boosts
0
Views
34
Activity
2d
RAW 9 bugs
Here are two additional RAW 9 bugs: Horizontal Line in RAW 9 decode: FB23076983 CIRAWFilter.scaleFactor not respected in RAW 9 for some RAW formats: FB23076984
Replies
1
Boosts
1
Views
36
Activity
2d
CIRAWFilter isDraftModeEnabled
In my testing, enabling isDraftModeEnabled improves decode performance for very large non-RAW images. Does it also affect decoding performance or quality for RAW images? Are there non-RAW formats that benefit more than others, or not at all, with draft mode? For non-RAW images, is it generally recommended to load them through CIRAWFilter or CIImage? If CIImage is preferred, is there an equivalent to isDraftModeEnabled for CIImage initialization? Conversely, if CIRAWFilter is acceptable for non-RAW images, how can I get the expandToHDR behavior that is available through CIImage options? Thanks!
Replies
3
Boosts
0
Views
48
Activity
2d
RAW Boost filter / EDR
When processing RAW images with CIRAWFilter's extendedDynamicRange, does the boost filter’s behavior change? How does boost handle extended range values and does boost have a limit before clamping? Also, in the RAW EDR case, should we still use the linearSpace filter or is it now fine to run filters post-boost (assuming the filters are linear-dependent and can run post-boost)
Replies
2
Boosts
0
Views
26
Activity
2d
Grounding and bounding boxes and segmentation
Which of the Vision framework‘s support for grounding/bounding boxes/segmentation carry over to the iOS 27 on-device Foundation Model image support?
Replies
1
Boosts
0
Views
28
Activity
2d
CIImage compression to JPEG
I am trying to compress a CIImage below a certain file size threshold when creating a JPEG. Currently, I am handling this in, potentially, multiple passes. In the first pass, I am essentially calling:
encodeJPEG(CIImage, quality: 0.85, colorSpace: CIImage.colorSpace ?? CGColorSpace(name: CGColorSpace.sRGB)!)
I am then checking the image size. If the image is still too big, I try to estimate how much I need to reduce it and try again until the image size is small enough. My question: Is there a more efficient way of doing this?
Replies
3
Boosts
0
Views
33
Activity
2d
Core Image processing for a photo overlaid text effect
The iPhone Lock Screen lets you display a small time and date at the top of the display that has a neat image processing effect where the text takes on the underlying colors with a blurred gradient while still maintaining legibility appearing largely white. In this screenshot you can see the white text features blue colors at the top left and transitions to a purple color on the right due to the underlying vibrant blue and purple blocks. Can you share how this text effect is achieved using Core Image?
Replies
1
Boosts
0
Views
22
Activity
2d
CIRAWFilter decoderVersion scope and RAW 9
I’m really excited about RAW 9, great work! Since RAW 9 is not enabled by default, are there any recommended use cases where it makes sense to select it by default, if supported? For compatibility handling, should I treat decoderVersion as covering all processing and adjustments performed by CIRAWFilter, or does it mostly affect the RAW decoding stage? When switching decoder versions, should I re-evaluate which CIRAWFilter adjustments are supported? Also, the macOS 26 headers appear to expose version9 and version9DNG. Is RAW 9 actually supported on macOS 26 runtimes? Thanks!
Replies
2
Boosts
0
Views
59
Activity
2d
CIRAWFilter despeckleAmount
CIRAWFilter documentation lists despeckleAmount as available on macOS 12.0+, but I cannot find this property in the macOS 26 headers. Could you provide a detailed description of how despeckleAmount affects the image so I can explain this adjustment accurately to my users? Does it run before or after linearSpaceFilter, and what should it be set to when generating scene-referred linear output? Thanks!
Replies
2
Boosts
1
Views
43
Activity
2d
RAW 9: High watermark for memory?
What is the high water mark for rendering an image in RAW 9 vs. RAW 8? I realize it depends on the megapixels of the render, but what can you share memory wise, especially compared to RAW 8, perhaps as a multiple? I'm mostly concerned about iOS in case that's relevant.
Replies
1
Boosts
1
Views
29
Activity
2d
RAW 9: System Requirements
What is the minimum iPhone system requirement for RAW 9? Anything that can run iOS 27 / iPadOS 27 / macOS 27? I was getting some ANE errors on an M1 MacBook Air this morning (FB23103074)
Replies
2
Boosts
0
Views
32
Activity
2d
Channel Mapping with AUHAL
Hi, our Mac app uses AUHAL for audio output. The data we’re playing has channels laid out in the WAVE default ordering (this is the same order as Core Audio’s AudioChannelBitmap). Playing this data usually results in the correct channel playing through the correct speaker, but there are cases where it does not: Outputting audio over HDMI usually results in the Center and LFE channels being swapped Or “Configure Speakers” in Audio MIDI Setup.app can be used to change the speaker<->channel mapping (i.e. even swapping L/R on a 2ch stereo setup), but this does not affect our app’s audio I’ve found that getting kAudioUnitProperty_AudioChannelLayout on an output unit returns the speaker<->channel mapping as set up in Audio MIDI Setup (i.e. it shows that Center/LFE swap, or if L/R are swapped). My question: is it possible to have Core Audio do the channel remapping between the WAVE default ordering, and the speaker<->channel mapping as returned by kAudioUnitProperty_AudioChannelLayout? I have tried setting kAudioUnitProperty_AudioChannelLayout on the input unit, and didn’t see a change. I’ve also tried to set kAudioOutputUnitProperty_ChannelMap on the output unit but cannot get it to work. Or do I need to remap the channels myself before giving samples to Core Audio?
Replies
4
Boosts
0
Views
123
Activity
2d
Supporting custom headers in CMCD
Today we are on-boarded to using CMCD data to allow us to get more diagnostic data of our streaming. We would like to have the ability to upload our own custom headers, this is supported on other players, but not supported on AVPlayer. The workaround today that works is adding in custom query parameters then querying based on that, but we would like support explicitly in the headers.
Replies
2
Boosts
1
Views
91
Activity
2d
PHAssetResource.originalFilename can be nil despite non-optional Swift API
In our PhotoKit backup/export code, we need to read PHAssetResource.originalFilename for each resource. The Swift API imports this property as non-optional: var originalFilename: String { get } However, we have customer reports from real Photos libraries where reading this property appears to crash because the underlying Objective-C value is nil at runtime. This affects actual users of our shipping app, not just a synthetic test case. The relevant code path is essentially this: import Photos func inspectAsset(_ asset: PHAsset) { let resources = PHAssetResource.assetResources(for: asset) for resource in resources { // This property is imported as non-optional String. // For some affected customer assets, reading it appears to trap/crash // because the underlying value is nil. let filename: String = resource.originalFilename print(filename) } } This makes the API difficult to use safely from Swift: because the property is non-optional, there is no normal Swift optional-handling path before the crash. We filed this as FB22589474. We also filed a related Feedback, FB22519412, for empty-string / otherwise unusable filename values. That is a separate issue, but it points in the same direction: apps that export, back up, or sync Photos resources cannot treat PHAssetResource.originalFilename as a guaranteed usable filename. For PHAssetResource.originalFilename, what is the intended contract?
Replies
3
Boosts
0
Views
82
Activity
2d
FairPlay multi key encryption in Safari on macOS
Is FairPlay multi key encryption supposed to work in Safari 26 on macOS? It stopped working for us some time last year. The very same content works on iOS in AVPlayer and single key encryption works fine. On macOS, the multi key content sometimes plays, sometimes not. It is as if there was some race condition on how fast the KSM responds with content keys. Very often, I can see just one KSM request in network console (instead of multiple for each video profile and audio) and the playback fails with Video error: code=3 msg=Media failed to decode. We've filed a Code Level Support ticket (19025185) and a bug report (FB22353469), but haven't really heard back. Just that engineering is actively investigating this issue, and was able to reproduce it.
Replies
1
Boosts
1
Views
52
Activity
2d
Updating downloaded manifest
Following up on: https://feedbackassistant.apple.com/feedback/22501602 We have a downloaded .movpkg (via AVAggregateAssetDownloadTask) where we need to update #EXT-X-DATERANGE tag URLs in the manifest Apple Developer Technical Support (FB15965814) confirmed "it is perfectly fine to update the m3u8 playlist inside the movpkg if that works well for you." Can you confirm this is safe to rely on long-term and won't break in future OS updates? Is there anything we should watch out for (e.g., movpkg integrity checks, asset download storage manager interactions)?
Replies
1
Boosts
0
Views
41
Activity
2d
AVAssetResourceLoaderDelegate and error handling
If we are using an AVAssetResourceLoaderDelegate, and performing our own playlist requests using a URLSession, what are some best practices for responding to the resource loading requests with errors? In other words, how best to populate the NSError domain, code, and userInfo values when calling finishLoadingWithError? SImilarly, are there any things to look out for when the URLSession has caching enabled, and we receive HTTP 304s for live playlists? The URLSession appears to transparently handle the 304, and return a 200 with the cached data in the URLSession callbacks. But could there be edge cases where the URLSession’s cache is becoming invalid just as we get a 304?
Replies
3
Boosts
0
Views
88
Activity
2d
How many concurrent VTCompressionSession / VTDecompressionSession can an app run, and can the limit be queried?
I'm batch-transcoding a library of clips into downscaled editing proxies. If I kick off two hardware transcodes at once, the encoder reliably falls over, so right now I run everything serially: each clip gets fully decoded, encoded, and muxed before the next one starts. It works, but it's slow, and the decode and encode hardware are mostly idle waiting on each other. A few things I can't pin down from the docs: Is there an actual ceiling on how many VTCompressionSession / VTDecompressionSession instances can be live at once, and does it depend on the device, the codec, or the resolution? Can I query that ceiling at runtime? I'd rather size my concurrency up front than find it by crashing. Decode and encode are separate hardware blocks, so can I safely run a decode session for one clip while the previous clip is still encoding, or does VideoToolbox serialize them anyway? When I do go over the limit, what should I be checking so I can back off cleanly? Right now I just get a crash instead of an error I can catch. Anything that gets me off the fully-serial pipeline would help. Thank you
Replies
5
Boosts
0
Views
215
Activity
2d