I'm creating a video (QuickTime .mov format, H.264 encoded) from a bunch of still images, and I want to add a chapter track in the process. The video is being created fine, and I am not detecting any errors, but QuickTime Player does not show any chapters. I am aware of this question but it does not solve my problem.
The old QuickTime Player 7, unlike recent versions, can show information about the tracks of a movie. When I open a movie with working chapters (created using old QuickTime code), I see a video track and a text track, and the video track knows that the text track is providing chapters for the video. Whereas, if I examine a movie created by my new code, there is a metadata track along with the video track, but QuickTime does not know that the metadata track is supposed to be providing chapters. Things I've read have led me to believe that one is supposed to use metadata for chapters, but has anyone actually gotten that to work? Would a text track work?
Here's how I am creating the AVAssetWriterInput for the metadata.
// Make dummy AVMetadataItem to get its format
AVMutableMetadataItem* dummyMetaItem = [AVMutableMetadataItem metadataItem];
dummyMetaItem.identifier = AVMetadataIdentifierQuickTimeUserDataChapter;
dummyMetaItem.dataType = (NSString*) kCMMetadataBaseDataType_UTF8;
dummyMetaItem.value = #"foo";
AVTimedMetadataGroup* dummyGroup = [[[AVTimedMetadataGroup alloc]
initWithItems: #[dummyMetaItem]
timeRange: CMTimeRangeMake( kCMTimeZero, kCMTimeInvalid )] autorelease];
CMMetadataFormatDescriptionRef metaFmt = [dummyGroup copyFormatDescription];
// Make the input
AVAssetWriterInput* metaWriterInput = [AVAssetWriterInput
assetWriterInputWithMediaType: AVMediaTypeMetadata
outputSettings: nil
sourceFormatHint: metaFmt];
CFRelease( metaFmt );
// Associate metadata input with video input
[videoInput addTrackAssociationWithTrackOfInput: metaWriterInput
type: AVTrackAssociationTypeChapterList];
// Associate metadata input with AVAssetWriter
[writer addInput: metaWriterInput];
// Create a metadata adaptor
AVAssetWriterInputMetadataAdaptor* metaAdaptor = [AVAssetWriterInputMetadataAdaptor
assetWriterInputMetadataAdaptorWithAssetWriterInput: metaWriterInput];
P.S. I tried using a text track instead (an AVAssetWriterInput of type AVMediaTypeText) and QuickTime Player says the result is "not a movie". Not sure what I'm doing wrong.
I managed to use a text track to provide chapters. I spent an Apple developer tech support incident and was told that this is the right way to do it.
Setup:
I assume that the AVAssetWriter has been created, and an AVAssetWriterInput for the video track has been assigned to it.
The trickiest part here is creating the text format description. The docs say that CMTextFormatDescriptionCreateFromBigEndianTextDescriptionData takes as input a TextDescription structure, but neglects to say where that structure is defined. It is in Movies.h, which is in QuickTime.framework, which is no longer part of the Mac OS SDK. Thanks, Apple.
// Create AVAssetWriterInput
AVAssetWriterInput* textWriterInput = [AVAssetWriterInput
assetWriterInputWithMediaType: AVMediaTypeText
outputSettings: nil ];
textWriterInput.marksOutputTrackAsEnabled = NO;
// Connect input to writer
[writer addInput: textWriterInput];
// Mark the text track as providing chapter for the video
[videoWriterInput addTrackAssociationWithTrackOfInput: textWriterInput
type: AVTrackAssociationTypeChapterList];
// Create the text format description, which we will need
// when creating each sample.
CMFormatDescriptionRef textFmt = NULL;
TextDescription textDesc;
memset( &textDesc, 0, sizeof(textDesc) );
textDesc.descSize = OSSwapHostToBigInt32( sizeof(textDesc) );
textDesc.dataFormat = OSSwapHostToBigInt32( 'text' );
CMTextFormatDescriptionCreateFromBigEndianTextDescriptionData( NULL,
(const uint8_t*)&textDesc, sizeof(textDesc), NULL, kCMMediaType_Text,
&textFmt );
Writing a Sample:
CMSampleTimingInfo timing =
{
CMTimeMakeWithSeconds( endTime - startTime, timeScale ), // duration
CMTimeMakeWithSeconds( startTime, timeScale ),
kCMTimeInvalid
};
CMSampleBufferRef textSample = NULL;
CMPSampleBufferCreateWithText( NULL, (CFStringRef)theTitle, true, NULL, NULL,
textFmt, &timing, &textSample );
[textWriterInput appendSampleBuffer: textSample];
The function CMPSampleBufferCreateWithText is taken from the open source CoreMediaPlus.
Related
Q1) How can I get video file details with macOS APIs?
Q2) How do I assess video quality of an mp4 file?
I need a program to separate a large archive of mp4 files based on the video quality - i.e., clarity, sharpness - roughly, where they'd appear along the TV spectrum of analog -> 720 -> 1080 -> 2/4k. In this case, audio, color levels, file size, CPU/GPU load, etc., are not considerations per se.
Q1) It is easy to find "natural" dimensions with AVPlayer. A bit more poking around (https://developer.apple.com/documentation/avfoundation/avpartialasyncproperty/3816116-formatdescriptions ), my files have "avc1" as the media subtype; I gather that means h264. Can't locate ways to get more details with Apple APIs, like bit rate, that even Quicktime Player provides.
Lots of info is available with ffprobe, so I added it to my program. You too can embed a CLI program that runs inside a macOS application in background - see code at bottom.
Q2) To a video noob, dimensions are the obvious first approximation for video quality ... and codec, but mine have previously been converted to h264. Then I consider bit rates from ffprobe.
For testing, I located two h264 files with same dimensions (1280, 720), bit depth (8), and similar file size, frame rate, duration, amount of motion, color content. To my eye, one of the two looks better, distinctly sharper; that file is smaller and has a lower video bit rate (20-40%), even when normalized for its slightly lower frame rate and duration.
From an info theory perspective, doesn't seem possible. I've learned codecs can provide "quality" optimizations during compression - way past my understanding - but I can't find, looking at the video stream data, indicators of any that would impact quality/sharpness. Nothing in per-frame and per-packet data from ffprobe stands out.
Are there any tell-tale signs I should look for? Is this a fool's errand?
Here's my swift hack to run ffprobe inside a macOS application (written with XC 13 on 11.6). If you know how to run a Process() that lives in /usr/bin/..., please post - I don't get the entitlements thing. (Aliases/links to home directory don't work.)
// takes a local fileURL and determines video properties using ffprobe
func runFFProbe(targetURL:URL){
func buildArguments(url:URL) -> [String] {
// for ffprobe introduction,see: https://ottverse.com/ffprobe-comprehensive-tutorial-with-examples/
// and for complete info: https://ffmpeg.org/ffprobe.html
var arguments:[String] = []
// note: don't interpolate URL paths - may have spaces in them
let argString = "-v error -hide_banner -of default=noprint_wrappers=0 -print_format flat -select_streams v:0 -show_entries stream=width,height,bit_rate,codec_name,codec_long_name,profile,codec_tag_string,time_base,avg_frame_rate,r_frame_rate,duration_ts,bits_per_raw_sample,nb_frames "
let _ = argString.split(separator: " ").map{arguments.append(String($0))}
// let _ suppresses compiler warning about unused result of map call
arguments.append(url.path) // spaces in URL path seem to be okay here
return arguments
}
let task = Process()
// task.executableURL = URL(fileURLWithPath: "/usr/local/bin/ffprobe")
// reports "doesn't exist", but really access is blocked by macOS :(
// statically-linked ffprobe is added to the app bundle
// downloadable here - https://evermeet.cx/ffmpeg/#sExtLib-ffprobe
task.executableURL = Bundle.main.url(forResource: "ffprobe", withExtension: nil)
task.arguments = buildArguments(url: targetURL)
let pipe = Pipe()
task.standardOutput = pipe // ffprobe writes console thru standardOutput
// (ffmpeg uses standardError)
let fh = pipe.fileHandleForReading
var cumulativeResults = "" // adds the result from each buffer dump
fh.waitForDataInBackgroundAndNotify() // setup handle for listening
// object must be specified when running multiple simultaneous calls
// otherwise every instance receives messages from all other filehandles too
NotificationCenter.default.addObserver(forName: .NSFileHandleDataAvailable, object: fh, queue: nil) {notif in
let closureFileHandle:FileHandle = notif.object as! FileHandle
// Get the data from the FileHandle
let data:Data = closureFileHandle.availableData
// print("received bytes: \(data.count)\n") // debugging
if data.count > 0 {
// re-arm fh for any addition data
fh.waitForDataInBackgroundAndNotify()
// append new data to the accumulator
let str = String(decoding: data, as: UTF8.self)
cumulativeResults += str
// optionally insert code here for intermediate reporting/parsing
// self.printToTextView(string: str)
}
}
task.terminationHandler = {task -> Void in
DispatchQueue.main.async(execute: {
// run the whole termination on the main queue
if task.terminationReason==Process.TerminationReason.exit {
// roll your own reporting method
self.printToTextView(string: targetURL.lastPathComponent)
self.printToTextView(string: targetURL.fileSizeString) //custom URL extension
self.printToTextView(string: cumulativeResults)
let str = "\nSuccess!\n"
self.printToTextView(string: str)
} else {
print("Task did not terminate properly")
// post an error in UI too
return
}
// successful conversion if this point is reached
}) // end dispatchqueue
} // end termination handler
do { try
task.run()
} catch let error as NSError {
print(error.localizedDescription)
// post in UI too
return
}
} // end runFFProbe()
I'm just getting started with TVOS and was wondering if anyone has addressed looping in TVOS / TVJS / TVML. I'm using this tutorial to create a simple interface for playing videos. My wrinkle is that I want these videos to play in a continuous seamless loop - basically a moving screensaver type effect - see example videos at art.chrisbaily.com.
Is there a simple way to do this, or do I need to build some kind of event listener to do the looping manually? I'd like the videos to be fairly hi res, and the videos would be somewhere between 1 and 3 minutes.
I was looking for an answer for this question as well. And found that the only way this would be possible is to create an event listener and adding a duplicate media item to the playlist. But honestly it is not that hard, given if you followed the tutorial that you listed in your post.
So the code would be something like
player.playlist = playlist;
player.playlist.push(mediaItem);
//Again push the same type of media item in playlist so now you have two of the same.
player.playlist.push(mediaItem);
player.present();
This will make sure that once your first video ends the second one starts playing which is essentially a loop. Then for the third one and on you implement an event listener using "mediaItemWillChange" property. This will make sure that once a video ends a new copy of the same video is added to the playlist.
player.addEventListener("mediaItemWillChange", function(e) {
player.playlist.push(mediaItem);
});
Just put the event listener before you start the player using
player.present();
Note this question and idea of this sort was already asked/provided on Apple's discussion board. I merely took the idea and had implemented it in my own project and now knowing that it works I am posting the solution. I found that if I pop the first video out of the playlist and add then push a new on the playlist as mentioned in the linked thread below, my videos were not looping. Below is the link to the original post.
How to repeat video with TVJS Player?
You could also set repeatMode on the playlist object
player.playlist.repeatMode = 1;
0 = no repeat
1 = repeat all items in playlist
2 = repeat current item
There's really no need to push a second media item onto the. Simply listen for the media to reach its end, then set the seek time back to zero. Here's the code.
Play the media.
private func playVideo(name: String) {
guard name != "" else {
return
}
let bundle = NSBundle(forClass:object_getClass(self))
let path = bundle.pathForResource(name, ofType:"mp4")
let url = NSURL.fileURLWithPath(path!)
self.avPlayer = AVPlayer(URL: url)
if (self.avPlayerLayer == nil) {
self.avPlayerLayer = AVPlayerLayer(player: self.avPlayer)
self.avPlayerLayer.frame = previewViews[1].frame
previewViews[1].layer.addSublayer(self.avPlayerLayer)
avPlayer.actionAtItemEnd = .None
//Listen for AVPlayerItemDidPlayToEndTimeNotification
NSNotificationCenter.defaultCenter().addObserver(self, selector: "playerItemDidReachEnd:", name: AVPlayerItemDidPlayToEndTimeNotification, object: self.avPlayer.currentItem)
}
avPlayer.play()
}
Play Again
func playerItemDidReachEnd(notification: NSNotification) {
let item = notification.object as? AVPlayerItem
item?.seekToTime(kCMTimeZero)
}
I have built some code to process video files on OSX, frame by frame. The following is an extract from the code which builds OK, opens the file, locates the video track (only track) and starts reading CMSampleBuffers without problem. However each CMSampleBufferRef I obtain returns NULL when I try to extract the pixel buffer frame. There's no indication in iOS documentation as to why I could expect a NULL return value or how I could expect to fix the issue. It happens with all the videos on which I've tested it, regardless of capture source or CODEC.
Any help greatly appreciated.
NSString *assetInPath = #"/Users/Dave/Movies/movie.mp4";
NSURL *assetInUrl = [NSURL fileURLWithPath:assetInPath];
AVAsset *assetIn = [AVAsset assetWithURL:assetInUrl];
NSError *error;
AVAssetReader *assetReader = [AVAssetReader assetReaderWithAsset:assetIn error:&error];
AVAssetTrack *track = [assetIn.tracks objectAtIndex:0];
AVAssetReaderOutput *assetReaderOutput = [[AVAssetReaderTrackOutput alloc]
initWithTrack:track
outputSettings:nil];
[assetReader addOutput:assetReaderOutput];
// Start reading
[assetReader startReading];
CMSampleBufferRef sampleBuffer;
do {
sampleBuffer = [assetReaderOutput copyNextSampleBuffer];
/**
** At this point, sampleBuffer is non-null, has all appropriate attributes to indicate that
** it's a video frame, 320x240 or whatever and looks perfectly fine. But the next
** line always returns NULL without logging any obvious error message
**/
CVImageBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
if( pixelBuffer != NULL ) {
size_t width = CVPixelBufferGetWidth(pixelBuffer);
size_t height = CVPixelBufferGetHeight(pixelBuffer);
CVPixelBufferLockBaseAddress(pixelBuffer, 0);
...
other processing removed here for clarity
}
} while( ... );
To be clear, I've stripped all error checking code but no problems were being indicated in that code. i.e. The AVAssetReader is reading, CMSampleBufferRef looks fine etc.
You haven't specified any outputSettings when creating your AVAssetReaderTrackOutput. I've run into your issue when specifying "nil" in order to receive the video track's original pixel format when calling copyNextSampleBuffer. In my app I wanted to ensure no conversion was happening when calling copyNextSampleBuffer for the sake of performance, if this isn't a big concern for you, specify a pixel format in the output settings.
The following are Apple's recommend pixel formats based on the hardware capabilities:
kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange
kCVPixelFormatType_420YpCbCr8BiPlanarFullRange
Because you haven't supplied any outputSettings you're forced to use the raw data contained within in the frame.
You have to get the block buffer from the sample buffer using CMSampleBufferGetDataBuffer(sampleBuffer), after you have that you need to get the actual location of the block buffer using
size_t blockBufferLength;
char *blockBufferPointer;
CMBlockBufferGetDataPointer(blockBuffer, 0, NULL, &blockBufferLength, &blockBufferPointer);
Look at *blockBufferPointer and decode the bytes using the frame header information for your required codec.
FWIW: Here is what official docs say for the return value of CMSampleBufferGetImageBuffer:
"Result is a CVImageBuffer of media data. The result will be NULL if the CMSampleBuffer does not contain a CVImageBuffer, or if the CMSampleBuffer contains a CMBlockBuffer, or if there is some other error."
Also note that the caller does not own the returned dataBuffer from CMSampleBufferGetImageBuffer, and must retain it explicitly if the caller needs to maintain a reference to it.
Hopefully this info helps.
I want to play Beep sound in my Mac Os X and specify duration and frequency. On Windows it can be done by using Beep function (Console.Beep in .Net). Is there anything equivalent in Mac? I am aware of NSBeep but it does not take any parameters.
On the Mac, the system alert sound is a sampled (prerecorded) sound that the user chooses. It often sounds nothing like a beep—it may be a honk, thunk, blare, or other sound that can't be as a simple constant waveform of fixed shape, frequency, and amplitude. It can even be a recording of the user's voice, or a clip from a TV show or movie or game or song.
It also does not need to be only a sound. One of the accessibility options is to flash the screen when an alert sound plays; this happens automatically when you play the alert sound (or a custom alert sound), but not when you play a sound through regular sound-playing APIs such as NSSound.
As such, there's no simple way to play a custom beep of a specified and constant shape, frequency, and amplitude. Any such beep would differ from the user's selected alert sound and may not be perceptible to the user at all.
To play the alert sound on the Mac, use NSBeep or the slightly more complicated AudioServicesPlayAlertSound. The latter allows you to use custom sounds, but even these must be prerecorded, or at least generated by your app in advance using more Core Audio code than is worth writing.
I recommend using NSBeep. It's one line of code to respect the user's choices.
PortAudio has cross platform C code for doing this here: https://subversion.assembla.com/svn/portaudio/portaudio/trunk/examples/paex_sine.c
That particular sample generates tones on the left and right speaker, but doesn't show how the frequencies are calculated. For that, you can use the formula in this code: Is there an library in Java for emitting a certain frequency constantly?
I needed a similar functionality for an app. I ended up writing a small, reusable class to handle this for me.
Source on GitHub
A reusable class for generating simple sine waveform audio tones with specified frequency and amplitude. Can play continuously or for a specified duration.
The interface is fairly straightforward and is shown below:
#interface TGSineWaveToneGenerator : NSObject
{
AudioComponentInstance toneUnit;
#public
double frequency;
double amplitude;
double sampleRate;
double theta;
}
- (id)initWithFrequency:(double)hertz amplitude:(double)volume;
- (void)playForDuration:(float)time;
- (void)play;
- (void)stop;
#end
Here's a way of doing this with the newer AVAudioEngine/AVAudioNode APIs, and Swift:
import AVFoundation
import Accelerate
// Specify the audio format we're going to use
let sampleRateHz = 44100
let numChannels = 1
let pcmFormat = AVAudioFormat(standardFormatWithSampleRate: Double(sampleRateHz), channels: UInt32(numChannels))
let noteFrequencyHz = 440
let noteDuration: NSTimeInterval = 1
// Create a buffer for the audio data
let numSamples = UInt32(noteDuration * Double(sampleRateHz))
let buffer = AVAudioPCMBuffer(PCMFormat: pcmFormat, frameCapacity: numSamples)
buffer.frameLength = numSamples // the buffer will be completely full
// The "standard format" is deinterleaved float, so we can assume the stride is 1.
assert(buffer.stride == 1)
for channelBuffer in UnsafeBufferPointer(start: buffer.floatChannelData, count: numChannels) {
// Generate a sine wave with the specified frequency and duration
var length = Int32(numSamples)
var dc: Float = 0
var multiplier: Float = 2*Float(M_PI)*Float(noteFrequencyHz)/Float(sampleRateHz)
vDSP_vramp(&dc, &multiplier, channelBuffer, buffer.stride, UInt(numSamples))
vvsinf(channelBuffer, channelBuffer, &length)
}
// Hook up a player and play the buffer, then exit
let engine = AVAudioEngine()
let player = AVAudioPlayerNode()
engine.attachNode(player)
engine.connect(player, to: engine.mainMixerNode, format: pcmFormat)
try! engine.start()
player.scheduleBuffer(buffer, completionHandler: { exit(1) })
player.play()
NSRunLoop.mainRunLoop().run() // Keep running in a playground
I want to play Beep sound in my Mac Os X and specify duration and frequency. On Windows it can be done by using Beep function (Console.Beep in .Net). Is there anything equivalent in Mac? I am aware of NSBeep but it does not take any parameters.
On the Mac, the system alert sound is a sampled (prerecorded) sound that the user chooses. It often sounds nothing like a beep—it may be a honk, thunk, blare, or other sound that can't be as a simple constant waveform of fixed shape, frequency, and amplitude. It can even be a recording of the user's voice, or a clip from a TV show or movie or game or song.
It also does not need to be only a sound. One of the accessibility options is to flash the screen when an alert sound plays; this happens automatically when you play the alert sound (or a custom alert sound), but not when you play a sound through regular sound-playing APIs such as NSSound.
As such, there's no simple way to play a custom beep of a specified and constant shape, frequency, and amplitude. Any such beep would differ from the user's selected alert sound and may not be perceptible to the user at all.
To play the alert sound on the Mac, use NSBeep or the slightly more complicated AudioServicesPlayAlertSound. The latter allows you to use custom sounds, but even these must be prerecorded, or at least generated by your app in advance using more Core Audio code than is worth writing.
I recommend using NSBeep. It's one line of code to respect the user's choices.
PortAudio has cross platform C code for doing this here: https://subversion.assembla.com/svn/portaudio/portaudio/trunk/examples/paex_sine.c
That particular sample generates tones on the left and right speaker, but doesn't show how the frequencies are calculated. For that, you can use the formula in this code: Is there an library in Java for emitting a certain frequency constantly?
I needed a similar functionality for an app. I ended up writing a small, reusable class to handle this for me.
Source on GitHub
A reusable class for generating simple sine waveform audio tones with specified frequency and amplitude. Can play continuously or for a specified duration.
The interface is fairly straightforward and is shown below:
#interface TGSineWaveToneGenerator : NSObject
{
AudioComponentInstance toneUnit;
#public
double frequency;
double amplitude;
double sampleRate;
double theta;
}
- (id)initWithFrequency:(double)hertz amplitude:(double)volume;
- (void)playForDuration:(float)time;
- (void)play;
- (void)stop;
#end
Here's a way of doing this with the newer AVAudioEngine/AVAudioNode APIs, and Swift:
import AVFoundation
import Accelerate
// Specify the audio format we're going to use
let sampleRateHz = 44100
let numChannels = 1
let pcmFormat = AVAudioFormat(standardFormatWithSampleRate: Double(sampleRateHz), channels: UInt32(numChannels))
let noteFrequencyHz = 440
let noteDuration: NSTimeInterval = 1
// Create a buffer for the audio data
let numSamples = UInt32(noteDuration * Double(sampleRateHz))
let buffer = AVAudioPCMBuffer(PCMFormat: pcmFormat, frameCapacity: numSamples)
buffer.frameLength = numSamples // the buffer will be completely full
// The "standard format" is deinterleaved float, so we can assume the stride is 1.
assert(buffer.stride == 1)
for channelBuffer in UnsafeBufferPointer(start: buffer.floatChannelData, count: numChannels) {
// Generate a sine wave with the specified frequency and duration
var length = Int32(numSamples)
var dc: Float = 0
var multiplier: Float = 2*Float(M_PI)*Float(noteFrequencyHz)/Float(sampleRateHz)
vDSP_vramp(&dc, &multiplier, channelBuffer, buffer.stride, UInt(numSamples))
vvsinf(channelBuffer, channelBuffer, &length)
}
// Hook up a player and play the buffer, then exit
let engine = AVAudioEngine()
let player = AVAudioPlayerNode()
engine.attachNode(player)
engine.connect(player, to: engine.mainMixerNode, format: pcmFormat)
try! engine.start()
player.scheduleBuffer(buffer, completionHandler: { exit(1) })
player.play()
NSRunLoop.mainRunLoop().run() // Keep running in a playground