I need to split a large video file into multiple pieces quickly and without files with errors. The basic idea is, I have a 2GB video file which I want to change to multiple formats. I have read some encoders can not make use of more than 1 core when encoding to a different format. So I had the idea to split the large file (which is very quick) into 4/8 pieces depending on how many cores I have available on individual servers. re-encode each piece to a new format and use these to display video in sequence.
so
testfile.mp4
becomes
testfile_piece_1.mp4
testfile_piece_2.mp4
testfile_piece_3.mp4
testfile_piece_4.mp4
these can then be individually be converted.
The solution should not be format specific. However I have found issues with mp4 files.
I have tried the command below, which works really well and fast but creates files with errors.
ffmpeg -i testfile.mp4 -ss 00:00:00 -t 00:20:00 -c copy testfile_piece_1.mp4
When I play the testfile_piece_1.mp4 on VLC it works fine. An issue arises when converting the split file to a different height and width mp4 file. I would get an error similar to "moov atom not found"
I tried adding -movflags faststart with no luck
I then came across this library https://code.google.com/archive/p/moovrelocator/ which fixed the moov issue but I would then get an error with regards to aac "Error while opening encoder for output stream #0.0 - maybe incorrect parameters such as bit_rate, rate, width or height"
The other way of splitting the files is useless but involved re-encoding the file. not too bad for smaller file size but the 2GB file would probably take days to complete.
Is there a way to split the largr file quickly without producing files with errors? I have been working on it for days with no luck.
Console output for comment - FFmpeg splitting large files
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/usr/share/nginx/html/uploads/testfile01.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf56.36.100
Duration: 00:05:02.08, start: 302.120000, bitrate: 3254 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709) , 1920x1080 [SAR 1:1 DAR 16:9], 3252 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (de fault)
Metadata:
handler_name : VideoHandler
[libx264 # 0x165ffc0] width not divisible by 2 (853x480)
Output #0, mp4, to '/usr/share/nginx/html/uploads/testfile01_480.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf56.36.100
Stream #0:0(und): Video: h264, none, q=2-31, 128 kb/s, SAR 2560:2559 DAR 0:0 , 25 fps (default)
Metadata:
handler_name : VideoHandler
encoder : Lavc56.41.100 libx264
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))
Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
console output 2
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/usr/share/nginx/html/uploads/testfile.mp4':
Metadata:
major_brand : dash
minor_version : 0
compatible_brands: iso6avc1mp41
creation_time : 2016-01-24 04:26:37
Duration: 01:15:58.08, start: 0.000000, bitrate: 3163 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 3161 kb/s, 25 fps, 25 tbr, 90k tbn, 50 tbc (default)
Metadata:
creation_time : 2016-01-24 04:26:37
handler_name : VideoHandler
[segment # 0x1197060] Codec for stream 0 does not use global headers but container format requires global headers
[mp4 # 0x11512a0] Codec for stream 0 does not use global headers but container format requires global headers
Output #0, segment, to '/usr/share/nginx/html/uploads/testfile%02d.mp4':
Metadata:
major_brand : dash
minor_version : 0
compatible_brands: iso6avc1mp41
encoder : Lavf56.36.100
Stream #0:0(und): Video: h264 (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 3161 kb/s, 25 fps, 25 tbr, 12800 tbn, 25 tbc (default)
Metadata:
creation_time : 2016-01-24 04:26:37
handler_name : VideoHandler
Stream mapping:
Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
[mp4 # 0x11512a0] Codec for stream 0 does not use global headers but container format requires global headers
[mp4 # 0x11512a0] Codec for stream 0 does not use global headers but container format requires global headers
[mp4 # 0x11512a0] Codec for stream 0 does not use global headers but container format requires global headers
[mp4 # 0x11512a0] Codec for stream 0 does not use global headers but container format requires global headers
[mp4 # 0x11512a0] Codec for stream 0 does not use global headers but container format requires global headers
[mp4 # 0x11512a0] Codec for stream 0 does not use global headers but container format requires global headers
[mp4 # 0x11512a0] Codec for stream 0 does not use global headers but container format requires global headers
[mp4 # 0x11512a0] Codec for stream 0 does not use global headers but container format requires global headers
[mp4 # 0x11512a0] Codec for stream 0 does not use global headers but container format requires global headers
[mp4 # 0x11512a0] Codec for stream 0 does not use global headers but container format requires global headers
[mp4 # 0x11512a0] Codec for stream 0 does not use global headers but container format requires global headers
[mp4 # 0x11512a0] Codec for stream 0 does not use global headers but container format requires global headers
[mp4 # 0x11512a0] Codec for stream 0 does not use global headers but container format requires global headers
[mp4 # 0x11512a0] Codec for stream 0 does not use global headers but container format requires global headers
[mp4 # 0x11512a0] Codec for stream 0 does not use global headers but container format requires global headers
Use the segment muxer to break the input into segments:
ffmpeg -i testfile.mp4 -c copy -f segment -segment_time 1200 testfile_piece_%02d.mp4
This will split the source at keyframes, so segments may not be exactly 1200 seconds long. And the timestamps aren't reset, so some players will fail to play the 2nd and latter segments. If playability is needed, insert -reset_timestamps 1.
After the parallel encoding, you can stitch the generated segments by first creating a text file seg.txt like this
file 'encoded_testfile_piece_00.mp4'
file 'encoded_testfile_piece_01.mp4'
file 'encoded_testfile_piece_02.mp4'
file 'encoded_testfile_piece_03.mp4'
And then running
ffmpeg -f concat -i seg.txt -c copy -fflags +genpts encoded_full.mp4
Related
I want to convert .AVI files coming from a camera-trap to individuals frames, ideally using ffmpeg. Up to now I am not succeeding.
The most simple thing I tried, to try to locate the problem, is this (I want all frames and my test file is test.avi):
ffmpeg -i test.avi output_%04d.png
It fails with the following console message:
[avi # 0x559fb596f8c0] unknown stream type 73647578
[avi # 0x559fb596f8c0] Something went wrong during header parsing, tag [0][0]id has size 338702712, I will ignore it and try to continue anyway.
[mjpeg # 0x559fb59709e0] No JPEG data found in image
Last message repeated 100 times
[avi # 0x559fb596f8c0] decoding for stream 0 failed
[avi # 0x559fb596f8c0] Could not find codec parameters for stream 0 (Video: mjpeg (MJPG / 0x47504A4D), none(bt470bg/unknown/unknown), 1280x720): unspecified pixel format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[avi # 0x559fb596f8c0] Could not find codec parameters for stream 1 (Video: none (JUNK / 0x4B4E554A), none, 11025x22050): unknown codec
Consider increasing the value for the 'analyzeduration' and 'probesize' options
Input #0, avi, from 'test.avi':
Duration: 00:00:10.50, start: 0.000000, bitrate: 28129 kb/s
Stream #0:0: Video: mjpeg (MJPG / 0x47504A4D), none(bt470bg/unknown/unknown), 1280x720, 20.01 fps, 20.01 tbr, 20.01 tbn, 20.01 tbc
Stream #0:1: Video: none (JUNK / 0x4B4E554A), none, 11025x22050, 11025 tbr, 11025 tbn, 11025 tbc
Stream mapping:
Stream #0:1 -> #0:0 (? (?) -> png (native))
Decoder (codec none) not found for input stream #0:1
The actual video duration is 10s (when read using vlc for instance, and this is indeed the length of video programmed on the camera-trap. ffmpeg says the duration is 10.50s, and says there is a problem with header parsing (see above).
I have no idea how to sort this, despite having looked at more than many ffmpeg 'convert video to frames' post. Any hint would be appreciated, thanks.
If you look at the error message, you will see that the file has 2 video streams.
Stream #0:0: Video: mjpeg (MJPG / 0x47504A4D), none(bt470bg/unknown/unknown), 1280x720, 20.01 fps, 20.01 tbr, 20.01 tbn, 20.01 tbc
Stream #0:1: Video: none (JUNK / 0x4B4E554A), none, 11025x22050, 11025 tbr, 11025 tbn, 11025 tbc
and ffmpeg is trying to read for the second one:
Stream mapping:
Stream #0:1 -> #0:0 (? (?) -> png (native))
You can use -map 0:0 to select the first stream.
I am trying to use ffmpeg to concatenate video segments with some black screen. To do that I've first generated a blank 10-second video (no audio track) with:
$ ffmpeg -f lavfi -i color=black:s=320x240:r=1 -f lavfi -i anullsrc -t 10 -vcodec libvpx -an blank.mkv
I then created the simplest possible scenario within input.txt file (contents below) in order to have three seconds of black screen followed by some video (no audio track):
file 'blank.mkv'
duration 3
file 'video_example.mkv'
And, finally, ran the following ffmpeg command to concat the contents of that input file:
$ ffmpeg -f concat -i input.txt -codec:v copy -codec:a copy output.mkv
The issue that I have is that the duration 3 is not considered, so the final video still has ten seconds of black frames (instead of three) followed by my video. And also "Non-monotonous DTS in output stream 0:0 ..." message is shown when using duration x in the file. If I remove duration the warnings are gone and getting the 10-second black screen first output as well.
Full output of the ffmpeg concat command:
$ ffmpeg -hide_banner -f concat -i input.txt -codec:v copy -codec:a copy output.mkv
Input #0, concat, from 'input.txt':
Duration: N/A, start: 0.000000, bitrate: N/A
Stream #0:0: Video: vp8, yuv420p(progressive), 320x240, SAR 1:1 DAR 4:3, 1 fps, 1 tbr, 1k tbn, 1k tbc
Metadata:
ENCODER : Lavc57.107.100 libvpx
DURATION : 00:00:10.000000000
File 'output.mkv' already exists. Overwrite ? [y/N] y
Output #0, matroska, to 'output.mkv':
Metadata:
encoder : Lavf57.83.100
Stream #0:0: Video: vp8 (VP80 / 0x30385056), yuv420p(progressive), 320x240 [SAR 1:1 DAR 4:3], q=2-31, 1 fps, 1 tbr, 1k tbn, 1k tbc
Metadata:
ENCODER : Lavc57.107.100 libvpx
DURATION : 00:00:10.000000000
Stream mapping:
Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
[concat # 000000000031a440] DTS 3000 < 9000 out of order
[matroska # 0000000000328420] Non-monotonous DTS in output stream 0:0; previous: 9000, current: 3000; changing to 9000. This may result in incorrect timestamps in the output file.
[matroska # 0000000000328420] Non-monotonous DTS in output stream 0:0; previous: 9000, current: 4001; changing to 9000. This may result in incorrect timestamps in the output file.
[matroska # 0000000000328420] Non-monotonous DTS in output stream 0:0; previous: 9000, current: 4998; changing to 9000. This may result in incorrect timestamps in the output file.
[matroska # 0000000000328420] Non-monotonous DTS in output stream 0:0; previous: 9000, current: 6004; changing to 9000. This may result in incorrect timestamps in the output file.
[matroska # 0000000000328420] Non-monotonous DTS in output stream 0:0; previous: 9000, current: 7002; changing to 9000. This may result in incorrect timestamps in the output file.
[matroska # 0000000000328420] Non-monotonous DTS in output stream 0:0; previous: 9000, current: 8005; changing to 9000. This may result in incorrect timestamps in the output file.
frame= 5794 fps=0.0 q=-1.0 Lsize= 7109kB time=01:37:09.70 bitrate= 10.0kbits/s speed=5.16e+004x
video:7043kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.926229%
Any idea what am I doing wrong? The warning seems to hint towards the issue here.
Other possibly useful info:
$ ffprobe -hide_banner blank.mkv
Input #0, matroska,webm, from 'blank.mkv':
Metadata:
ENCODER : Lavf57.83.100
Duration: 00:00:10.00, start: 0.000000, bitrate: 1 kb/s
Stream #0:0: Video: vp8, yuv420p(progressive), 320x240, SAR 1:1 DAR 4:3, 1 fps, 1 tbr, 1k tbn, 1k tbc (default)
Metadata:
ENCODER : Lavc57.107.100 libvpx
DURATION : 00:00:10.000000000
$ ffprobe -hide_banner video_example.mkv
Input #0, matroska,webm, from 'video_example.mkv':
Metadata:
encoder : GStreamer matroskamux version 1.8.1.1
creation_time : 2018-05-04T17:57:04.000000Z
Duration: 01:37:08.70, start: 15434.269000, bitrate: 9 kb/s
Stream #0:0(eng): Video: vp8, yuv420p(progressive), 320x240, SAR 1:1 DAR 4:3, 1 fps, 1 tbr, 1k tbn, 1k tbc (default)
Metadata:
title : Video
$ ffmpeg -v
ffmpeg version 3.4.2 Copyright (c) 2000-2018 the FFmpeg developers
built with gcc 7.3.0 (GCC)
For video and audio files, inpoint/outpoint have to be used.
file 'blank.mkv'
outpoint 3
file 'video_example.mkv'
duration is useful for single images, like when making a slideshow, or raw audio/video streams where ffmpeg can't reliably discover the media duration.
I get the error "No NVENC capable devices found" when trying a simple encoding like this, even skipping audio to make sure it's not an audio problem:
ffmpeg.exe -i input.mp4 -c:v hevc_nvenc -an out.mp4
I also tried with more details, like setting the pixel format, the preset, the rate-control, the format.
On the documentation page there:
https://trac.ffmpeg.org/wiki/HWAccelIntro
it says that if we get this error we should check for the pixel format. The video has yuv420p here and even specifying the format results in the same error.
i also checked the NVidia supported cards and it says GeForce, but no details about the models:
https://developer.nvidia.com/video-encode-decode-gpu-support-matrix#Encoder
I tried h264_nvenc and it works perfectly however, the problem is only with hevc_nvenc
Has anyone encountered this problem?
Complete console output:
[h264 # 0000000002534560] Reinit context to 1280x544, pix_fmt: yuv420p
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'input.mp4':
Metadata:
major_brand : isom
minor_version : 1
compatible_brands: isom
creation_time : 2014-05-23T13:04:15.000000Z
Duration: 01:54:03.95, start: 0.000000, bitrate: 3193 kb/s
Stream #0:0(und): Video: h264 (High), 1 reference frame (avc1 / 0x31637661), yuv420p(left), 1280x544 [SAR 1:1 DAR 40:17], 2750 kb/s, 23.98 fps, 23.98 tbr, 24k tbn, 47.95 tbc (default)
Metadata:
creation_time : 2014-05-23T11:25:27.000000Z
Stream #0:1(spa): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, 5.1, fltp, 439 kb/s (default)
Metadata:
creation_time : 2014-05-23T12:56:43.000000Z
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> hevc (hevc_nvenc))
Press [q] to stop, [?] for help
[h264 # 000000000260ea40] Reinit context to 1280x544, pix_fmt: yuv420p
[graph 0 input from stream 0:0 # 0000000003857ec0] w:1280 h:544 pixfmt:yuv420p tb:1/24000 fr:24000/1001 sar:1/1 sws_param:flags=2
[hevc_nvenc # 00000000038574e0] Loaded Nvenc version 8.0
[hevc_nvenc # 00000000038574e0] Nvenc initialized successfully
[hevc_nvenc # 00000000038574e0] 1 CUDA capable devices found
[hevc_nvenc # 00000000038574e0] [ GPU #0 - < GeForce GTX 950M > has Compute SM 5.0 ]
[hevc_nvenc # 00000000038574e0] Codec not supported
[hevc_nvenc # 00000000038574e0] No NVENC capable devices found
[hevc_nvenc # 00000000038574e0] Nvenc unloaded
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
Conversion failed!
950M doesn't support h265 codec indeed.
From nvidia nvenc page or the detailed support matrix, we can learn that h265/hevc is supported only from 2nd generation maxwell GPU.
Also check the maxwell wiki page and list of NVIDIA GPU. You can see that 950M's code name is GM107 which is a 1st generation maxwell architecture, that means no h265 support. Sorry, you have to fall back to h264.
I'm getting the error bitrate tolerance too small for bitrate so far no problem. I know that there are several switches to increase that but nothing works.
ffmpeg -y -r 1/30 -b:v 999999k -bt 999999k -maxrate 999999k -i in.flv out%03d.jpg
The source of that commandline is directly from ffmpeg. But that crashes:
ffmpeg version N-44123-g5d55830 Copyright (c) 2000-2012 the FFmpeg developers
built on Sep 2 2012 20:23:29 with gcc 4.7.1 (GCC)
[...]
Input #0, flv, from 'in.flv':
Duration: 00:05:00.13, start: 0.000000, bitrate: 259 kb/s
Stream #0:0: Video: flv1, yuv420p, 320x240, 1k tbr, 1k tbn, 1k tbc
Stream #0:1: Audio: nellymoser, 22050 Hz, mono, s16
[mjpeg # 04356860] bitrate tolerance too small for bitrate
[mjpeg # 04317540] ff_frame_thread_encoder_init failed
Output #0, image2, to 'out%03d.jpg':
Stream #0:0: Video: mjpeg, yuvj420p, 320x240, q=2-31, 200 kb/s, 90k tbn, 0.03 tbc
Stream mapping:
Stream #0:0 -> #0:0 (flv -> mjpeg)
Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
Some ideas what I'm doing wrong?
I'm converting .mov files to .mpg files using avconv. The command being run by my php application is as follows:
avconv -y -i '$finalvideo' -target ntsc-dvd -aspect 4:3 '$mpgvideo' > $logs
I'm able to convert small .mov files to .mpg without any problems. However, I'm unable to convert videos that are over ten or fifteen minutes long. The log file is completely empty. When I run the command directly the frame stops somewhere around 34000 no matter which video I pick.
The cpu shows 97% usage on this process, however, nothing is happening.
OS Ubuntu 10~
How can I gather more information about this stalled process?
Here's the frozen output
avconv version 0.7, Copyright (c) 2000-2011 the Libav developers
built on Nov 3 2011 13:39:09 with gcc 4.3.3
Seems stream 0 codec frame rate differs from container frame rate: 180000.00 (180000/1) -> 23.98 (24000/1001)
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/var/www/sites/default/files/compiled_videos/573-stream.mov':
Metadata:
major_brand : qt
minor_version : 512
compatible_brands: qt
creation_time : 1970-01-01 00:00:00
encoder : Lavf53.0.3
Duration: 00:18:53.49, start: 0.000000, bitrate: 1430 kb/s
Stream #0.0(eng): Video: h264 (Main), yuv420p, 854x480, 1387 kb/s, 25.60 fps, 23.98 tbr, 90k tbn, 180k tbc
Metadata:
creation_time : 1970-01-01 00:00:00
Stream #0.1(eng): Audio: aac, 48000 Hz, stereo, s16, 126 kb/s
Metadata:
creation_time : 1970-01-01 00:00:00
[buffer # 0x9d88820] w:854 h:480 pixfmt:yuv420p
[scale # 0x9d88b60] w:854 h:480 fmt:yuv420p -> w:720 h:480 fmt:yuv420p flags:0x4
Incompatible sample format 's16' for codec 'ac3', auto-selecting format 'flt'
Output #0, dvd, to '/var/www/sites/default/files/compiled_videos/573.mpg':
Metadata:
major_brand : qt
minor_version : 512
compatible_brands: qt
creation_time : 1970-01-01 00:00:00
encoder : Lavf53.10.0
Stream #0.0(eng): Video: mpeg2video, yuv420p, 720x480 [PAR 8:9 DAR 4:3], q=2-31, 6000 kb/s, 90k tbn, 29.97 tbc
Metadata:
creation_time : 1970-01-01 00:00:00
Stream #0.1(eng): Audio: ac3, 48000 Hz, stereo, flt, 448 kb/s
Metadata:
creation_time : 1970-01-01 00:00:00
Stream mapping:
Stream #0.0 -> #0.0 (h264 -> mpeg2video)
Stream #0.1 -> #0.1 (aac -> ac3)
Press ctrl-c to stop encoding
[mpeg2video # 0x9d8bf20] rc buffer underflow
Input stream #0.1 frame changed from rate:48000 fmt:s16 ch:2 to rate:48000 fmt:flt ch:2
frame=33910 fps= 91 q=2.0 size= 151922kB time=336.58 bitrate=3697.7kbits/s dup=5475 drop=530
Oh interesting. I used -loglevel debug -debug. I was seeing this information
stream #0:
keyframe=0
duration=0.000
dts=1133.449 pts=1133.533
size=103
*** 1 dup!
stream #0:
keyframe=0
duration=0.000
dts=1133.449 pts=1133.449
size=104
Until I finally received this message
*** drop!
I have posted my bug with libav
http://bugzilla.libav.org/show_bug.cgi?id=67
thanks for your help.
Try running avconv with a higher log level: -loglevel debug.
That should give you more data. Also, try -dump and -debug.
Are you sure it's not because you're hitting a storage cap?
I ran into this recently with ffmpeg while converting some oddly sized video clips to a "standard" 16:9 x264 & AAC streams inside a mp4 container. When that "Incompatible sample format 's16' for codec 'ac3', auto-selecting format 'flt'" message would show up, my ffmpeg process would frequently just hang indefinitely at 100% CPU use.
I finally found that the issue, at least in my case, was that because of the conversions I was doing (transcoding to an intermediary format then to the final, desired output), it was actually the number of audio stream channels that caused the encode process to hang. By telling ffmpeg to convert the audio stream to 2-channel, I was able to fix this issue permanently.
The same solution will probably apply to avconv. In the ffmpeg world (and because I'm stuck with an outdated version), I merely needed to add -ac 2 to my ffmpeg command and everything worked perfectly!
Hope that helps :)