Parse csv into variables and execute ffmpeg command in loop - bash

I am trying perform a batch operation to extract specific frames from multiple video files and save as a PNG, using a bash script. I hope to do this by using ffmpeg within a bash script, supplemented by a csv file that contains the name of the input video, the specific frame number to be extracted from the input video, and the output name of the PNG file.
The video files and the csv file would all be placed within the same folder. The script can also be placed in there if necessary.
My csv - called "select.csv" - currently takes the following format (input,output,frame):
mad0.m4v,mad0_out1,9950
mad0.m4v,mad0_out2,4500
mad1.m4v,mad1_out1,3200
My current script - called "frame.sh" - takes the following form:
#!/bin/bash
OLDIFS=$IFS
IFS=“,”
SDIR=/Users/myuser/Desktop/f-input/
cd $SDIR;
while read input output frame
do
echo "$input"
echo "$output"
echo "$frame"
input1=$input
output1=$output
frame1=$frame
ffmpeg -i "$input1" -vf select='eq(n\,'"$frame1"')' -vsync 0
"$output1".png
done < $1
IFS=$OLDIFS
This should allow me to run ./frame.sh select.csv to then process all relevant files in the "f-input" folder on my desktop and extract the specified frames.
I ended up echoing the variables read from the csv so that they could actually be used as variables and looped in the ffmpeg command because carrying out the ffmpeg command using $input, $frame and $output directly after the read operation only ever completed the process on the first line of the csv, without progressing further.
Essentially I would like the following to actually loop through each csv entry, instead of only the first line:
#!/bin/bash
OLDIFS=$IFS
IFS=“,”
SDIR=/Users/myuser/Desktop/f-input/
cd $SDIR;
while read input output frame
do
ffmpeg -i "$input" -vf select='eq(n\,'"$frame"')' -vsync 0 "$output".png
done < $1
IFS=$OLDIFS
Any and all advice appreciated!
Many thanks

Replace IFS=“,” with IFS=",".

I wrote a similar script that reads csv and process movies by ffmpeg.
It works well on the first line of the csv but fails after the second lines.
I found ffmpeg in a loop seems to affect the "read" command and trim the first character of the lines after the second line.
So I ended up with adding extra "garbage" column on the left-most side of the csv and let ffmpeg trim it.
my csv is like:
101,movie1.mp4
102,movie2.mp4
103,movie3.mp4
...
and the (simplified) script is like:
while IFS="," read id movie; do
ffmpeg -v quiet -s 1280x720 -i "$movie" "$id-$movie" </dev/null
done
it generates "101-movie1.mp4" for the first line of the csv just like I expect
but after the second line it generates "02-movie1.mp4" "03-movie3.mp4" and so force because ffmpeg (seems to have) trimmed the first character of the lines.
I added a garbage column on the 1st column like this
x,101,movie1.mp4
x,102,movie2.mp4
x,103,movie3.mp4
and fix the script:
while IFS="," read garbage id movie; do
ffmpeg -v quiet -s 1280x720 -i "$movie" "$id-$movie" </dev/null
done
this worked for me.

Related

How do I resolve ffmpeg concat command error "unknown keyword..."?

I am trying to concatenate two video files using ffmpeg, and I am receiving an error.
To eliminate compatibility issues between the two videos, I have been concatenating the same video with itself, and the same error persists.
ffmpeg \
-f concat \
-safe 0 \
-i intro_prepped.avi intro_prepped.avi \
-c copy \
concat.avi
And the error output I receive is....
[concat # 0x220d420] Line 1: unknown keyword 'RIFFf�?'
intro_prepped.avi: Invalid data found when processing input
I have tried various combinations of concat flags and have not been able to get it to work. Has anyone seen this error before?
This is a bit late for the original post, but I was just searching for answers to the same problem so I think it's still relevant and I haven't found any more recent posts answering the same problem.
I found that my .txt file was encoded wrong. I opened the file in Notepad and did a 'Save As...'
I changed the encoding to UTF-8 and the ffmpeg concat command worked.
Docs for several ways of concatenating files: https://trac.ffmpeg.org/wiki/Concatenate
Here's a command I use to concatenate videos:
ffmpeg \
-i "concat:input1.avi|input2.avi|input3.avi" \
-c:a copy \
-c:v copy \
output.avi
I tried all of the aforementioned and it didn't work.
It looks like that the file names in the list have to be specially formatted to look like:
file '/path/to/file1.wav'
with a word file included. I spend a lot of time trying to guess why ffmpeg encountered an error trying to read the file names. It didn't matter if they were in the list or in the command line. So only after I utilized a command
for f in *.wav; do echo "file '$f'" >> mylist.txt; done
to make list from ffmpeg's manual I had success. The only difference was an additional word file.
Here you can read it yourself: https://trac.ffmpeg.org/wiki/Concatenate#demuxer
Your text file is likely encoded in UTF-16.
Fix: (Windows 10)
Open text file
Select 'Save as'
Look by the save button, you get to pick encoding with a drop down box, select UTF-8.
Save and run ffmpeg again.
I used the Powershell code on ffmpegs webpage to make a text file with filenames, and Powershell seems to save text files as some variant of UTF-16, so I chose the safer UTF-8.
The input file should be a text file, not an avi. The text file lists the files to concatenate.
See the concat demuxer documentation and FFmpeg Wiki: Concat.
Nobody had a full, working, concat text file batch file anywhere. So I am posting it
md ts
for %%x in (input\*.m4a) do (ffmpeg -i "%%x" -c copy -bsf:v h264_mp4toannexb ts\%%~nx.ts)
for %%c in ("ts\*ts") do (echo file '%%c')>>list.txt
for %%f in (input\*.m4a) do (set fn=%%~nf)
ffmpeg -f concat -safe 0 -i "list.txt" -c copy "%cd%\%fn%".m4a
I struggled with all these instructions above on my Mac (M1, Ventura, ffmpeg version N-109530-g4a80db5fc2-tessus). None of them worked for me - but a combination of all did the trick!
This is how I got it running:
place all input files the same folder
create input.txt in this folder - content looks like this:
file 'input1.mp4'
file 'input2.mp4'
file 'input3.mp4'
file 'input4.mp4'
Note:
file encoding must be UTF-8
file keyword must be present
filename must not be fully qualified (I got exceptions using '/path/to/input1.mp4')
filename must be enclosed by '
navigate to this folder in the terminal
execute ffmpeg -f concat -i input.txt -c copy ffmpegOUT.mp4

downloading and concatenating parts of videos from youtube

I'm trying to create a video quiz, that will contain small parts of other videos, concatenated together (with the purpose, that people will identify from where these short snips are taken from).
For this purpose I created a file that contain the URL of the video, the starting time of the "snip", and its length. for example:
https://www.youtube.com/watch?v=5-j6LLkpQYY 00:00 01:00
https://www.youtube.com/watch?v=b-DqO_D1g1g 14:44 01:20
https://www.youtube.com/watch?v=DPAgWKseVhg 12:53 01:00
Meaning that the first part should take the video from the first URL from its beginning and last for a minute, the second part should be taken from the second URL starting from 14:44 (minutes:seconds) and last one minute and 20 seconds and so forth.
Then all these parts should be concatenated to a single video.
I'm trying to write a script (I use ubuntu and fluent in several scripting languages) that does that, and I tried to use youtube-dl command line package and ffmpeg, but I couldn't find the right options to achieve what I need.
Any suggestions will be appreciated.
Considering the list of videos is in foo.txt, and the output video to be foo.mp4, this bash script should do the job:
eval $(cat foo.txt | while read u s d; do echo "cat <(youtube-dl -q -o - $u | ffmpeg -v error -hide_banner -i - -ss 00:$s -t 00:$d -c copy -f mpegts -);"; done | tee /dev/tty) | ffmpeg -i - -c copy foo.mp4
This is using a little trick with process substitution and eval to avoid intermediate files, container mpegts to enable simple concat protocol, and tee /dev/tty just for debugging.
I have tested with youtube-dl 2018.09.26-1 and ffmpeg 1:4.0.2-3.

timelapse images into a movie, 500 at a time

I am trying to make a script to turn a bunch of timelapse images into a movie, using ffmpeg.
The latest problem is how to loop thru the images in, say, batches of 500.
There could be 100 images from the day, or there could be 5000 images.
The reason for breaking this apart is due to running out of memory.
Afterwards I would need to cat them using MP4Box to join all together...
I am entirely new to bash, but not entirely programming.
What I think needs to happen is this
1) read in the folders contents as the images may not be consecutively named
2) send ffmpeg a list of 500 at a time to process (https://trac.ffmpeg.org/wiki/Concatenate)
2b) while you're looping thru this, set a counter to determine how many loops you've done
3) use the number of loops to create the MP4Box cat command line to join them all at the end.
the basic script that works if there's only say 500 images is:
#!/bin/bash
dy=$(date '+%Y-%m-%d')
ffmpeg -framerate 24 -s hd1080 -pattern_type glob -i "/mnt/cams/Camera1/$dy/*.jpg" -vcodec libx264 -pix_fmt yuv420p Cam1-"$dy".mp4
MP4Box's cat command looks like:
MP4Box -cat Cam1-$dy7.mp4 -cat Cam1-$dy6.mp4 -cat Cam1-$dy5.mp4 -cat Cam1-$dy4.mp4 -cat Cam1-$dy3.mp4 -cat Cam1-$dy2.mp4 -cat Cam1-$dy1.mp4 "Cam1 - $dy1 to $dy7.mp4"
Needless to say help is immensely appreciated for my project
Here is something to get you started. It sorts the individual frames into time order, and then chunks them up into chunks of 500 and loops through all the chunks:
#!/bin/bash
# User-changeable number of frames per chunk
chunksize=500
# Rename files by date/time so they collate in order
jhead -n%Y-%m-%d_%H-%M-%S *.jpg
# Remove any remnants from previous runs (which may have been longer)
rm chunk* sub-*mp4
# Split filename list into chunks - chunkaa, chunkab, chunkac ...
ls *jpg | split -l $chunksize - chunk
# Put 'file' keyword before each filename
sed -i.bak 's/^/file /' chunk*
n=0
for c in chunk*; do
# Generate zero-padded output filename so that collates for final assembly too
out=$(printf "sub-%03d.mp4" $n)
echo Processing chunk $c into sequence $out
ffmpeg -f concat -i "$c" ... "$out"
((n+=1))
done
# Final assembly of "sub-*.mp4"
ffmpeg ... sub-*mp4 ...

ffmpeg - invalid duration

For a project I'm working on I have a small bash script that loops over an input csv file of timecodes, and uses ffmpeg to create screenshots of a given film at each timecode. The csv file is in the format hh:mm:ss,id - it looks like this (extract)
00:00:08,1
00:00:49,2
00:01:30,3
00:02:38,4
00:03:46,5
00:04:08,6
00:04:26,7
00:04:37,8
00:04:49,9
00:05:29,10
00:05:52,11
00:06:00,12
00:06:44,13
00:07:49,14
00:08:32,15
00:09:28,16
00:10:17,17
00:10:44,18
00:11:48,19
00:12:07,20
I've used it without issue in the past, but today I've come to update some of the films and I'm getting a weird issue where ffmpeg is complaining that my input timecode is invalid, despite being in the right format.
The new input csv files are the same format as the old ones, but it seems like every so often ffmpeg drops the hours from the hh:mm:ss timestamp. If I comment out the ffmpeg line, everything prints to the terminal as expected (but obviously I get no screenshots).
This is my loop code:
while read code a
do
echo $code
f="$(printf "%03d" $i)"
ffmpeg -loglevel error -y -ss $code -i $FILM -vframes 1 -q:v 2 $OUTPUT/$f.jpg
((i++))
done < $INPUT
I've tried all sorts, including padding the csv with extra 0s - which works until the hours tick over to 01.
Does anyone have any ideas? I'm scratching my head.
Cheers
Ok - I'm not sure why, but I duplicated the id column to the left of the timestamp column.... so it goes id timestamp id. And it now works as expected.

Tailing a logfile and processing each line is missing data when converting a file with ffmpeg

I am running a script to tail a log file as per the code snippet below. I am running into a problem where by the line passed into $line is missing a number amount of bytes from the beginning when several lines are written to the log file at nearly the same time.
I can check the file afterwards and see that the offending line is complete in the file so why is it incomplete in the script. Some kind of buffering issue perhaps?
The processing can sometimes take several seconds to complete would that make a difference?
#!/bin/bash
tail -F /var/log/mylog.log | while read line
do
log "$line"
ffmpeg -i "from.wav" "to.mp3"
done
Full line in file
"12","","765467657","56753763","test"
example logged $line
657","56753763","test"
Update
I have done some more debugging of my code and it seems the processing that is causing the problem is a call to ffmpeg used to convert a wav to mp3. If I swap that with just a sleep then the problem goes away. Could ffmpeg effect the buffer somehow?
If you are on a platform with a reasonably recent version of GNU Coreutils (e.g. any fairly recent Linux distro), you can use stdbuf to force line buffering.
The example in the stdbuf manpage is highly relevant:
tail -f access.log | stdbuf -oL cut -d ' ' -f1 | uniq
This will immedidately display unique entries from access.log
In a while loop ffmpeg reads from std input, consuming all the arguments at once. To prevent this behavior a common workaround is redirecting ffmpeg's std input to /dev/null, as shown below:
tail -F /var/log/mylog.log | while read line
do
log "$line"
ffmpeg -i "from.wav" "to.mp3" < /dev/null
done
There are also other commands, such as ssh, mplayer, HandBrakeCLI ..., that display the same behavior in a while loop.

Resources