Sorry for this questions I imagine the answer is pretty straightforward however I have had a search and I haven't found anything that answers it.
I have written the following:
while read p # reads in each line of the text
do # file- folder_list.txt each iteration.
echo "$p" # print out current line
if [ -f $p/frame_number1.jpg ] # checks if the image exists in the specific folder
then
echo "$p" #prints specific folder name
sleep 1 #pause for 1 second
ffmpeg -f image2 -r 10 -i $p/frame_number%01d.jpg -r 30 $p/out.mp4 #create video
fi # end if statement
done <folder_list.txt #end of while loop
The script is supposed to read a text file which contains the folder tree structure, then check if the folder contains the specified JPEG if it does then the code should create a video from the images contained within the specified folder.
However what appears to be happening is the script skips whole folders that definitely contain images. I was wondering if the while loop is continuing to iterate whilst the video is being created or if something else is happening?
Any help would be greatly appreciated.
Many thanks in advance
Laurence
It could be because some folders contain spaces, or other weird characters.
Try this (won't work for ALL but for MOST cases):
while read p # reads in each line of the text file- folder_list.txt
do #each iteration
echo "$p" #print out current line
if [ -f "${p}/frame_number1.jpg" ] #checks wether the image exists in the specific folder
then echo "$p" #printsout specific folder name
sleep 1 #pause for 1 second
ffmpeg -f image2 -r 10 -i "${p}/frame_number%01d.jpg" -r 30 "${p}/out.mp4" #create video
fi # end if statement
done <folder_list.txt #end of while loop
I just added quotes around arguments involving "$p" so that if it contains space (and other things) it's not separated into several arguments, breaking the command
If this doesn't work, tell us exactly a few of the directories that are OK and the ones that are NOT ok :
ls -ald /some/ok/dir /some/NOTOK/dir #will show which directories belong to who
id #will tell us which user you are using, so we can compare to the previous and find out why you can't access some
Related
I'm attempting to write a program that moves zipped files that arrive in a directory, unzips them and then outputs the contents.
#!/bin/bash
shopt -s extglob
echo "Press [CTRL+C] to stop.."
#begin loop
while :
do
#search folder test_source for files and append names to array
queue+=($(ls /home/ec2-user/glutton/test_source | egrep 'test[0-9]{1}.gz'))
for i in $queue; do
#move file in test_source to temp folder
mv /home/ec2-user/glutton/test_source/${queue[i]} /home/ec2-user/glutton/temp
#unzip file
gunzip /home/ec2-user/glutton/temp/${queue[i]}
#add new file name to variable unzipped
unzipped=($(ls /home/ec2-user/glutton/temp | egrep 'test[0-9]{1}'))
cat temp/$unzipped
#Test for successful run
exit_status=$?
if [ $exit_status -eq 1 ]; then
#If fail move file to bank and output
mv /home/ec2-user/glutton/temp/$unzipped /home/ec2-user/glutton/logs/bank
echo "Error! $unzipped moved to /home/ec2-user/glutton/logs/bank"
#write to error.log
echo "moved ${queue[i]} to bank due to error" >> /home/ec2-user/glutton/logs/error.log
else
#If success delete file
rm /home/ec2-user/glutton/temp/$unzipped
fi
#wipe variable
unset unzipped
i=$i+1
done
#wipe array
unset queue
i=0
#go to top of loop
done
This has worked pretty well up until I added the unzipping feature and now my program outputs this error when attempting to move the .gz file:
./glutton.sh: line 11: test0.gz: syntax error: invalid arithmetic operator (error token is ".gz")
When I run the first part of my script in the command line it seems to work perfectly, but doesn't when I run it on its own, I'm pretty confused.
Your main issue is that when you iterate an array like you are doing, you get the first item of the array, not the index. So in your case, $i is not a number, it is the filename (i.e. test1.gz) and it will only see the first file. The cleanest way I have seen to iterate the items in an array would be for i in "${arrayName[#]}".
Also, using '{1}' in your regex is redundant, the character class will already match only 1 character if you don't specify a modifier.
It shouldn't matter depending on the contents of your 'temp' folder, but I would be more specific on your egreps too, if you add -x then it will have to match the whole string. As it is currently, a file called 'not_a_test1' would match.
I have a number (say, 100) of CSV files, out of which some (say, 20) are empty (i.e., 0 bytes file). I would like to concatenate the files into one single CSV file (say, assorted.csv), with the following requirement met:
For each empty file, there must be a blank line in assorted.csv.
It appears that simply doing cat *.csv >> assorted.csv skips the empty files completely in the sense that they do not have any lines and hence there is nothing to concatenate.
Though I can solve this problem using any high-level programming language, I would like to know if and how to make it possible using Bash.
Just make a loop and detect if the file is not empty. If it's empty, just echo the file name+comma in it: that will create a near blank line. Otherwise, prefix each line with the file name+comma.
#!/bin/bash
out=assorted.csv
# delete the file prior to doing concatenation
# or if ran twice it would be counted in the input files!
rm -f "$out"
for f in *.csv
do
if [ -s "$f" ] ; then
#cat "$f" | sed 's/^/$f,/' # cat+sed is too much here
sed "s/^/$f,/" "$f"
else
echo "$f,"
fi
done > $out
Short story: I'm trying to write a script that will use FFmpeg to convert the many files stored in one directory to a "standard" mp4 format and save the converted files in another directory. It's been a learning experience (a fun one!) since I haven't done any real coding since using Pascal and FORTRAN on an IBM 370 mainframe was in vogue.
Essentially the script takes the filename, strips the path and extension off it, reassembles the filename with the path and an mp4 extension and calls FFmpeg with some set parameters to do the conversion. If the directory contains only video files with without spaces in the names, then everything works fine. If the filenames contain spaces, then FFmpeg is not able to process the file and moves on to the next one. The error indicates that FFMpeg is only seeing the filename up to the first space. I've included both the script and output below.
Thanks for any help and suggestions you may have. If you think I should be doing this in another way, please by all means, give me your suggestions. As I said, it's been a long time since I did anything like this. I'm enjoying it though.
I've include the code first followed by example output.
for file in ./TBC/*.mp4
do
echo "Start of iteration"
echo "Full text of file name:" $file
#Remove everything up to "C/" (filename without path)
fn_orig=${file#*C/}
echo "Original file name:" $fn_orig
#Length of file name
fn_len=${#fn_orig}
echo "Filename Length:" $fn_len
#file name without path or extension
fn_base=${fn_orig:0:$fn_len-4}
echo "Base file name:" $fn_base
#new filename suffix
newsuffix=".conv.mp4"
fn_out=./CONV/$fn_base$newsuffix
echo "Converted file name:" $fn_out
ffmpeg -i $file -metadata title="$fn_orig" -c:v libx264 -c:a libfdk_aac -b:a 128k $fn_out
echo "End of iteration"
echo
done
echo "Script completed"
With the ffmpeg line commented out, and two files in the ./TBC directory, this is the output that I get
Start of iteration
Full text of file name: ./TBC/Test file with spaces.mp4
Original filename: Test file with spaces.mp4
Filename Length: 25
Base filename: Test file with spaces
Converted file name: ./CONV/Test file with spaces.conv.mp4
End of iteration
Start of iteration
Full text of file name: ./TBC/Test_file_with_NO_spaces.mp4
Original file name: Test_file_with_NO_spaces.mp4
Filename Length: 28
Base file name: Test_file_with_NO_spaces
Converted file name: ./CONV/Test_file_with_NO_spaces.conv.mp4
End of iteration
Script completed
I won't bother to post the results when ffmpeg is uncommented, other than to state that it fails with the error:
./TBC/Test: No such file or directory
The script then continues to the next file which completes successfully because it has no spaces in its name. The actual filename is "Test file with spaces.mp4" so you can see that ffmpeg stops after the word "Test" when it encounters a space.
I hope this has been clear and concise and hopefully someone will be able to point me in the right direction. There is a lot more that I want to do with this script such as parsing subdirectories and ignoring non-video files, etc.
I look forward to any insight you can give!
try quoting you output file:
ffmpeg -i "$file" ... "$fn_out"
bash separates arguments based on spaces, so you have to tell him that $fn_out is one single argument; whence the "" to show that this is one argument.
There is another edge-case where spaces break bash for loops.
"BASH for loop works nicely under UNIX / Linux / Windows and OS X while working on set of files. However, if you try to process a for loop on file name with spaces in them you are going to have some problem. For loop uses $IFS variable to determine what the field separators are. By default $IFS is set to the space character..."
https://www.cyberciti.biz/tips/handling-filenames-with-spaces-in-bash.html
Before:
for file in $(find . -name '*.txt'); do echo "$file"; done
Outputs:
./files/my
documents/item1.txt
./files/my
documents/item2.txt
./files/my
documents/item3.txt
Therefore you should set IFS to ignore spaces.
After:
IFS=$'\n'
for file in $(find . -name '*.txt'); do echo "$file"; done
Outputs:
./files/my documents/item1.txt
./files/my documents/item2.txt
./files/my documents/item3.txt
Is there a shorthand in bash to select an arbitrary file? * enumerates all files in the current directory, but what if I only want one file and don't care which it is?
FWIW I'm testing several different ffmpeg commands in a directory with similarly named video files, so tab-complete is cumbersome.
Here's the robust way of getting the first or a random file in a directory, handling the edge case of not having any files:
#!/bin/bash
# Let globs expand to 0 elements instead of themselves if no matches
shopt -s nullglob
# Add all the files in the current dir to an array
files=(*)
# Check if the array has any elements
if [[ ${#files[#]} -gt 0 ]]
then
first_file=${files[0]}
random_file=${files[RANDOM%${#files[#]}]}
echo "The first file is ${first_file}"
echo "A random file is ${random_file}"
else
echo "There are no files in the current directory."
fi
If you just want something short and hacky for interactive testing, you can create an array and reference it unindexed to get the first element with minimal typing:
$ testfile=( *.avi )
$ ffmpeg -i "$testfile" test.mp3
You can also bind Tab to zsh style completion:
$ bind 'TAB:menu-complete'
now, for the rest of this session, when you press Tab you'll get a complete filename instead of just a prefix (press Tab again to cycle through matches). This will let you conveniently pick a file with a single keystroke.
Occasionally I was using the shuf:
find -name '*whatever*' | shuf | head -n 1
The shuf is a tool, part of GNU coreutils, which prints the input lines in random order. In other words, it shuffles the lines.
this is my first post so hopefully I will make my question clear.
I am new to shell scripts and my task with this one is to add a new value to every line of a csv file. The value that needs added is based on the first 3 digits of the filename.
I bit of background. The csv files I am receiving are eventually being loaded into partitioned oracle tables. The start of the file name (e.g. BATTESTFILE.txt) contains the partitioned site so I need to write a script that takes the first 3 characters of the filename (in this example BAT) and add this to the end of each line of the file.
The closest I have got so far is when I stripped the code to the bare basics of what I need to do:
build_files()
{
OLDFILE=${filename[#]}.txt
NEWFILE=${filename[#]}.NEW.txt
ABSOLUTE='path/scripts/'
FULLOLD=$ABSOLUTE$OLDFILE
FULLNEW=$ABSOLUTE$NEWFILE
sed -e s/$/",${j}"/ "${FULLOLD}" > "${FULLNEW}"
}
set -A site 'BAT'
set -A filename 'BATTESTFILE'
for j in ${site[#]}; do
for i in ${filename[#]}; do
build_files ${j}
done
done
Here I have set up an array site as there will be 6 'sites' and this will make it easy to add additionals sits to the code as the files come through to me. The same is to be siad for the filename array.
This codes works, but it isn't as automated as I need. One of my most recent attempts has been below:
build_files()
{
OLDFILE=${filename[#]}.txt
NEWFILE=${filename[#]}.NEW.txt
ABSOLUTE='/app/dss/dsssis/sis/scripts/'
FULLOLD=$ABSOLUTE$OLDFILE
FULLNEW=$ABSOLUTE$NEWFILE
sed -e s/$/",${j}"/ "${FULLOLD}" > "${FULLNEW}"
}
set -A site 'BAT'
set -A filename 'BATTESTFILE'
for j in ${site[#]}; do
for i in ${filename[#]}; do
trust=echo "$filename" | cut -c1-3
echo "$trust"
if ["$trust" = 'BAT']; then
${j} = 'BAT'
fi
build_files ${j}
done
done
I found the code trust=echo "$filename" | cut -c1-3 through another question on StackOverflow as I was researching, but it doesn't seem to work for me. I added in the echo to test what trust was holding, but it was empty.
I am getting 2 errors back:
Line 17 - BATTESTFILE: not found
Line 19 - test: ] missing
Sorry for the long winded questions. Hopefully It contains helpful info and shows the steps I have taken. Any questions, comment away. Any help or guidance is very much appreciated. Thanks.
When you are new with shells, try avoiding arrays.
In an if statement use spaces before and after the [ and ] characters.
Get used to surrounding your shell variables with {} like ${trust}
I do not know how you fill your array, when the array is hardcoded, try te replace with
SITE=file1
SITE="${SITE} file2"
And you must tell unix you want to have the rightside eveluated with $(..) (better than backtics):
trust=$(echo "${filename}" | cut -c1-3)
Some guidelines and syntax help can be found at Google
Just use shell parameter expansion:
$ var=abcdefg
$ echo "${var:0:3}"
abc
Assuming you're using a reasonably capable shell like bash or ksh, for example
Just in case it is useful for anyone else now or in the future, I got my code to work as desired by using the below. Thanks Walter A below for his answer to my main problem of getting the first 3 characters from the filename and using them as a variable.
This gave me the desired output of taking the first 3 characters of the filename, and adding them to the end of each line in my csv file.
## Get the current Directory and file name, create a new file name
build_files()
{
OLDFILE=${i}.txt
NEWFILE=${i}.NEW.txt
ABSOLUTE='/app/dss/dsssis/sis/scripts/'
FULLOLD=$ABSOLUTE$OLDFILE
FULLNEW=$ABSOLUTE$NEWFILE
## Take the 3 characters from the filename and
## add them onto the end of each line in the csv file.
sed -e s/$/";${j}"/ "${FULLOLD}" > "${FULLNEW}"
}
## Loop to take the first 3 characters from the file names held in
## an array to be added into the new file above
set -A filename 'BATTESTFILE'
for i in ${filename[#]}; do
trust=$(echo "${i}" | cut -c1-3)
echo "${trust}"
j="${trust}"
echo "${i} ${j}"
build_files ${i} ${j}
done
Hope this is useful for someone else.