This question already has an answer here:
Script fails with spaces in directory names
(1 answer)
Closed 6 years ago.
I found this script online that I'm trying to edit, but as I'm testing it I can see that it will spit out a bunch of errors for all the files I have with spaces. This is the kind of error log I get on the terminal window:
Skipping 1-03 as ./mp3/basename "$input_file" .wav.mp3 exists.
Skipping The as ./mp3/basename "$input_file" .wav.mp3 exists.
Skipping power, as ./mp3/basename "$input_file" .wav.mp3 exists.
And this is the script:
#!/bin/bash
# Title: wav_to_mp3.sh
# Purpose: Converts all WAV files present in the folder to MP3
# Author: Karthic Raghupathi, IVR Technology Group LLC
# Last Revised: 2014.01.28
# references
sox="/usr/local/bin/sox"
sox_options="-S"
# variables
source_folder="${1:-.}"
destination_folder="${source_folder}/mp3"
environment="${2:-DEVELOPMENT}"
# check to see if an environment flag was supplied
if [ $environment = "PRODUCTION" ] || [ $environment = "production" ]; then
sox="/usr/bin/sox"
environment="PRODUCTION"
fi
# print all params so user can see
clear
echo "Script operating using the following settings and parameters....."
echo ""
echo "which SoX: ${sox}"
echo "SoX options: ${sox_options}"
echo "Environment: ${environment}"
echo "Source: ${source_folder}"
echo "Destination: ${destination_folder}"
echo ""
read -e -p "Do you wish to proceed? (y/n) : " confirm
if [ $confirm = "N" ] || [ $confirm = "n" ]; then
exit
fi
# create destination if it does not exist
if [ ! -d "${destination_folder}" ]; then
mkdir -p "${destination_folder}"
fi
# loop through all files in folder and convert them to
for input_file in $(ls -1 $1 | grep .wav)
do
name_part=`basename "$input_file" .wav`
output_file="$name_part.mp3"
# create mp3 if file does not exist
if [ ! -f "$destination_folder/$output_file" ]; then
$sox $sox_options "${source_folder}/$input_file" "$destination_folder/$output_file"
else
echo "Skipping ${input_file} as $destination_folder/$output_file exists."
fi
done
I know I'm supposed to make it escape the space characters, but I can't figure out how. I tried changing some quotes here and there but I'm just breaking it.
BTW, if anyone would be so kind as to link a good tutorial for learning how to make bash scripts on Mac OS (or Unix), that would be much appreciated. I already know a bit of web programming so I'm not a complete n00b, but still, I'm having trouble creating very simple scripts and I would like to learn independently without constantly bugging the internet for help :)
This is wrong:
for input_file in $(ls -1 $1 | grep .wav)
See here why. Also, inside $1, try this to see that filenames with spaces give trouble:
for i in $(ls -1 | grep wav); do echo $i; done
Try this instead:
for input_file in $1/*.wav
You can escape spaces by inserting a backslash character before the space.
Change:
This file name
To:
This\ file\ name
It might be an idea to write a function to do this for you, iterate through each character in a string and adding a \ caracter before any spaces. that way you don't need to worry about pre-formatting the file names and escaping each individual space - just run the file name through the function and capture the result.
Related
I have list of files at a location ${POWERCENTER_FILE_DIR} .
The files consist of row header and values.
MART_Apple.csv
MART_SAMSUNG.csv
MART_SONY.csv
MART_BlackBerry.csv
Requirements:
select only those files which has atleast 1 row.
Add time stamp to the files which has at least 1 row.
For example:
If all the files except MART_BlackBerry.csv has atleast one row then my output files names should be
MART_Apple_20170811112807.csv
MART_SAMSUNG_20170811112807.csv
MART_SONY_20170811112807.csv
Code tried so far
#!/bin/ksh
infilename=${POWERCENTER_FILE_DIR}MART*.csv
echo File name is ${infilename}
if [ wc -l "$infilename"="0" ];
then
RV=-1
echo "input file name cannot be blank or *"
exit $RV
fi
current_timestamp=`date +%Y%m%d%H%M%S`
filename=`echo $infilename | cut -d"." -f1 `
sftpfilename=`echo $filename`_${current_timestamp}.csv
cp -p ${POWERCENTER_FILE_DIR}$infilename ${POWERCENTER_FILE_DIR}$sftpfilename
RV=$?
if [[ $RV -ne 0 ]];then
echo Adding timestamp to ${POWERCENTER_FILE_DIR}$infilename failed ... Quitting
echo Return Code $RV
exit $RV
fi
Encountering errors like:
line 3: [: -l: binary operator expected
cp: target `MART_Apple_20170811121023.csv' is not a directory
failed ... Quitting
Return Code 1
to be frank, i am not able to apprehend the errors nor i am sure i am doing it right. Beginner in unix scripting.Can any experts guide me where to the correct way.
Here's an example using just find, sh, mv, basename, and date:
find ${POWERCENTER_FILE_DIR}MART*.csv ! -empty -execdir sh -c "mv {} \$(basename -s .csv {})_\$(date +%Y%m%d%H%M%S).csv" \;
I recommend reading Unix Power Tools for more ideas.
When it comes to shell scripting there is rarely a single/one/correct way to accomplish the desired task.
Often times you may need to trade off between readability vs maintainability vs performance vs adhering-to-some-local-coding-standard vs shell-environment-availability (and I'm sure there are a few more trade offs). So, fwiw ...
From your requirement that you're only interested in files with at least 1 row, I read this to also mean that you're only interested in files with size > 0.
One simple ksh script:
#!/bin/ksh
# define list of files
filelist=${POWERCENTER_FILE_DIR}/MART*.csv
# grab current datetime stamp
dtstamp=`date +%Y%m%d%H%M%S`
# for each file in our list ...
for file in ${filelist}
do
# each file should have ${POWERCENTER_FILE_DIR} as a prefix;
# uncomment 'echo' line for debugging purposes to display
# the contents of the ${file} variable:
#echo "file=${file}"
# '-s <file>' => file exists and size is greater than 0
# '! -s <file>' => file doesn't exist or size is equal to 0, eg, file is empty in our case
#
# if the file is empty, skip/continue to next file in loop
if [ ! -s ${file} ]
then
continue
fi
# otherwise strip off the '.csv'
filebase=${file%.csv}
# copy our current file to a new file containing the datetime stamp;
# keep in mind that each ${file} already contains the contents of the
# ${POWERCENTER_FILE_DIR} variable as a prefix; uncomment 'echo' line
# for debug purposes to see what the cp command looks like:
#echo "cp command=cp ${file} ${filebase}.${dtstamp}.csv"
cp ${file} ${filebase}.${dtstamp}.csv
done
A few good resources for learning ksh:
O'Reilly: Learning the Korn Shell
O'Reilly: Learning the Korn Shell, 2nd Edition (includes the newer ksh93)
at your UNIX/Linux command line: man ksh
A simplified script would be something like
#!/bin/bash
# Note I'm using bash above, can't guarantee (but I hope) it would work in ksh too.
for file in ${POWERCENTER_FILE_DIR}/*.csv # Check Ref [1]
do
if [ "$( wc -l "$file" | grep -Eo '^[[:digit:]]+' )" -ne 0 ] # checking at least one row? Check Ref [2]
then
mv "$file" "${file%.csv}$(date +'%Y%m%d%H%M%S').csv" # Check Ref [3]
fi
done
References
File Globbing [1]
Command Substitution [2]
Parameter Substitution [3]
This question already has an answer here:
Mac OS X - Passing pathname with spaces as arguments to bashscript and then issue open Terminal command
(1 answer)
Closed 8 years ago.
So my code works. It's doing what I want to. Essentially my script renames files to match the last two directories in which they are placed, followed by zero padding. Also it takes an argument where if you type in the directory it'll change the files in the specified directory.
Here's my code:
r="$#"
if [ -d "$r" ]; then # if string exists and is a directory then do the following commands
cd "$r" # change directory to the specified name
echo "$r" # print directory name
elif [ -z "$1" ]; then # if string argument is null then do following command
echo "Current Directory" # Print Current Directory
else # if string is not a directory or null then do nothing
echo "No such Directory" # print No such Directory
fi
e=`pwd | awk -F/ '{ print $(NF-1) "_" $NF }'` # print current directory | print only the last two fields
echo $e
X=1;
for i in `ls -1`; do # loop. rename all files in directory to "$e" with 4 zeroes padding.
mv $i $e.$(printf %04d.%s ${X%.*} ${i##*.}) # only .jpg files for now, but can be changed to all files.
let X="$X+1"
done
And here is the output:
Testdir_pics.0001.jpg
Testdir_pics.0002.jpg
...
However, just as the title suggests, it creates errors when the filenames have spaces in them. How do I fix this?
If there are spaces in the file names, then these two lines will fail:
for i in `ls -1`; do
mv $i $e.$(printf %04d.%s ${X%.*} ${i##*.})
Replace them with:
for i in *; do
mv "$i" "$e.$(printf %04d.%s "${X%.*}" "${i##*.}")"
Comments:
for i in * will work for all file names even those with the most difficult characters. By contrast, the for i in $(ls -1) formulation is very fragile.
Unless, for some strange reason, you really want word splitting to be performed on your variables, always place them in double-quotes. Thus, mv $i ... should be replaced with mv "$1" ....
I am quite new to bash scripting, but haven't found an answer to the following problem yet. I hope somebody can either tell me or give me tips on how to do it.
Background: I have a program (say "program") that accepts an ini-file (say "input.ini") as input taking a while to execute. A variable in the ini-file for the programm might be "number" for instance, which might be set to number=1.
Problem: I have to call ./program input.ini quite often, but with different values for "number", say 1,2,3,4,5.
I thought, I could write a bash script executing the program in a for-loop setting "number" accordingly. The loop is not a problem, but setting "number" in the ini-file. (I tried e.g. number=$VALUE in the ini-file with VALUE being set in the script, but this does not work.)
Question: How can I set a variable in the ini-file using a bash-script? (This does not have to be permanent, but only for that run of the program.)
Additional question: Setting a variable in the ini-file is one thing. In addition, it would be great to do the following as well (I thought that might work similarly...): The program produces some output files. The names of these files can also be set in the ini-file, say via "output_name=filename.out". It would be great now if there was something like "output_name=filename_$VALUE.out" to set the output names accordingly.
I hope it is clear what I try to do and I would be really grateful if somebody had a solution or hints on how to do it.
Thanks,
Cari
If you have a file that contains number=something, you can replace "something" with "5" using sed "/^number=/s/=.*/=5/.
This is something you can do once off with process substituion:
./program <(sed "/^number=/s/=.*/=5/" baseinput.ini)
Or you can create a new ini file based on the old one, as in
sed "/^number=/s/=.*/=5/" baseinput.ini > input.ini
./program input.ini
You could also define the entire ini file the script, and substitute in a here document:
N=5
./program - << EOF
[Section]
number=$N
foo=bar
EOF
full parsed and set ini file (section,key,value) and save on root.
sudo_setini ()
{
fkey=false
fsec=false
tsec=false
res=""
if [ -f "$1" ]; then
while IFS= read -r LINE
do
TLINE=`echo $LINE`
if [[ $TLINE == \[*] ]]; then
TLINE=`echo ${TLINE:1:${#TLINE}-2}`
if [[ _${TLINE} == _$2 ]]; then
tsec=true
fsec=true
else
if $tsec && ! $fkey ; then
res+=$'\n'$3=$4
fi
tsec=false
fi
res+=$'\n'${LINE}
else
TLINE=`echo ${TLINE%%=*}`
if $tsec && [[ _${TLINE} == _$3 ]]; then
fkey=true
res+=$'\n'${LINE%%=*}=$4
else
res+=$'\n'${LINE}
fi
fi
done < $1
fi
if $tsec && ! $fkey ; then
res+=$'\n'$3=$4
fi
if ! $fsec ; then
res+=$'\n'[$2]
res+=$'\n'$3=$4
fi
echo "$res" | sudo tee "$1" > /dev/null
}
sudo_setini 'test.ini' 'General' 'Type' 'Digital_'
Not quite sure whether this helps or not:
This calls the program script five times:
for n in 1 2 3 4 5
do
./program $n input.ini
done
Then in program, refer to the first parameter $n as $1.
The second parameter input.ini is $2.
If you have git available and you're not worried about indentation, a hack could be to use git config.
Example:
$ git config -f settings.ini server.ip 123.123.123.123
$ cat settings.ini
[server]
ip = 123.123.123.123
$ git config -f settings.ini server.ip 123.123.123.124
$ cat settings.ini
[server]
ip = 123.123.123.124
I am looking to search for strings within a file using variables.
I have a script that will accept 3 or 4 parameters: 3 are required; the 4th isn't mandatory.
I would like to search the text file for the 3 parameters matching within the same line, and if they do match then I want to remove that line and replace it with my new one - basically it would update the 4th parameter if set, and avoid duplicate entries.
Currently this is what I have:
input=$(egrep -e '$domain\s+$type\s+$item' ~/etc/security/limits.conf)
if [ "$input" == "" ]; then
echo $domain $type $item $value >>~/etc/security/limits.conf
echo \"$domain\" \"$type\" \"$item\" \"$value\" has been successfully added to your limits.conf file.
else
cat ~/etc/security/limits.conf | egrep -v "$domain|$type|$item" >~/etc/security/limits.conf1
rm -rf ~/etc/security/limits.conf
mv ~/etc/security/limits.conf1 ~/etc/security/limits.conf
echo $domain $type $item $value >>~/etc/security/limits.conf
echo \"$domain\" \"$type\" \"$item\" \"$value\" has been successfully added to your limits.conf file.
exit 0
fi
Now I already know that the input=egrep etc.. will not work; it works if I hard code some values, but it won't accept those variables. Basically I have domain=$1, type=$2 and so on.
I would like it so that if all 3 variables are not matched within one line, than it will just append the parameters to the end of the file, but if the parameters do match, then I want them to be deleted, and appended to the file. I know I can use other things like sed and awk, but I have yet to learn them.
This is for a school assignment, and all help is very much appreciated, but I'd also like to learn why and how it works/doesn't, so if you can provide answers to that as well that would be great!
Three things:
To assign the output of a command, use var=$(cmd).
Don't put spaces around the = in assignments.
Expressions don't expand in single quotes: use double quotes.
To summarize:
input=$(egrep -e "$domain\s+$type\s+$item" ~/etc/security/limits.conf)
Also note that ~ is your home directory, so if you meant /etc/security/limits.conf and not /home/youruser/etc/security/limits.conf, leave off the ~
You have several bugs in your script. Here's your script with some comments added
input=$(egrep -e '$domain\s+$type\s+$item' ~/etc/security/limits.conf)
# use " not ' in the string above or the shell can't expand your variables.
# some versions of egrep won't understand '\s'. The safer, POSIX character class is [[:blank:]].
if [ "$input" == "" ]; then
# the shell equality test operator is =, not ==. Some shells will also take == but don't count on it.
# the normal way to check for a variable being empty in shell is with `-z`
# you can have problems with tests in some shells if $input is empty, in which case you'd use [ "X$input" = "X" ].
echo $domain $type $item $value >>~/etc/security/limits.conf
# echo is unsafe and non-portable, you should use printf instead.
# the above calls echo with 4 args, one for each variable - you probably don't want that and should have double-quoted the whole thing.
# always double-quote your shell variables to avoid word splitting ad file name expansion (google those - you don't want them happening here!)
echo \"$domain\" \"$type\" \"$item\" \"$value\" has been successfully added to your limits.conf file.
# the correct form would be:
# printf '"%s" "%s" "%s" "%s" has been successfully added to your limits.conf file.\n' "$domain" "$type" "$item" "$value"
else
cat ~/etc/security/limits.conf | egrep -v "$domain|$type|$item" >~/etc/security/limits.conf1
# Useless Use Of Cat (UUOC - google it). [e]grep can open files just as easily as cat can.
rm -rf ~/etc/security/limits.conf
# -r is for recursively removing files in a directory - inappropriate and misleading when used on a single file.
mv ~/etc/security/limits.conf1 ~/etc/security/limits.conf
# pointless to remove the file above when you're overwriting it here anyway
# If your egrep above failed to create your temp file (e.g. due to memory or permissions issues) then the "mv" above would zap your real file. the correct way to do this is:
# egrep regexp file > tmp && mv tmp file
# i.e. use && to only do the mv if creating the tmp file succeeded.
echo $domain $type $item $value >>~/etc/security/limits.conf
# see previous echo comments.
echo \"$domain\" \"$type\" \"$item\" \"$value\" has been successfully added to your limits.conf file.
# ditto
exit 0
# pointless and misleading having an explicit "exit <success>" when that's what the script will do by default anyway.
fi
This line:
input=$(egrep -e '$domain\s+$type\s+$item' ~/etc/security/limits.conf)
requires double quotes around the regex to allow the shell to interpolate the variable values.
input=$(egrep -e "$domain\s+$type\s+$item" ~/etc/security/limits.conf)
You need to be careful with backslashes; you probably don't have to double them up in this context, but you should be sure you know why.
You should be aware that your first egrep commands is much more restrictive in what it selects than the second egrep which is used to delete data from the file. The first requires the entry with the three fields in the single line; the second only requires a match with any one of the words (and that could be part of a larger word) to delete the line.
Since ~/etc/security/limits.conf is a file, there is no need to use the -r option of rm; it is advisable not to use the -r unless you intend to remove directories.
I'm writing a bash shell script that uses a case with three options:
If the user enters "change -r txt doc *", a file extension gets changed in a subdirectory.
If a user enters "change -n -r doc ", it should rename files that end with .-r or .-n (this will rename all files in the current directory called *.-r as *.doc)
If the user enters nothing, as in "change txt doc *", it just changes a file extension in the current directory.
Here's the code i produced for it (the last two options, i'm not sure how to implement):
#!/bin/bash
case $1 in
-r)
export currectFolder=`pwd`
for i in $(find . -iname "*.$2"); do
export path=$(readlink -f $i)
export folder=`dirname $path`
export name=`basename $path .$2`
cd $folder
mv $name.$2 $name.$3
cd $currectFolder
done
;;
-n)
echo "-n"
;;
*)
echo "all"
esac
Can anyone fix this for me? Or at least tell me where i'm going wrong?
What you should brush up on are string substitutions. All kinds of them actually. Bash is very good with those. Page 105 (recipe 5.18) of the Bash Cookbook is excellent reading for that.
#!/bin/bash
# Make it more flexible for improving command line parsing later
SWITCH=$1
EXTENSIONSRC=$2
EXTENSIONTGT=$3
# Match different cases for the only allowed switch (other than file extensions)
case $SWITCH in
-r|--)
# If it's not -r we limit the find to the current directory
[[ "x$SWITCH" == "x-r" ]] || DONTRECURSE="-maxdepth 1"
# Files in current folder with particular pattern (and subfolders when -r)
find . $DONTRECURSE -iname "*.$EXTENSIONSRC"|while read fname; do
# We use a while to allow for file names with embedded blank spaces
# Get canonical name of the item into CFNAME
CFNAME=$(readlink -f "$fname")
# Strip extension through string substitution
NOEXT_CFNAME="${CFNAME%.$EXTENSIONSRC}"
# Skip renaming if target exists. This can happen due to collisions
# with case-insensitive matching ...
if [[ -f "$NOEXT_CFNAME.$EXTENSIONTGT" ]]; then
echo "WARNING: Skipping $CFNAME"
else
echo "Renaming $CFNAME"
# Do the renaming ...
mv "$CFNAME" "$NOEXT_CFNAME.$EXTENSIONTGT"
fi
done
;;
*)
# The -e for echo means that escape sequences like \n and \t get evaluated ...
echo -e "ERROR: unknown command line switch\n\tSyntax: change <-r|--> <source-ext> <target-ext>"
# Exit with non-zero (i.e. failure) status
exit 1
esac
The syntax is obviously given in the script. I took the freedom to use the convention of -- separating command line switches from file names. This way it looks cleaner and is easier to implement, actually.
NB: it is possible to condense this further. But here I was trying to get a point across, rather than win the obfuscated Bash contest ;)
PS: also handles the case-insensitive stuff now in the renaming part. However, I decided to make it skip if the target file already exists. Can perhaps be rewritten to be a command line option.