First of all, this question is purely theoric; it involves creates a directory with a newline, thing that should NEVER be done.
That said, I'm trying to use mkdir -pv output to remove the created directories in a specific moment of my script, but only the newly created, not the ones that previously existed.
Command mkdir -pv will print one line per directory not-existent before this command call so that I can re-inject in a rm -rf command. It works OK except in the case that directory contains a newline, and I can't see what is wrong with it.
My minimal working example:
declare -a created
# Delete previous traces
mkdir_out=$(mkdir -pv 'new 10'{1,2,3,$'\n',"'a",4})
# Convert to array
IFS=$'\n' read -d '' -a created < <(printf '%s' "${mkdir_out}")
# Debug
printf '=>[1] %s\n' "${created[#]}"
# We only want content between first and last quote
created=( "${created[#]%[\'\"]}" )
created=( "${created[#]#*[\'\"]}" )
# Debug
printf '=>[2] %s\n' "${created[#]}"
rm -rfv "${created[#]}"
ls # Directory "new 10\n" is still there!!
So, what is the safe way to do that?
Output like mkdir: created directory 'foo' is only meant for humans. Don't try to parse it.
If you want to handle all possible filenames and you can't deal in \0 separated lists, you have to do them one by one. Here's an example:
declare -a created dirs
dirs=( 'new 10'{1,2,3,$'\n',"'a",4} )
created=()
for dir in "${dirs[#]}"
do
if [[ ! -d "$dir" ]] && mkdir -p "$dir"
then
created+=( "$dir" )
fi
done
rm -rfv "${created[#]}"
ls # Directory "new 10\n" is not there.
Related
Below is a bash script to move files around and rename them. The problem is it doesn't work when there is more than one file in the directory. I'm assuming because the last parameter in the mv command is a file. Any suggestions?
'#!/bin/bash'
'INPUTDIR="/home/southern-uniontn/S001007420"'
'OUTPUTDIR="/mnt/edi-06/southern-uniontn/flats-in"'
'BACKUPDIR="/backup/southern-uniontn/S001007420"'
YEAR=`date +%Y`
MONTH=`date +%m`
DAY=`date +%d`
HOUR=`date +%H`
MINUTE=`date +%M`
######## Do some error checking #########
# Does backup dir exist?
if [ ! -d $BACKUPDIR/$YEAR ]
then
mkdir $BACKUPDIR/$YEAR
fi
if [ ! -d $BACKUPDIR/$YEAR/$MONTH ]
then
mkdir $BACKUPDIR/$YEAR/$MONTH
fi
if [ ! -d $BACKUPDIR/$YEAR/$MONTH/$DAY ]
then
mkdir $BACKUPDIR/$YEAR/$MONTH/$DAY
fi
if [[ $(find $INPUTDIR -type f | wc -l) -gt 0 ]];
then
###### Rename the file, move it to Backup, then copy to the Output Directory #####
for f in $INPUTDIR/*
do
echo "`date` - Move recurring txt flat file to BackupDir for Union TN from Southern"
mv $INPUTDIR/* $BACKUPDIR/$YEAR/$MONTH/$DAY/UnionTN-S001007420-$YEAR$MONTH$DAY-$HOUR$MINUTE.txt
sleep 2
echo "`date` - Copy backup file to the Union TN Output Directory"
cp $BACKUPDIR/$YEAR/$MONTH/$DAY/UnionTN-S001007420-$YEAR$MONTH$DAY-$HOUR$MINUTE.txt $OUTPUTDIR/
done;
fi
Some notes:
Get out of the habit of using ALLCAPS variable names, leave those as reserved
by the shell. One day you'll write PATH=something and then wonder
why your script is
broken.
mkdir -p can create parent directories, and will not error if the dir already exists
store the filenames in an array. Then the shell does not have to duplicate
the work, and you don't need to count how many there are: if there are no
files, the loop has zero iterations
if you want to keep the same directory hierarchy in the outputdir,
you need to do that by hand.
use read to get the date parts
with bash v4.2+, printf can be used instead of calling out to date
use magic value "-1" to mean "now".
printf '%(%Y-%m-%d)T\n' -1 prints "2021-10-25" (as of the day I write this)
This is, I think, what you want:
#!/bin/bash
inputdir='/home/southern-uniontn/S001007420'
outputdir='/mnt/edi-06/southern-uniontn/flats-in'
backupdir='/backup/southern-uniontn/S001007420'
read year month day hour minute < <(printf '%(%Y %m %d %H %M)T\n' -1)
# create backup dirs if not exists
date_dir="$year/$month/$day"
mkdir -p "$backupdir/$date_dir"
mkdir -p "$outputdir/$date_dir"
mapfile -t files < <(find $inputdir -type f)
for f in "${files[#]}"
do
###### Rename the file, move it to Backup, then copy to the Output Directory #####
backup_file="UnionTN-S001007420-$year$month$day-$hour$minute.txt"
printf '%(%c)T - Move recurring txt flat file to backupdir for Union TN from Southern\n' -1
mv "$f" "$backupdir/$date_dir/$backup_file"
printf '%(%c)T - Copy backup file to the Union TN Output Directory\n' -1
cp "$backupdir/$date_dir/$backup_file" "$outputdir/$date_dir/$backup_file"
done
When using a glob with mv, the target must be an existing directory, and all matching files will be moved inside that directory.
In your case,
mv $INPUTDIR/* $BACKUPDIR/$YEAR/$MONTH/$DAY/UnionTN-S001007420-$YEAR$MONTH$DAY-$HOUR$MINUTE.txt
tells mv to move all file inside the $INPUTDIR/* directory to a directory named $BACKUPDIR/$YEAR/$MONTH/$DAY/UnionTN-S001007420-$YEAR$MONTH$DAY-$HOUR$MINUTE.txt.
I'm not sure what you're trying to do, but I hope this help.
Some more advice you could use:
Don't put the shebang (the first line beginning with "#") and the first three variable declarations inside single-quotes.
Some argue it is more portable and better to write /usr/bin/env bash instead of /bin/bash in the shebang
if [ CONDITION ] /then ACTION /fi statements can be simplified by writing [ CONDITION ] && ACTION
You reduce your likely hood of encountering unexpected behaviour when double-quoting your strings and variable (i.e. write "${year}/${month}/" instead of $year/$month.
No need to call mkdir a, followed by mkidr a/b, then mkdir a/b/c and so on, you can just call mkdir -p a/b/c. The p flag tells mkdir to create parent directories if they don't already exist.
It is unnecessary to validate the existence of a directory before calling mkdir since mkdir already validates that for you.
As pointed out by commenters, all-caps variables are conventions for special POSIX related variables. You should use another type of casing.
You could use date to do the formatting for you: date +%Y/%m/%d will print 2021/10/25
Strings without interpolation can have single-quotes.
(Optional, prevent undesired behaviors) Put set -e at the beginning of your scripts, after the shebang, to tell bash to halt if an error is encountered
And finally, use man <command_name> for built-in documentation!
I have a small script that I use to organizes files in directories, and now I am attempting to run it on a folder or directories. MasterDir/dir1, MasterDir/dir2, etc.
Running the following code on MasterDir results in an error as the $dir is only a relative path, but I can't figure out how to get the full path of $dir in this case
for dir in */; do
echo $dir
cd $dir
cwd="$PWD"
mkdir -p "VSI"
mv -v *.vsi "$cwd/VSI"
mv -v _*_ "$cwd/VSI"
done
I'd suggest using parentheses to run the loop body in a subshell:
for dir in */; do
(
echo $dir
cd $dir
cwd="$PWD"
mkdir -p "VSI"
mv -v *.vsi "$cwd/VSI"
mv -v _*_ "$cwd/VSI"
)
done
Since that's running in a subshell, the cd command won't affect the parent shell executing the script.
The problem you are having is that after you cd "$dir" the first time, you are one directory below where you generated your list of directories with for dir in */. So the next time you call cd "$dir" it fails because you are still in the first subdirectory you cd'ed into and the next "$dir" in your list is one level above.
There are several ways to handle this. One simple one is to use pushd to change to the directory instead of cd, so you can popd and return to your original directory. (though in this case you could simply add cd .. to change back to the parent directory since you are only one-level deep)
Using pushd/popd you could do:
for dir in */; do
echo $dir
pushd "$dir" &>/dev/null || {
printf "error: failed to change to %s\n" "$dir" >&2
continue
}
cwd="$PWD"
mkdir -p "VSI" || {
printf "error: failed to create %s\n" "$cwd/VSI" >&2
continue
}
mv -v *.vsi "$cwd/VSI"
mv -v _*_ "$cwd/VSI"
popd &>/dev/null || {
printf "error: failed to return to parent dir\n" >&2
break
}
done
(note: the || tests validate the return of pushd, mkdir, popd causing the loop to either continue to the next dir or break the loop if you can't return to the original directory. Also note the &>/dev/null just suppresses the normal output of pushd/popd, and redirection of output to >&2 sends the output to stderr instead of stdout)
As mentioned in the comment, you can always use readlink -f "$dir" to generate the absolute path to "$dir" -- though it's not really needed here.
This is one of the reasons I tend to avoid using cd in shell scripts -- all relative file paths change meaning when you cd, and if you aren't very careful about that you can get into trouble. The other is that cd can fail (e.g. because of a permissions error), in which case you'd better have an error check & handler in place, or something even weirder will happen.
IMO it's much safer to just use explicit paths to refer to files in other directories. That is, instead of cd somedir; mkdir -p "VSI", use `mkdir -p "somedir/VSI". Here's a rewrite of your loop using this approach:
for dir in */; do
echo $dir
mkdir -p "${dir}/VSI"
mv -v "${dir}"/*.vsi "${dir}/VSI"
mv -v "${dir}"/_*_ "${dir}/VSI"
done
Note: the values of $dir will end with a slash, so using e.g. ${dir}/VSI will give results like "somedir//VSI". The extra slash is redundant, but shouldn't cause trouble; I prefer to use it for clarity. If it's annoying (e.g. in the output of mv -v), you can leave off the extra slash.
I have this shell script that I'm using to clean up some temp files.
The script is stored in: /root/cronjobs.
When I run the script from this location ./cleanUploader.sh, it deletes all the files in the current folder along with itself.
Here's the script:
#!/bin/bash
# cleanUploader.sh
# Batch file to remove various temp directories and files left over from the Uploader
clear
echo
INHOUSEFILES=/var/www/html/inhouseweb/officedb/uploader/files
shopt -s extglob
if [ -d $INHOUSEFILES ]; then
echo "Removing directory $INHOUSEFILES"
rm -rf $INHOUSEFILES/* !(".gitignore")
else
echo "directory $INHOUSEFILES not found"
fi
echo
shopt -u extglob
echo
echo "Done"
What am I doing wrong?
rm -rf $INHOUSEFILES/* !(".gitignore")
This deletes all files in $INHOUSEFILES/*, and then it also deletes everything in the current directory except .gitignore. That's what !(".gitignore") does when it's a separate argument.
If your intention is to delete everything in $INHOUSEFILES/ except .gitignore then combine the two arguments:
rm -rf $INHOUSEFILES/!(".gitignore")
It's also a good idea to quote variable expansions. (And conversely you don't need them around a literal string like .gitignore.)
rm -rf "$INHOUSEFILES"/!(.gitignore)
I have a folder containing these app files:
mles:fairs-ionic mles$ ls build
build/com.solutions.enterprise.fairs.dev_v2.2.0.17_20170720_1423-enterprise.ipa
build/com.solutions.enterprise.fairs_v2.2.0.17_20170720_1423-enterprise.ipa
build/de.fairs.dev_v2.2.0.17_20170720_1423-enterprise.apk
build/de.fairs_v2.2.0.17_20170720_1423-appstore.ipa
build/de.fairs_v2.2.0.17_20170720_1423-enterprise.apk
I need to uppload all of them except the *-appstore.ipa one. More specifically these one:
mles:fairs-ionic mles$ ls build/{*-enterprise.ipa,*.apk}
build/com.solutions.enterprise.fairs.dev_v2.2.0.17_20170720_1423-enterprise.ipa
build/com.solutions.enterprise.fairs_v2.2.0.17_20170720_1423-enterprise.ipa
build/de.fairs.dev_v2.2.0.17_20170720_1423-enterprise.apk
build/de.fairs_v2.2.0.17_20170720_1423-enterprise.apk
In my bash script I've tried:
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
appFiles=($(cd ${DIR}/build;ls {*-enterprise.ipa,*.apk}))
echo ${appFiles};
echo "loop"
for appFile in "${appFiles[#]}"
do
echo ${appfile}
#app_upload "${appfile}"
done
this yields:
mles:fairs-ionic ben$ ./test.sh
com.solutions.enterprise.fairs.dev_v2.2.0.17_20170720_1423-enterprise.ipa
loop
appFiles only contains one row, and the appfile variable in the loop is always empty.
How can I iterate over all the files in the build folder except the .ipa files with appstore in the filename (build/de.fairs_v2.2.0.17_20170720_1423-appstore.ipa) ?
Variable appfile is not defined only appFile, So change
echo ${appfile}
to
echo ${appFile}
Edit the line displaying all the data
echo ${appFiles};
To
echo ${appFiles[*]};
You can just do this simply with extended glob features (see Options which change globbing behavior) provided by bash, turn the options on and stash the values to an array. Run it outside the build/ folder.
shopt -s extglob nullglob
fileList=( build/!(*-appstore.ipa) )
now loop over the array and do whatever you want to do with it.
for file in "${fileList[#]}"; do
printf "%s\n" "$file"
done
You don't need to create an array of selected files as you can directly iterate them using a glob with brace expansion like this:
for file in build/*{-enterprise.ipa,.apk}; do
echo "$file"
# app_upload "$file"
done
I have a file that contains some keywords and I intend to create subdirectories into the same directory of the same keyword using a bash script. Here is the code I am using but it doesn't seem to be working.
I don't know where I have gone wrong. Help me out
for i in `cat file.txt`
do
# if [[ ! -e $path/$i ]]; then
echo "creating" $i "directory"
mkdir $path/$i
# fi
grep $i file >> $path/$i/output.txt
done
echo "created the files in "$path/$TEMP/output.txt
You've gone wrong here, and you've gone wrong here.
while read i
do
echo "Creating $i directory"
mkdir "$path/$i"
grep "$i" file >> "$path/$i"/output.txt
done < file.txt
echo "created the files in $path/$TEMP/output.txt"
78mkdir will refuse to create a directory, if parts of it do not exist.
e.g. if there is no /foo/bar directory, then mkdir /foo/bar/baz will fail.
you can relax this a bit by using the -p flag, which will create parent directories if necessary (in the example, it might create /foo and /foo/bar).
you should also use quotes, in case your paths contain blanks.
mkdir -p "${path}/${i}"
finally, make sure that you are actually allowed to create directories in $path