Anyone able to help me out? I have a shell script I am working on but for the loop below the command after "echo "first file is $firstbd" is not being executed.. the $PROBIN/proutil ?? Not sure why this is...
Basically I have a list of files in a directory (*.list), I grab them and read the first line and pass it as a parameter to the cmdlet then move the .list and the content of the .list to another directory (the .list has a list of files with full path).
for i in $(ls $STAGEDIR/*.list); do
echo "Working with $i"
# grab first .bd file
firstbd=`head -1 $i`
echo "First file is $firstbd"
$PROBIN/proutil $DBENV/$DBNAME -C load $firstbd tenant $TENANT -dumplist $STAGEDIR/$i.list >> $WRKDIR/$i.load.log
#move the list and its content to finished folder
binlist=`cat $i`
for movethis in $binlist; do
echo "Moving file $movethis to $STAGEDIR/finished"
mv $movethis $STAGEDIR/finished/
done
echo "Finished working with list $i"
echo "Moving it to $STAGEDIR/finished"
mv $i $STAGEDIR/finished/
done
The error I was getting is..
./tableload.sh: line 107: /usr4/dlc/bin/proutil /usr4/testdbs/xxxx2 -C load /usr4/dumpdir/xxxxx.bd tenant xxxxx -dumplist /usr4/dumpdir/PUB.xxxxx.list >> /usr4/dumpdir/PUB.xxxx.list.load.log: A file or directory in the path name does not exist... however if I run "/usr4/dlc/bin/proutil"
The fix was to remove ">> $WRKDIR/$i.load.log".. the binary utility wouldn't run when trying to output results to file.. strange..
A couple of really bad practices here
parse the output of ls
not quoting variables
iterating the lines of a file with cat and for
As shelter comments, you don't check that you've created all the directories in the path for your log file.
A rewrite:
for i in "$STAGEDIR"/*.list; do
echo "Working with $i"
# grab first .bd file
firstbd=$(head -1 "$i")
echo "First file is $firstbd"
# ensure the output directory exists
logfile="$WRKDIR/$i.load.log"
mkdir -p "$(dirname "$logfile")"
"$PROBIN"/proutil "$DBENV/$DBNAME" -C load "$firstbd" tenant "$TENANT" -dumplist "$STAGEDIR/$i.list" >> "$logfile"
# move the list and its content to finished folder
while IFS= read -r movethis; do
echo "Moving file $movethis to $STAGEDIR/finished"
mv "$movethis" "$STAGEDIR"/finished/
done < "$i"
echo "Finished working with list $i"
echo "Moving it to $STAGEDIR/finished"
mv "$i" "$STAGEDIR"/finished/
done
Related
I'm trying to automate the packaging of files and contents from various sources using a bash script.
I have a main directory which contains pdf files, a csv file, and various folders with additional contents. The folders are named with the location code they pertain to, e.g. 190, 191, etc.
A typical row in my csv file looks like this: form_letters_Part1.pdf,PX_A31_smith.adam.pdf,190,
Where the first column is the original pdf name, the second is what it will be renamed to, and the third column is the location code the person belongs to.
The first part of my script renames the pdf files from the cover letters format to the PX_A31... format, and then creates a directory for each file and moves them into it.
#!/usr/bin/tcsh bash
sed 's/"//g' rename_list_lab.csv | while IFS=, read orig new num; do
mv "$orig" "$new"
done
echo 'Rename Done.'
for file in *.pdf; do
mkdir "${file%.*}"
mv "$file" "${file%.*}"
done
echo 'Directory creation done.'
What needs to happen next is the folders with the location-specific contents get copied into those new directories just created, corresponding to the location code from the csv file.
So I tried this after the above echo 'Directory Creation Done.' line:
echo 'Directory Creation Done.'
sed 's/"//g' rename_list.csv | while IFS=, read orig new num; do
for folder in *; do
if [[ -d .* = "$num" ]]; then
cp -R "$folder" "${file%.*}"
fi
done
echo 'Code Folder Contents Sort Done.'
However this results in a syntax error:
syntax error in conditional expression
syntax error near `='
` if [[ -d .* = "$num" ]]; then'
EDIT: To clarify the second part if statement, the intended logic of the statement is as follows: For the items in the current directory, if it is a directory, and the name of the directory matches the location code from the csv, that directory should be copied to any directories which have that same corresponding location code in the csv.
In other words, if the newly created directory from the first part is PX_A31_smith.adam whose location code in the csv line above is 190, then the folder called 190 should be copied into the directory PX_A31_smith.adam.
If three other people also have the 190 code in the csv, the 190 directory should also be copied to those as well.
EDIT 2: I resolved the syntax error, and also realized I had an nonterminated do statement. Fixing those, still seem to be having trouble with the evaluation of the if statement. Updated script below:
#!/usr/bin/tcsh bash
sed 's/"//g' rename_list.csv | while IFS=, read orig new num; do
mv "$orig" "$new"
done
echo '1 Done.'
for file in *.pdf; do
mkdir "${file%.*}"
mv "$file" "${file%.*}"
done
echo '2 done.'
sed 's/"//g' rename_list.csv | while IFS=, read orig new num; do
for folder in * ; do
if [[ .* = "$num" ]]; then
cp -R "$folder" "${file%.*}"
else echo "No matches found."
fi
done
done
echo '3 Done.'
I'm not really sure if this answers your question, but I think it will at least set you on the right track. Structurally, I just combined all of the loops into one. This removes some of the possible logic errors that would not be considered syntax errors like the use of $file in the second part. This is a local variable to the loop in the first part and no longer exists. However, this would be interpreted as an empty string.
#!/usr/bin/bash
#^Fixed shebang line.
sed 's/"//g' rename_list.csv | while IFS=, read -r orig new num; do
if [[ -f $orig ]]; then #If the file we want to rename is indeed a file.
mkdir "${new%.*}" #make the directory from the file name you want
mv "$orig" "${new%.*}/$new" #Rename when we move the file into the new directory
if [[ -d $num ]]; then #If the number directory exists
cp -R "$num" "${new%.*}" #Fixed this based on your edit.
else
#Here you can handle what to do if the number directory does not exist.
echo "$num is not a directory."
fi
else
#Here you can handle what to do if the file does not exist.
echo "The file $orig does not exist."
fi
done
Edited based on your clarification
Note: This is pretty lacking as far as error checking goes. Remember, any of these functions could fail, which will have unwanted behavior. Either check if [[ $? != 0 ]] to check the exit status (0 being success) of the last issued command. You could also do something like mkdir somedir || exit 2 to exit on failure.
I am relatively new to shell scripting. I am writing a script to compress all the files in current and target directory. I have found success in compressing the files of a current directory but I'm unable to write a script for compressing files in a target directory can anyone guide me?
I want to do something like this
% myCompress -t /home/users/bigFoot/ pdf ppt jpg
next time try to spread your code (it will make it easier to answer):
#!/bin/bash
if [[ $# == 0 ]]; then
echo "This shell script compress files with a specific extensions"
echo "Call Syntax: compress <extension_list>"
exit
fi
for ext in $*; do
for file in ls *.$ext; do
gzip -k $file
done
done
Mistakes made
1) $* - all args coming after command - so.... -t and path are not $ext variables
2) ls *.$ext is red in loop as 2 strings "ls and *.$ext" should be written as $(ls *.$ext) to get ls command executed
My script for your request
#!/bin/bash
script_name=`basename "$0"`
if [[ $# == 0 ]]; then
echo "This shell script compress files with a specific extensions"
echo "Call Syntax: $script_name <dirctories_list> <extension_list>"
exit
fi
# check if $1 is a directory
path=". "
file_type=""
for check_type in $* ; do
if [[ -d $check_type ]]; then
path=$path$check_type" "
else
file_type=$file_type"*."$check_type" "
fi
done
echo paths to gzip $path
echo files type to check "$file_type"
for x in $path; do
cd $x
for file in $(ls $file_type); do
gzip $file
done
cd -
done
Explanation
1) basename "$0" - get scripts name - it is more generic for usage - in case you change script's name
2) path=". " - variable hold a string of all directories to be compressed, your request is to run it also on current directory ". "
file_type="" - variable hold a string of all extensions to be compressed in $path string
3) running a loop on all input ARGS and concatenate directories names to $path string and other file types to $file_type
4) for each of the directories inserted to script:
i. cd $x - enter directorie
ii. gzip - compress all files with inserted extensions
iii. cd - - go back to base directories
Check gzip
I'm not familiar with the gzip command , check that you have -k flag
Unfortunately my knowledge in bash not so well and I have very non-standard task.
I have a file with the files list.
Example: /tmp/my/file1.txt /tmp/my/file2.txt
How can I write a script which can check that files from folder /tmp/my exist and to have two types messages after script is done.
1 - Files exist and show files:
/tmp/my/file1.txt
/tmp/my/file2.txt
2 - The folder /tmp/my including files and folders which are not in your list. The files and folders:
/tmp/my/test
/tmp/my/1.txt
You speak of files and folders, which seems unclear.
Anyways, I wanted to try it with arrays, so here we go :
unset valid_paths; declare -a valid_paths
unset invalid_paths; declare -a invalid_paths
while read -r line
do
if [ -e "$line" ]
then
valid_paths=("${valid_paths[#]}" "$line")
else
invalid_paths=("${invalid_paths[#]}" "$line")
fi
done < files.txt
echo "VALID PATHS:"; echo "${valid_paths[#]}"
echo "INVALID PATHS:"; echo "${invalid_paths[#]}"
You can check for the files' existence (assuming a list of files, one filename per line) and print the existing ones with a prefix using this
# Part 1 - check list contents for files
while read thefile; do
if [[ -n "$thefile" ]] && [[ -f "/tmp/my/$thefile" ]]; then
echo "Y: $thefile"
else
echo "N: $thefile"
fi
done < filelist.txt | sort
# Part 2 - check existing files against list
for filepath in /tmp/my/* ; do
filename="$(basename "$filepath")"
grep "$filename" filelist.txt -q || echo "U: $filename"
done
The files that exist are prefixed here with Y:, all others are prefixed with N:
In the second section, files in the tmp directory that are not in the file list are labelled with U: (unaccounted for/unexpected)
You can swap the -f test which checks that a path exists and is a regular file for -d (exists and is a directory) or -e (exists)
See
man test
for more options.
I have this snippet:
#!/bin/bash
parent=/parent
newfolder=/newfolder
mkdir "$newfolder"
for folder in "$parent"/*; do
if [[ -d $folder ]]; then
foldername="${folder##*/}"
for file in "$parent"/"$foldername"/*; do
filename="${file##*/}"
newfilename="$foldername"_"$filename"
cp "$file" "$newfolder"/"$newfilename"
done
fi
done
I do need to turn it around in a way that the copied files would be named after the folder they are being moved to (e.g. moving to the /root/Case22 files would be renamed to case22_1.jpg, case22_2.docx, case22_3.JPG etc). The files would be copied from USB and both destination and source directries would be entered by the user. I have written everything else and it works apart from actual renaming and thought I could adapt this snippet.
thanks
p.s. the snippet is written by Jahid and found on stackoverflow
you can try something like this;
#!/bin/bash
parent=/root
a=1
for file in $parent/Case22*; do
filename="${file%.*}"
extension="${file##*.}"
newfilename=$(printf "$filename"_"$a"."$extension")
mv -- "$file" "$newfilename"
let a=a+1
done
Thanks for the help. I have found the solution and thought I might post it here in case someone else will be looking at this.
As the title suggests I needed a Linux shell script to copy and rename multiple files keeping original directory tree (the file source and archive locations would be specified by the user of the script). Here is the code that I came up with after few days research of different sources (it includes a trap so only one instance of script would be running at a time):
lockdir=/var/tmp/mylock #directory for lock file creation
pidfile=/var/tmp/mylock/pid #directory to get the process ID number
if ( mkdir ${lockdir} ) 2> /dev/null; then #main argument to create lock file
echo $$ > $pidfile #if successful script will proceed, otherwise it will skip to the else part of the statement at the end of the script
trap 'rm -rf "$lockdir"; exit $?' INT TERM EXIT #trap to capture reasons of script termination and removal of the lock file so it could be launched again
#start of the main script body, part of successful "if" statement
# read entry_for_testing_only #request for user entry to keep script running and try to run another script instance
findir="$2/$(basename "$1")" #variable that defines final directory to which files from USB will be copied
if [ ! -d "$1" ]; then #testing if first directory entry is a valid directory’’
echo "$1" "is not a directory"
echo ""
exit
else
if [ ! -d "$2" ]; then #testing if second entry is a valid directory
echo "archive directory non existant"
exit
else
if [ -d "$findir" ] && [ "$(ls -A "$findir")" ]; then #testing if second entry directory contains the same name folders and if the folders are empty - to avoid file overwriting
echo "such folder already there and it's not empty"
exit
else
if [ ! -d "$findir" ] ; then #last archive directory argument to create final archive directory
mkdir "$findir"
else true
fi
fi
fi
fi
rsync -a "$1"/ "$findir" #command to copy all files from the source to the archive retaining the directory tree
moved_files="$(find "$findir" -type f)" #variable that finds all files that have been copied to the archive directory
for file in $moved_files; do #start of the loop that renames copied files
counter="$((counter+1))" #incrementation variable
source_name="$(basename "$1")" #variable that captures the name of the source directory
new_name="$source_name"_"$counter" #variable that puts start of the file name and incrementation element together
if echo "$file" | grep "\." #argument that captures the extension of the file
then
extension="$(echo "$file" | cut -f2 -d. )"
else
extension=
fi
full_name="$new_name"."$extension" #variable that defines the final new name of the file
dir="$(dirname "${file}")" #variable that captures the directorry address of currently looped file
mv "$file" "$dir/$full_name" #move command to rename currently looped file with the final new name
done
#end of the main script body, unsuccessful "if" statement continues here
else
echo "Another instance of this script is already running. PID: $(cat $pidfile)"
fi
I have a file that contains some keywords and I intend to create subdirectories into the same directory of the same keyword using a bash script. Here is the code I am using but it doesn't seem to be working.
I don't know where I have gone wrong. Help me out
for i in `cat file.txt`
do
# if [[ ! -e $path/$i ]]; then
echo "creating" $i "directory"
mkdir $path/$i
# fi
grep $i file >> $path/$i/output.txt
done
echo "created the files in "$path/$TEMP/output.txt
You've gone wrong here, and you've gone wrong here.
while read i
do
echo "Creating $i directory"
mkdir "$path/$i"
grep "$i" file >> "$path/$i"/output.txt
done < file.txt
echo "created the files in $path/$TEMP/output.txt"
78mkdir will refuse to create a directory, if parts of it do not exist.
e.g. if there is no /foo/bar directory, then mkdir /foo/bar/baz will fail.
you can relax this a bit by using the -p flag, which will create parent directories if necessary (in the example, it might create /foo and /foo/bar).
you should also use quotes, in case your paths contain blanks.
mkdir -p "${path}/${i}"
finally, make sure that you are actually allowed to create directories in $path