I an new to shell scripting. We have a requirement to write a shell
script for below requirement. We will get daily files in the below
format.
1. /app/dstage/BAL/Activ Bal_pen_20200129.xls
2. /app/dstage/BAL/Activ Bal_pen_20200130.xls
3. /app/dstage/BAL/Activ Bal_pen_20200131.xls We need to write a shell script as soon as the files arrvies in the > direcotry I need move
them to another staging directory. I am writing below script. However
i am getting message file not found even the file "Activ
Bal_pen_20200129.xls" present in the directory. I am seeing the
problem is with the space in the file name. How to resolve the space
issue in the file name. my Script is below:
#!/bin/ksh
# Validation the number of arguments
if (( $# == 4 )); then
Pattern=$1
numFilesExpected=$2
stgDir=$3
maxWaitTime=$4
else
print -u2 "Wrong number of arguments (${#}): Usage FileValidation.ksh <Pattern> <numFilesExpected> <stgDir> <maxWaitTime>"
return 8
fi
# While max waiting duration is not obtained find files and set the array containing the file names; closing STDERR in case no files are found
waitTime=0
while (( $waitTime < $maxWaitTime )) ; do
set -A fileList $(ls $Pattern 2<&-)
if (( ${#fileList[#]} != 0 )); then
break
fi
sleep 5
((waitTime++))
done
# Resetting the fileList in case it did not go in the loop
set -A fileList $(ls $Pattern 2<&-)
# Verifying if the pattern returned a file
if (( ${#fileList[#]} == 0 )); then
print -u2 "No file found with pattern: $Pattern"
return 1
#
elif [[ $(basename "$Pattern") = "*" ]]; then
print -u2 "Found a source file pattern with no prefix (only a path with wildcard *): $Pattern"
return 1
# Validation of the number of files expected
elif (( $numFilesExpected != 0 && ${#fileList[#]} != $numFilesExpected )); then
print -n "${#fileList[#]}"
return 2
fi
Related
I'm running this in bash and even though there is a .txt file it prints out "no new folders to create" in the terminal.
Am I missing something?
FILES=cluster-02/*/*
for f in $FILES
do
if [[ $f == *."txt" ]]
then
cat $f | xargs mkdir -p
else
echo "No new folders to create"
fi
done;
As mentioned in the first comment, the behaviour is indeed as you might expect from your script: you run through all files, text files and other ones. In case your file is a text file, you perform the if-case and in case your file is another type of file, you perform the else-case.
In order to solve this, you might decide not to take the other files into account (only handle text files), I think you might do this as follows:
FILES=cluster-02/*/*.txt
You're looping over multiple files, so the first result may trigger the if and the second can show the else.
You could save the wildcard result in an array, check if there's something in it, and loop if so:
shopt -s nullglob
FILES=( foo/* )
if (( ${#FILES[#]} )); then
for f in "${FILES[#]}"; do
if [[ $f == *."txt" ]]; then
echo $f
fi
done
else
echo "No new folders to create"
fi
#!/usr/bin/env bash
# Create an array containing a list of files
# This is safer to avoid issues with files having special characters such
# as spaces, glob-characters, or other characters that might be cumbersome
# Note: if no files are found, the array contains a single element with the
# string "cluster-02/*/*"
file_list=( cluster-02/*/* )
# loop over the content of the file list
# ensure to quote the list to avoid the same pitfalls as above
for _file in "${file_list[#]}"
do
[ "${_file%.txt}" == "${_file}" ] && continue # skip, not a txt
[ -f "${_file}" ] || continue # check if the file exists
[ -r "${_file}" ] || continue # check if the file is readable
[ -s "${_file}" ] || continue # check if the file is empty
< "${_file}" xargs mkdir -p -- # add -- to avoid issues with entries starting with -
_c=1
done;
[ "${_c}" ] || echo "No new folders to create"
I recenly moved from bash 4.2 to 5.0 and don't understand why this function don't skip when called from a while loop
alreadyInQueue ()
{
# skip files if already in queue
for task in "$HOME/.encode_queue/queue/"*
do
# Break if no task found
[[ -f "$task" ]] || break
# Sed line two from task file in queue (/dev/null error on empty queue)
line=$( sed '2q;d' "$task" 2>/dev/null )
# Store initial IFS
OLD_IFS=$IFS
# Extract tag and file from line
IFS='|' read nothing tag file <<< "$line"
# Restore IFS
IFS=$OLD_IFS
# Skip files already in queue with same preset (tag)
if [[ "$tag" == "${tag_prst[x]}" && "$file" == "$1" ]]; then
# Silent skip $2 argument: echo only if $2 = 1
[[ "$2" -eq "1" ]] && echo -e "\n** INFO ** Encode Queue, skip file already in queue:\n$i"
# Continue n
continue 2
fi
done
}
while loop calling function
# Find specified files
find "$job_path" "${args_files[#]}" | sort | while read -r i
do
# Extracts var from $i
fileSub
# Skip files already in queue
alreadyInQueue "$i" "1"
echo "i should be skipped"
done
script echo: ** INFO ** Encode Queue, skip file already in queue: ...
but doesn't continue to next file iteration
When continue is not executed inside a function call it works
# Find specified files
find "$job_path" "${args_files[#]}" | sort | while read -r i
do
# Extracts var from $i
fileSub
# Skip files already in queue
#alreadyInQueue "$i" "1"
# skip files if already in queue waiting to be encoded
for task in "$HOME/.encode_queue/queue/"*
do
# Break if no task found
[[ -f "$task" ]] || break
# Sed line two from task file in queue (/dev/null error on empty queue)
line=$( sed '2q;d' "$task" 2>/dev/null )
# Store initial IFS
OLD_IFS=$IFS
# Extract tag and file from line
IFS='|' read nothing tag file <<< "$line"
# Restore IFS
IFS=$OLD_IFS
# Skip files already in queue with same preset (tag)
if [[ "$tag" == "${tag_prst[x]}" && "$file" == "$i" ]]; then
# Silent skip $2 argument: echo only if $2 = 1
[[ "1" -eq "1" ]] && echo -e "\n** INFO ** Encode Queue, skip file already in queue:\n$i"
# Continue n
continue 2
fi
done
echo "i should be skipped"
done
help appreciated
This was a bug fix made in bash 4.4:
xx. Fixed a bug that could allow `break' or `continue' executed from shell
functions to affect loops running outside of the function.
I am working for a file validations in shell script where we need to read file size and file names from a tar file and validate the file size is greater than particular size and also the tar file contains all the list of mandatory files.
Below is the code I have written, is there any way we can check file name and file size at a single time one loop.
any inputs is appreciated. Thanks in advance...
#!/bin/bash
minimumsize=90000
mandatoryFiles=(party.dat test1.dat test2.dat)
ALL_FILE_NAMES=`tar -tvf testData/test_daily.tgz | awk '{print $6}' ` #get file names in tar file
ALL_FILE_SIZES=`tar -tvf testData/test_daily.tgz | awk '{print $3}' ` #get file sizez in bytes
echo "File names :::::::::::"$ALL_FILE_NAMES
echo "File sizes :::::::::::"$ALL_FILE_SIZES
#condition to check file size is greater than minimum size
for actualsize in $ALL_FILE_SIZES; do
if [ $actualsize -ge $minimumsize ]; then
echo size is over $minimumsize bytes
else
echo size is under $minimumsize bytes
exit 0
fi
done
#condition to check all the mandatory files are included in the taz file.
for afile in $ALL_FILE_NAMES; do
if [[ ${mandatoryFiles[*]} =~ $afile ]]; then
echo $afile is present
else
echo $afile not present so existing the bash
exit 0
fi
done
Your approach is a bit awkward. You only need to capture all filenames to loop over the mandatory files, but you cannot do your check the way you are doing or any additional files in your archive (beyond mandatory files) will cause your test to fail.
A cleaner approach is to use process substitution to feed the size and filename to a loop allowing you to test each of the file sizes (any file less than minimumsize will cause the archive to fail), while you fill your array of all_names. You are done with the read loop at that point.
A final loop over all_names checking if they exist in mandatoryFiles and incrementing a counter will allow you to check if there was a match against each of the mandatoryFiles.
One approach would be:
#!/bin/bash
fname="${1:-testData/test_daily.tgz}" ## filename to read
minimumsize=90000 ## min size
mandatoryFiles=(party.dat test1.dat test2.dat) ## mandatory files
declare -a all_names ## all_names array
declare -i mandatory_count=0; ## mandatory count
while read -r size name; do ## read/compare sizes, fill array
all_names+=( "${name##*/}" ); ## store each file name in array w/o path
#condition to check file size is greater than minimum size
if [ "$size" -ge $minimumsize ]; then
echo "$size is over $minimumsize bytes"
else
echo "$size is under $minimumsize bytes"
exit 0
fi
done < <(tar -tzvf "$fname" | awk '{print $3, $6}')
#condition to check all the mandatory files are included in the taz file.
for afile in "${all_names[#]}"; do
if [[ ${mandatoryFiles[#]} =~ "$afile" ]]; then
((mandatory_count++)) ## increment mandatory_count
fi
done
## test if mandatory_count less than number of mandatory files
if [ "$mandatory_count" -lt "${#mandatoryFiles[#]}" ]; then
echo "mandatoryFiles not present - exiting"
exit 1
fi
echo "all files good"
(note: if the file is a .tgz (g-zipped tar archive), you need to add the 'z' option as done above)
Look things over and let me know if you have further questions.
I have to write a shell script that creates a file containing the name of each text files from a folder (given as parameter) and it's subfolders that contain words longer than n characters (read n from keyboard).
I wrote the following code so far :
#!/bin/bash
Verifies if the first given parameter is a folder:
if [ ! -d $1 ]
then echo $1 is not a directory\!
exit 1
fi
Reading n
echo -n "Give the number n: "
read n
echo "You entered: $n"
Destination where to write the name of the files:
destinatie="destinatie"
the actual part that i think it makes me problems:
nr=0;
#while read line;
#do
for fisier in `find $1 -type f`
do
counter=0
for word in $(<$fisier);
do
file=`basename "$fisier"`
length=`expr length $word`
echo "$length"
if [ $length -gt $n ];
then counter=$(($counter+1))
fi
done
if [ $counter -gt $nr ];
then echo "$file" >> $destinatie
fi
done
break
done
exit
The script works but it does a few more steps that i don't need.It seems like it reads some files more than 1 time. If anyone can help me please?
Does this help?
egrep -lr "\w{$n,}" $1/* >$destinatie
Some explanation:
\w means: a character that words consist of
{$n,} means: number of consecutive characters is at least $n
Option -l lists files and does not print the grepped text and -r performs a recursive scan on your directory in $1
Edit:
a bit more complete version around the egrep command:
#!/bin/bash
die() { echo "$#" 1>&2 ; exit 1; }
[ -z "$1" ] && die "which directory to scan?"
dir="$1"
[ -d "$dir" ] || die "$dir isn't a directory"
echo -n "Give the number n: "
read n
echo "You entered: $n"
[ $n -le 0 ] && die "the number should be > 0"
destinatie="destinatie"
egrep -lr "\w{$n,}" "$dir"/* | while read f; do basename "$f"; done >$destinatie
This code has syntax errors, probably leftovers from your commented-out while loop: It would be best to remove the last 3 lines: done causes the error, break and exit are unnecessary as there is nothing to break out from and the program always terminates at its end.
The program appears to output files multiple times because you just append to $destinatie. You could simply delete that file when you start:
rm "$destinatie"
You echo the numbers to stdout (echo "$length") and the file names to $destinatie (echo "$file" >> $destinatie). I do not know if that is intentional.
I found the problem.The problem was the directory in which i was searching.Because i worked on the files from the direcotry and modified them , it seems that there remained some files which were not displayed in file explorer but the script would find them.i created another directory and i gived it as parameter and it works. Thank you for your answers
.
I built this array to list a directory's contents that would allow the user to input a number instead of the exact directory name. I do not know how I can get the picked directory's name from the input which is only a number? I need it to identify what exact directory was picked from the array that will be used. Thanks to all those willing to help.
if [ -d $TDDIR ]; then
# List folders in $TDDIR
VAR1=1
FILES=( `ls $TDDIR` )
FILECNT=${#FILES[*]}
# List number of folders available
clear
echo "Current list of folders to pick from."
echo
while [ $VAR1 -lt $FILECNT ]; do
echo -e "[$VAR1] \t ${FILES[VAR1]}"
let "VAR1 += 1"
done
# Reduce VAR1 by one
let "VAR1 -= 1"
# Have user choose which to use for scan
read -p "Which file would you like to scan 1 - $VAR1 or 0 to exit? :" INPUT
if [ $INPUT -eq '0' ]; then
echo "You chose to quit. Exiting now."
sleep 2
exit 0
fi
while (( $INPUT > $FILECNT || $INPUT < 1 || $INPUT > $VAR1 )); do
echo
echo "You selected $INPUT."
read -p "$INPUT is not valid. Choose a valid option: " INPUT
done
else exit
fi
echo ${FILES[(($VAR - 1))]}
The -1 is because the index for bash arrays starts from 0.
A side note, ls is not appropriate for the task (see: Parsing ls outputs)
Also, you may try using select for the menu (see: Bash Script Menus and Functions)