I recenly moved from bash 4.2 to 5.0 and don't understand why this function don't skip when called from a while loop
alreadyInQueue ()
{
# skip files if already in queue
for task in "$HOME/.encode_queue/queue/"*
do
# Break if no task found
[[ -f "$task" ]] || break
# Sed line two from task file in queue (/dev/null error on empty queue)
line=$( sed '2q;d' "$task" 2>/dev/null )
# Store initial IFS
OLD_IFS=$IFS
# Extract tag and file from line
IFS='|' read nothing tag file <<< "$line"
# Restore IFS
IFS=$OLD_IFS
# Skip files already in queue with same preset (tag)
if [[ "$tag" == "${tag_prst[x]}" && "$file" == "$1" ]]; then
# Silent skip $2 argument: echo only if $2 = 1
[[ "$2" -eq "1" ]] && echo -e "\n** INFO ** Encode Queue, skip file already in queue:\n$i"
# Continue n
continue 2
fi
done
}
while loop calling function
# Find specified files
find "$job_path" "${args_files[#]}" | sort | while read -r i
do
# Extracts var from $i
fileSub
# Skip files already in queue
alreadyInQueue "$i" "1"
echo "i should be skipped"
done
script echo: ** INFO ** Encode Queue, skip file already in queue: ...
but doesn't continue to next file iteration
When continue is not executed inside a function call it works
# Find specified files
find "$job_path" "${args_files[#]}" | sort | while read -r i
do
# Extracts var from $i
fileSub
# Skip files already in queue
#alreadyInQueue "$i" "1"
# skip files if already in queue waiting to be encoded
for task in "$HOME/.encode_queue/queue/"*
do
# Break if no task found
[[ -f "$task" ]] || break
# Sed line two from task file in queue (/dev/null error on empty queue)
line=$( sed '2q;d' "$task" 2>/dev/null )
# Store initial IFS
OLD_IFS=$IFS
# Extract tag and file from line
IFS='|' read nothing tag file <<< "$line"
# Restore IFS
IFS=$OLD_IFS
# Skip files already in queue with same preset (tag)
if [[ "$tag" == "${tag_prst[x]}" && "$file" == "$i" ]]; then
# Silent skip $2 argument: echo only if $2 = 1
[[ "1" -eq "1" ]] && echo -e "\n** INFO ** Encode Queue, skip file already in queue:\n$i"
# Continue n
continue 2
fi
done
echo "i should be skipped"
done
help appreciated
This was a bug fix made in bash 4.4:
xx. Fixed a bug that could allow `break' or `continue' executed from shell
functions to affect loops running outside of the function.
Related
i'm trying to generate a new output file from each existing file in a directory of .txt files. I want to check line by line in each file for two substrings. And append the lines that match that substring to each new output file.
I'm having trouble generating the new files.
This is what i currently have:
#!/bin/sh
# My first Script
success="(Compiling)\s\".*\"\s\-\s(Succeeded)"
failure="(Compiling)\s\".*\"\s\-\s(Failed)"
count_success=0
count_failure=0
for i in ~/Documents/reports/*;
do
while read -r line;
do
if [[$success=~$line]]; then
echo $line >> output_$i
count_success++
elif [[$failure=~$]]; then
echo $line >> output_$i
count_failure++
fi
done
done
echo "$count_success of jobs ran succesfully"
echo "$count_failure of jobs didn't work"
~
Any help would be appreciated, thanks
Please, use https://www.shellcheck.net/ to check your shell scripts.
If you use Visual Studio Code, you could install "ShellCheck" (by Timon Wong) extension.
About your porgram.
Assume bash
Define different extensions for input and output files (really important if there are in the same directory)
Loop on report, input, files only
Clear output file
Read input file
if sequence:
if [[ ... ]] with space after [[ and before ]]
spaces before and after operators (=~)
reverse operands order for operators =~
Prevent globbing with "..."
#! /bin/bash
# Input file extension
declare -r EXT_REPORT=".txt"
# Output file extension
declare -r EXT_OUTPUT=".output"
# RE
declare -r success="(Compiling)\s\".*\"\s\-\s(Succeeded)"
declare -r failure="(Compiling)\s\".*\"\s\-\s(Failed)"
# Counters
declare -i count_success=0
declare -i count_failure=0
for REPORT_FILE in ~/Documents/reports/*"${EXT_REPORT}"; do
# Clear output file
: > "${REPORT_FILE}${EXT_OUTPUT}"
# Read input file (see named file in "done" line)
while read -r line; do
# does the line match the success pattern ?
if [[ $line =~ $success ]]; then
echo "$line" >> "${REPORT_FILE}${EXT_OUTPUT}"
count_success+=1
# does the line match the failure pattern ?
elif [[ $line =~ $failure ]]; then
echo "$line" >> "${REPORT_FILE}${EXT_OUTPUT}"
count_failure+=1
fi
done < "$REPORT_FILE"
done
echo "$count_success of jobs ran succesfully"
echo "$count_failure of jobs didn't work"
What about using grep?
success='Compiling\s".*"\s-\sSucceeded'
failure='Compiling\s".*"\s-\sFailed'
count_success=0
count_failure=0
for i in ~/Documents/reports/*; do
(( count_success += $(grep -E "$success" "$i" | tee "output_$i" | wc -l) ))
(( count_failure += $(grep -E "$failure" "$i" | tee -a "output_$i" | wc -l) ))
done
echo "$count_success of jobs ran succesfully"
echo "$count_failure of jobs didn't work"
I am trying to automate the below: Any help, please.
We have 2 directories as mentioned below, whenever we get new files in Directory-1, only they should be copied and replaced into Directory-2. How to achieve this in Linux scripting. Filename remains the same but the version will be different.
Directory-1:
FileOne_2.0.0.txt
FileTwo_3.0.0.txt
Directory-2:
FileOne_1.0.0.txt
FileTwo_2.0.0.txt
FileThree_3.0.0.txt
FileFive_5.0.0.txt
Try this code (on a test setup before you trust your real directories and files with it):
#! /bin/bash -p
shopt -s extglob # Enable extended globbing ( +([0-9]) ... )
shopt -s nullglob # Globs that match nothing expand to nothing
shopt -s dotglob # Globs match files with names starting with '.'
srcdir='Directory-1'
destdir='Directory-2'
# A(n extended) glob pattern to match a version string (e.g. '543.21.0')
readonly kVERGLOB='+([0-9]).+([0-9]).+([0-9])'
# shellcheck disable=SC2231 # (Bad warning re. unquoted ${kVERGLOB})
for srcpath in "$srcdir"/*_${kVERGLOB}.txt; do
srcfile=${srcpath##*/} # E.g. 'FileOne_2.0.0.txt'
srcbase=${srcfile%_*} # E.g. 'FileOne'
# Set and check the path that the file will be moved to
destpath=$destdir/$srcfile
if [[ -e $destpath ]]; then
printf "Warning: '%s' already exists. Skipping '%s'.\\n" \
"$destpath" "$srcpath" >&2
continue
fi
# Make a list of the old versions of the file
# shellcheck disable=SC2206 # (Bad warning re. unquoted ${kVERGLOB})
old_destpaths=( "$destdir/$srcbase"_${kVERGLOB}.txt )
# TODO: Add checks that the number of old files (${#old_destpaths[*]})
# is what is expected (exactly one?)
# Move the file
if mv -i -- "$srcpath" "$destpath"; then
printf "Moved '%s' to '%s'\\n" "$srcpath" "$destpath" >&2
else
printf "Warning: Failed to move '%s' to '%s'. Skipping '%s'.\\n" \
"$srcpath" "$destpath" "$srcpath" >&2
continue
fi
# Remove the old version(s) of the file (if any)
for oldpath in "${old_destpaths[#]}"; do
if rm -- "$oldpath"; then
printf "Removed '%s'\\n" "$oldpath" >&2
else
printf "Warning: Failed to remove '%s'.\\n" "$oldpath" >&2
fi
done
done
The code is Shellcheck-clean. Two Shellcheck suppression comments are used because the unquoted expansions are necessary here.
srcdir and destdir are set to constant values. You might want to take them from command line parameters, or set them to different constant values.
The code could be made shorter by removing checks. However, moves and removes are destructive operations that can do a lot of damage if they are done incorrectly. I'd add even more checks if it was my own data.
See glob - Greg's Wiki for an explanation of the "extended globbing" used in the code.
See Parameter expansion [Bash Hackers Wiki] for an explanation of ${srcpath##*/} and ${srcfile%_*}.
mv -i is used as a double protection against overwriting an existing file.
All external commands are invoked with -- to explicitly end options, in case they are ever used with paths that begin with -.
Make sure that you understand the code and test it VERY carefully before using it for real.
source_dir=./files/0
dest_dir=./files/1/
for file in $source_dir/*
do
echo $file
echo "processing"
if [[ "1" == "1" ]]; then
mv $file $dest_dir
fi
done
Where processing and the 1 == 1 is whatever your 'prechecks' are (which you haven't told us)
If your coreutils sort is newer than or equal to v7.0 (2008-10-5) after which sort command
supports -V option (version-sort), would you please try:
declare -A base2ver base2file
# compare versions
# returns 0 if $1 equals to $2
# 1 if $1 is newer than $2
# -1 if $1 is older than $2
vercomp() {
if [[ $1 = $2 ]]; then
echo 0
else
newer=$(echo -e "$1\n$2" | sort -Vr | head -n 1)
if [[ $newer = $1 ]]; then
echo 1
else
echo -1
fi
fi
}
for f in Directory-1/*.txt; do
basename=${f##*/}
version=${basename##*_}
version=${version%.txt} # version number such as "2.0.0"
basename=${basename%_*} # basename such as "FileOne"
base2ver[$basename]=$version # associates basename with version number
base2file[$basename]=$f # associates basename with full filename
done
for f in Directory-2/*.txt; do
basename=${f##*/}
version=${basename##*_}
version=${version%.txt}
basename=${basename%_*}
if [[ -n ${base2ver[$basename]} ]] && (( $(vercomp "${base2ver[$basename]}" "$version") > 0 )); then
# echo "${base2file[$basename]} is newer than $f"
rm -- "$f"
cp -p -- "${base2file[$basename]}" Directory-2
fi
done
I an new to shell scripting. We have a requirement to write a shell
script for below requirement. We will get daily files in the below
format.
1. /app/dstage/BAL/Activ Bal_pen_20200129.xls
2. /app/dstage/BAL/Activ Bal_pen_20200130.xls
3. /app/dstage/BAL/Activ Bal_pen_20200131.xls We need to write a shell script as soon as the files arrvies in the > direcotry I need move
them to another staging directory. I am writing below script. However
i am getting message file not found even the file "Activ
Bal_pen_20200129.xls" present in the directory. I am seeing the
problem is with the space in the file name. How to resolve the space
issue in the file name. my Script is below:
#!/bin/ksh
# Validation the number of arguments
if (( $# == 4 )); then
Pattern=$1
numFilesExpected=$2
stgDir=$3
maxWaitTime=$4
else
print -u2 "Wrong number of arguments (${#}): Usage FileValidation.ksh <Pattern> <numFilesExpected> <stgDir> <maxWaitTime>"
return 8
fi
# While max waiting duration is not obtained find files and set the array containing the file names; closing STDERR in case no files are found
waitTime=0
while (( $waitTime < $maxWaitTime )) ; do
set -A fileList $(ls $Pattern 2<&-)
if (( ${#fileList[#]} != 0 )); then
break
fi
sleep 5
((waitTime++))
done
# Resetting the fileList in case it did not go in the loop
set -A fileList $(ls $Pattern 2<&-)
# Verifying if the pattern returned a file
if (( ${#fileList[#]} == 0 )); then
print -u2 "No file found with pattern: $Pattern"
return 1
#
elif [[ $(basename "$Pattern") = "*" ]]; then
print -u2 "Found a source file pattern with no prefix (only a path with wildcard *): $Pattern"
return 1
# Validation of the number of files expected
elif (( $numFilesExpected != 0 && ${#fileList[#]} != $numFilesExpected )); then
print -n "${#fileList[#]}"
return 2
fi
I have a function with a parameter file. And I want to read it line by line.
Condition
If the lines are between <?bash and ?> then I do bash -c '$line' else I display the line.
Here my file (file):
<html><head></head><body><p>Hello
<?bash
echo "world !"
?>
</p></body></html>
Here my Bash script (bashtml):
#!/bin/bash
function generation()
{
while read line
do
if [ $line = '<?bash' ]
then
while [ $line != '?>' ]
do
bash -c '$line'
done
else
echo $line
fi
done
}
generation $file
I execute this script:
./bashhtml
I am novice in Bash script and I'm lost
I think this is what you mean. However, this code is HIGHLY DANGEROUS! Any command inserted into those bash tags would be executed under your user id. It could change your password, delete all your files, read or alter data, and so on. Don't do it!
#!/bin/bash
function generation
{
# If you don't use local (or declare) then variables are global
local file="$1" # Parameter passed to function, in a local variable
local start=False # A flag to indicate tags
local line
while read -r line
do
if [[ $line == '<?bash' ]]
then
start=True
elif [[ $line == '?>' ]]
then
start=False
elif "$start"
then
bash -c "$line" # Double quotes needed here
else
echo "$line"
fi
done < "$file" # Notice how the filename is redirected into read
}
infile="$1" # This gets the filename from the command-line
generation "$infile" # This calls the function
I need to validate my log files:
-All new log lines shall start with date.
-This date will respect the ISO 8601 standard. Example:
2011-02-03 12:51:45,220Z -
Using shell script, I can validate it looping on each line and verifying the date pattern.
The code is below:
#!/bin/bash
processLine(){
# get all args
line="$#"
result=`echo $line | egrep "[0-9]{4}-[0-9]{2}-[0-9]{2} [012][0-9]:[0-9]{2}:[0-9]{2},[0-9]{3}Z" -a -c`
if [ "$result" == "0" ]; then
echo "The log is not with correct date format: "
echo $line
exit 1
fi
}
# Make sure we get file name as command line argument
if [ "$1" == "" ]; then
echo "You must enter a logfile"
exit 0
else
file="$1"
# make sure file exist and readable
if [ ! -f $file ]; then
echo "$file : does not exists"
exit 1
elif [ ! -r $file ]; then
echo "$file: can not read"
exit 2
fi
fi
# Set loop separator to end of line
BAKIFS=$IFS
IFS=$(echo -en "\n\b")
exec 3<&0
exec 0<"$file"
while read -r line
do
# use $line variable to process line in processLine() function
processLine $line
done
exec 0<&3
# restore $IFS which was used to determine what the field separators are
IFS=$BAKIFS
echo SUCCESS
But, there is a problem. Some logs contains stacktraces or something that uses more than one line, in other words, stacktrace is an example, it can be anything. Stacktrace example:
2011-02-03 12:51:45,220Z [ERROR] - File not found
java.io.FileNotFoundException: fred.txt
at java.io.FileInputStream.<init>(FileInputStream.java)
at java.io.FileInputStream.<init>(FileInputStream.java)
at ExTest.readMyFile(ExTest.java:19)
at ExTest.main(ExTest.java:7)
...
will not pass with my script, but is valid!
Then, if I run my script passing a log file with stacktraces for example, my script will failed, because it loops line by line.
I have the correct pattern and I need to validade the logger date format, but I don't have wrong date format pattern to skip lines.
I don't know how I can solve this problem. Does somebody can help me?
Thanks
You need to anchor your search for the date to the start of the line (otherwise the date could appear anywhere in the line - not just at the beginning).
The following snippet will loop over all lines that do not begin with a valid date. You still have to determine if the lines constitute errors or not.
DATEFMT='^[0-9]{4}-[0-9]{2}-[0-9]{2} [012][0-9]:[0-9]{2}:[0-9]{2},[0-9]{3}Z'
egrep -v ${DATEFMT} /path/to/log | while read LINE; do
echo ${LINE} # did not begin with date.
done
So just (silently) discard a single stack trace. In somewhat verbose bash:
STATE=idle
while read -r line; do
case $STATE in
idle)
if [[ $line =~ ^java\..*Exception ]]; then
STATE=readingexception
else
processLine "$line"
fi
;;
readingexception)
if ! [[ $line =~ ^' '*'at ' ]]; then
STATE=idle
processLine "$line"
fi
;;
*)
echo "Urk! internal error [$STATE]" >&2
exit 1
;;
esac
done <logfile
This relies on processLine not continuing on error, else you will need to track a tad more state to avoid two consecutive stack traces.
This makes 2 assumptions.
lines that begin with whitespace are continuations of previous lines. we're matching a leading space, or a leading tab.
lines that have non-whitespace characters starting at ^ are new log lines.
If a line matching #2 doesn't match the date format, we have an error, so print the error, and include the line number.
count=0
processLine() {
count=$(( count + 1 ))
line="$#"
result=$( echo $line | egrep '^[0-9]{4}-[0-9]{2}-[0-9]{2} [012][0-9]:[0-9]{2}:[0-9]{2},[0-9]{3}Z' -a -c )
if (( $result == 0 )); then
# if result = 0, then my line did not start with the proper date.
# if the line starts with whitespace, then it may be a continuation
# of a multi-line log entry (like a java stacktrace)
continues=$( echo $line | egrep "^ |^ " -a -c )
if (( $continues == 0 )); then
# if we got here, then the line did not start with a proper date,
# AND the line did not start with white space. This is a bad line.
echo "The line is not with correct date format: "
echo "$count: $line"
exit 1
fi
fi
}
Create a condition to check if the line starts with a date. If not, skip that line as it is part of a multi-line log.
processLine(){
# get all args
line="$#"
result=`echo $line | egrep "[0-9]{4}-[0-9]{2}-[0-9]{2} [012][0-9]:[0-9]{2}:[0-9]{2},[0-9]{3}Z" -a -c`
if [ "$result" == "0" ]; then
echo "Log entry is multi-lined - continuing."
fi
}