How come the additional 'Line' insideecho "Line $line" is not prepended to all files inside the for loop?
#!/bin/bash
INPUT=targets.csv
IFS=","
[ ! -f $INPUT ] && { echo "$INPUT file not found"; exit 99; }
while read target user password path
do
result=$(sshpass -p "$password" ssh -n "$user"#"$target" ls "$path"*file* 2>/dev/null)
if [ $? -ne 0 ]
then
echo "No Heap dumps detected."
else
echo "Found a Heap dump! Possible OOM issue detected"
for line in $result
do
echo "Line $line"
done
fi
done < $INPUT
.csv file contents ..
rob#laptop:~/scripts$ cat targets.csv
server.com,root,passw0rd,/root/
script output ..
rob#laptop:~/scripts$ ./checkForHeapdump.sh
Found a Heap dump! Possible OOM issue detected
Line file1.txt
file2.txt
The statement:
for line in $result
performs word splitting on $result to get each element that $line should be set to. Word splitting uses the delimiters in $IFS. Earlier in the script you set this to just ,. So this loop will iterate over comma-separated data in $result. Since there aren't any commas in it, it's just a single element.
If you want to split it by lines, do:
IFS="
"
for line in $result
Related
I have file which does not have any data in it
Need to check below scenario and return file is empty otherwise not empty
if file contains no data but as only spaces return it as FILE is EMPTY
if file contains no data but as only tabs return it as FILE is EMPTY
if file contains no data but as only empty new line return it as FILE is EMPTY
Does this below code will satisfy all my above cases ? or any best approach all in one go
if [ -s /d/dem.txt ]
then
echo "FILE IS NOT EMPTY AS SOME DATA"
else
echo "FILE IS EMPTY NOT DATA AVAILABLE"
fi
You may use this awk for this:
awk 'NF {exit 1}' file && echo "empty" || echo "not empty"
Condition NF will be true only if there is non-whitespace character in the file.
Your description is a bit unclear (what do you want to do with a file that contains spaces, tabs, and newlines?), but it sounds like you just want to know if the file contains any non-whitespace characters. So:
if grep -q '[^[:space:]]' "$file"; then
printf "%s\n" "$file is not empty";
else
printf "%s\n" "$file contains only whitespace"
fi
If you had run your code you would have realized that no, -s considers that files with spaces, tabs and/or new lines are not empty. I would do it like this:
myfile="some_file.txt"
T=$(sed -e 's/\s//g' "$i")
if [ -n "$T" ]; then
echo "$i is NOT empty"
else
echo "$i is empty"
fi
I have a script with a loop over some directories and in each of them it executes a program.
folders=( "1" "2" )
for i in "${folders[#]}"
do
cd $i
output=$(program)
while read -r line; do
match "$line"
done <<< "$output"
some code here
cd ..
done
Now i want the script to stop the running program if $line matches with a given string and then start working on the next element of ${folders[#]}. Basically Strg+c from inside the script.
Edit: I cannot access the program and make it stop itself should the string appear.
Thanks
Now i want the script to stop the running program if $line matches
with a given string
if [ "$line" = "Put some similar text in here" ]
then
exit 0
fi
This will stop the programm, like you wanted.
then start working on the next element of ${folders[#]}
This is something different.
You can try to switch the code like this ...
folders=( "1" "2" )
for i in "${folders[#]}"
do
cd $i
output=$(program)
while read -r line; do
if [ "$line" = "Put some similar text in here" ]
then
break
fi
done <<< "$output"
# some commands ...
done
The if condition checks for similar text in a string and the break command will close the while loop.
Addition
The same code without using $output as temporary storage...
folders=( "1" "2" )
for i in "${folders[#]}"
do
cd $i
while read -r line; do
if [ "$line" = "Put some similar text in here" ]
then
break
fi
done <<< "$(program)"
# some commands ...
done
This way you will exit the extern programm in the loop.
This seems simple, list the directory from the first field then list the directory from the second field. The fields from the input file are comma separated, e.g.: XXXX1111111111112222,cool.com.
I run the command:
./list_directories some_file.csv
The list_directories script is this:
#!/bin/bash
INPUT=$1
OLDIFS=$IFS
IFS=,
[ ! -f $INPUT ] && { echo "$INPUT file not found"; exit 99; }
while read WORKING STORE
do
echo $STORE
ls $STORE
echo $WORKING
ls $WORKING
done < $INPUT
IFS=$OLDIFS
Here's the output:
/pathtothe/som/coolplace/Imlookingfor/cool.com/place/123/XXXX1111111111112222 : No such file or directorye/Imlookingfor/cool.com/place/123/XXXX1111111111112222 /pathtothe/som/coolplace/Imlookingfor/cool.com/placing/123/XXXX1111111111112222 fileindir.txt otherfileindir.txt lastofthefilesindir.txt
I know that both directories exist. Not sure if I'm getting caught up on the loop or on the IFS.
The following program reads a file and it intends to store the all values (each line) into a variable but doesn't store the last line. Why?
file.txt :
1
2
.
.
.
n
Code :
FileName=file.txt
if test -f $FileName # Check if the file exists
then
while read -r line
do
fileNamesListStr="$fileNamesListStr $line"
done < $FileName
fi
echo "$fileNamesListStr" // 1 2 3 ..... n-1 (but it should print up to n.)
Instead of reading line-by-line, why not read the whole file at once?
[ -f $FileName ] && fileNameListStr=$( tr '\n' ' ' < $FileName )
One probable cause is that there misses a newline after the last line n.
Use the following command to check it:
tail -1 file.txt
And the following fixes:
echo >> file.txt
If you really need to keep the last line without newline, I reorganized the while loop here.
#!/bin/bash
FileName=0
if test -f $FileName ; then
while [ 1 ] ; do
read -r line
if [ -z $line ] ; then
break
fi
fileNamesListStr="$fileNamesListStr $line"
done < $FileName
fi
echo "$fileNamesListStr"
The issue is that when the file does not end in a newline, read returns non-zero and the loop does not proceed. The read command will still read the data, but it will not process the loop. This means that you need to do further processing outside of the loop. You also probably want an array instead of a space separated string.
FileName=file.txt
if test -f $FileName # Check if the file exists
then
while read -r line
do
fileNamesListArr+=("$line")
done < $FileName
[[ -n $line ]] && fileNamesListArr+=("$line")
fi
echo "${fileNameListArr[#]}"
See the "My text files are broken! They lack their final newlines!" section of this article:
http://mywiki.wooledge.org/BashFAQ/001
As a workaround, before reading from the text file a newline can be appended to the file.
echo "\n" >> $file_path
This will ensure that all the lines that was previously in the file will be read. Now the file can be read line by line.
I need to validate my log files:
-All new log lines shall start with date.
-This date will respect the ISO 8601 standard. Example:
2011-02-03 12:51:45,220Z -
Using shell script, I can validate it looping on each line and verifying the date pattern.
The code is below:
#!/bin/bash
processLine(){
# get all args
line="$#"
result=`echo $line | egrep "[0-9]{4}-[0-9]{2}-[0-9]{2} [012][0-9]:[0-9]{2}:[0-9]{2},[0-9]{3}Z" -a -c`
if [ "$result" == "0" ]; then
echo "The log is not with correct date format: "
echo $line
exit 1
fi
}
# Make sure we get file name as command line argument
if [ "$1" == "" ]; then
echo "You must enter a logfile"
exit 0
else
file="$1"
# make sure file exist and readable
if [ ! -f $file ]; then
echo "$file : does not exists"
exit 1
elif [ ! -r $file ]; then
echo "$file: can not read"
exit 2
fi
fi
# Set loop separator to end of line
BAKIFS=$IFS
IFS=$(echo -en "\n\b")
exec 3<&0
exec 0<"$file"
while read -r line
do
# use $line variable to process line in processLine() function
processLine $line
done
exec 0<&3
# restore $IFS which was used to determine what the field separators are
IFS=$BAKIFS
echo SUCCESS
But, there is a problem. Some logs contains stacktraces or something that uses more than one line, in other words, stacktrace is an example, it can be anything. Stacktrace example:
2011-02-03 12:51:45,220Z [ERROR] - File not found
java.io.FileNotFoundException: fred.txt
at java.io.FileInputStream.<init>(FileInputStream.java)
at java.io.FileInputStream.<init>(FileInputStream.java)
at ExTest.readMyFile(ExTest.java:19)
at ExTest.main(ExTest.java:7)
...
will not pass with my script, but is valid!
Then, if I run my script passing a log file with stacktraces for example, my script will failed, because it loops line by line.
I have the correct pattern and I need to validade the logger date format, but I don't have wrong date format pattern to skip lines.
I don't know how I can solve this problem. Does somebody can help me?
Thanks
You need to anchor your search for the date to the start of the line (otherwise the date could appear anywhere in the line - not just at the beginning).
The following snippet will loop over all lines that do not begin with a valid date. You still have to determine if the lines constitute errors or not.
DATEFMT='^[0-9]{4}-[0-9]{2}-[0-9]{2} [012][0-9]:[0-9]{2}:[0-9]{2},[0-9]{3}Z'
egrep -v ${DATEFMT} /path/to/log | while read LINE; do
echo ${LINE} # did not begin with date.
done
So just (silently) discard a single stack trace. In somewhat verbose bash:
STATE=idle
while read -r line; do
case $STATE in
idle)
if [[ $line =~ ^java\..*Exception ]]; then
STATE=readingexception
else
processLine "$line"
fi
;;
readingexception)
if ! [[ $line =~ ^' '*'at ' ]]; then
STATE=idle
processLine "$line"
fi
;;
*)
echo "Urk! internal error [$STATE]" >&2
exit 1
;;
esac
done <logfile
This relies on processLine not continuing on error, else you will need to track a tad more state to avoid two consecutive stack traces.
This makes 2 assumptions.
lines that begin with whitespace are continuations of previous lines. we're matching a leading space, or a leading tab.
lines that have non-whitespace characters starting at ^ are new log lines.
If a line matching #2 doesn't match the date format, we have an error, so print the error, and include the line number.
count=0
processLine() {
count=$(( count + 1 ))
line="$#"
result=$( echo $line | egrep '^[0-9]{4}-[0-9]{2}-[0-9]{2} [012][0-9]:[0-9]{2}:[0-9]{2},[0-9]{3}Z' -a -c )
if (( $result == 0 )); then
# if result = 0, then my line did not start with the proper date.
# if the line starts with whitespace, then it may be a continuation
# of a multi-line log entry (like a java stacktrace)
continues=$( echo $line | egrep "^ |^ " -a -c )
if (( $continues == 0 )); then
# if we got here, then the line did not start with a proper date,
# AND the line did not start with white space. This is a bad line.
echo "The line is not with correct date format: "
echo "$count: $line"
exit 1
fi
fi
}
Create a condition to check if the line starts with a date. If not, skip that line as it is part of a multi-line log.
processLine(){
# get all args
line="$#"
result=`echo $line | egrep "[0-9]{4}-[0-9]{2}-[0-9]{2} [012][0-9]:[0-9]{2}:[0-9]{2},[0-9]{3}Z" -a -c`
if [ "$result" == "0" ]; then
echo "Log entry is multi-lined - continuing."
fi
}