I got such a piece of bash code:
var="empty"
find $path1 -maxdepth 3 | while read line; do
find $path2 -maxdepth 1 | while read line2; do
if [[ $line2 != $var ]]; then
echo "new value"
fi
var=$line2
done <<< "$line2"
done
The question is... how to make var stay changed? Because I would like to echo on every new value found by loops but it doesn't work ;( var="empty" every time that the second loop starts iteration.
How to make var=$line2 for every iteration?
You are reading the value into line2 with a read from stdin, and feeding the value of line2 into the loop at the done with a here-string on stdin. bash gives the here-string precedence, so line2 is only ever being assigned from line2, which means it's never set.
echo -e "one\nthree\nfive" | while read num
do echo $num
done <<< "two"
Output is two. The input stream is totally ignored.
You are also defining a nested loop for no reason, since you are never using the outer loop. Clean your code before posting please.
find ~ | while read f; do var=$f; echo $f; done
This works fine.
Related
I am writing a shell script that contains a multi-threaded while loop. My loop iterates through the values of an array. Within the loop, I am calling a function. At the end of the function I am saving the results as a string variable. I want to add this string variable to an array on each iteration, and then be able to retrieve the contents of this array when the while loop completes.
From my understanding running the multi-threaded while loop, is what is causing for the array to be empty once the while loop completes. Each thread is ran in its own environment and the array value does not extend outside that environment. I would like to be able to extend this array value outside of the thread if possible. Currently I am just writing the string value to a temp file and then after the while loop, reading the contents of the temp file and saving that as my array. This method works, as the file generally isn't "too" large, but I would like to avoid writing to file if possible
My Code - doDeepLookup actually is a API call, but for the sake of argument lets just say it appends some text in-front of the read line from the while loop
#!/bin/bash
n=0
maxjobs=20
resultsArray=""
while IFS= read -r line
do
IPaddress="$(echo $line | sed 's/ /\n/g' | grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}")"
doDeepLookup "$line" "$IPaddress" &
if(( $(($((++n)) % $maxjobs)) == 0 )) ; then
wait
fi
done <<< "$(printf '%s\n' "${SomeOtherArray[#]}")"
printf '%s\n' "${resultsArray[#]}" #Returns NULL
doDeepLookup() {
results="$(echo "help me : $line")"
resultsArray+=($results)
}
Thanks to William
#!/bin/bash
n=0
maxjobs=20
WhileLoopFunction() {
resultsArray=""
while IFS= read -r line
do
IPaddress="$(echo $line | sed 's/ /\n/g' | grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}")"
doDeepLookup "$line" "$IPaddress" &
if(( $(($((++n)) % $maxjobs)) == 0 )) ; then
wait
fi
done <<< "$(printf '%s\n' "${SomeOtherArray[#]}")"
}
doDeepLookup() {
results="$(echo "help me : $line")"
echo $results
}
resultsArray=( $(WhileLoopFunction"${DeepArray[#]}") )
printf '%s\n' "${resultsArray[#]}"
With parset from GNU Parallel you would do something like:
parset resultsArray doDeepLookup ::: "${DeepArray[#]}"
printf '%s\n' "${resultsArray[#]}"
I am horribly perplexed.
I've written a bash script to sort lines into categories based on substrings within that line.
Here's my example "lines.txt"
i am line1
i am line2
If a line contains "line1", then it should be sorted into group "l1". If it contains "line2", then it should be sorted into group "l2"
The problem is that the variable which holds the category isn't retaining its value, and I have no clue why. Here's the script.
#!/bin/bash
categories="l1 l2"
l1="
line1
"
l2="
line2
"
# match line1
cat lines.txt | while read fline
do
cate="no match"
for c in $categories
do
echo "${!c}" | while read location
do
if [ ! -z "$location" ] && [[ "$fline" =~ "$location" ]]
then
echo "we are selecting category $c"
cate="$c"
break
fi
done
if [ "$cate" != "no match" ]
then
echo "we found a match"
break
fi
done
echo "$cate:$fline"
done
exit 0
And when I run it, I see the output
we are selecting category l1
no match:i am line1
we are selecting category l2
no match:i am line2
This means that we are selecting the correct group, but we don't remember it when we exit the nested "while" loop.
Why is my variable not retaining its value, and how could I fix that?
The while loop is executed in a subshell because of the pipe. That means that the name 'cate' really refers to two different variables. One outside the while loop and the other inside the loop inside the subshell. When the subshell exits that value is lost.
A way to get around this is to use a redirect like this
while read line; do
...
done < $myfile
If the expression is more complicated and you need something executed in a subshell, then you can use process substitution (Thanks to David Rankin for reminding me about this one).
while read -r line; do
...
done < <(find . -iname "*sh")
I'm trying to write a small script that will count entries in a log file, and I'm incrementing a variable (USCOUNTER) which I'm trying to use after the loop is done.
But at that moment USCOUNTER looks to be 0 instead of the actual value. Any idea what I'm doing wrong? Thanks!
FILE=$1
tail -n10 mylog > $FILE
USCOUNTER=0
cat $FILE | while read line; do
country=$(echo "$line" | cut -d' ' -f1)
if [ "US" = "$country" ]; then
USCOUNTER=`expr $USCOUNTER + 1`
echo "US counter $USCOUNTER"
fi
done
echo "final $USCOUNTER"
It outputs:
US counter 1
US counter 2
US counter 3
..
final 0
You are using USCOUNTER in a subshell, that's why the variable is not showing in the main shell.
Instead of cat FILE | while ..., do just a while ... done < $FILE. This way, you avoid the common problem of I set variables in a loop that's in a pipeline. Why do they disappear after the loop terminates? Or, why can't I pipe data to read?:
while read country _; do
if [ "US" = "$country" ]; then
USCOUNTER=$(expr $USCOUNTER + 1)
echo "US counter $USCOUNTER"
fi
done < "$FILE"
Note I also replaced the `` expression with a $().
I also replaced while read line; do country=$(echo "$line" | cut -d' ' -f1) with while read country _. This allows you to say while read var1 var2 ... varN where var1 contains the first word in the line, $var2 and so on, until $varN containing the remaining content.
Always use -r with read.
There is no need to use cut, you can stick with pure bash solutions.
In this case passing read a 2nd var (_) to catch the additional "fields"
Prefer [[ ]] over [ ].
Use arithmetic expressions.
Do not forget to quote variables! Link includes other pitfalls as well
while read -r country _; do
if [[ $country = 'US' ]]; then
((USCOUNTER++))
echo "US counter $USCOUNTER"
fi
done < "$FILE"
minimalist
counter=0
((counter++))
echo $counter
You're getting final 0 because your while loop is being executed in a sub (shell) process and any changes made there are not reflected in the current (parent) shell.
Correct script:
while read -r country _; do
if [ "US" = "$country" ]; then
((USCOUNTER++))
echo "US counter $USCOUNTER"
fi
done < "$FILE"
I had the same $count variable in a while loop getting lost issue.
#fedorqui's answer (and a few others) are accurate answers to the actual question: the sub-shell is indeed the problem.
But it lead me to another issue: I wasn't piping a file content... but the output of a series of pipes & greps...
my erroring sample code:
count=0
cat /etc/hosts | head | while read line; do
((count++))
echo $count $line
done
echo $count
and my fix thanks to the help of this thread and the process substitution:
count=0
while IFS= read -r line; do
((count++))
echo "$count $line"
done < <(cat /etc/hosts | head)
echo "$count"
USCOUNTER=$(grep -c "^US " "$FILE")
Incrementing a variable can be done like that:
_my_counter=$[$_my_counter + 1]
Counting the number of occurrence of a pattern in a column can be done with grep
grep -cE "^([^ ]* ){2}US"
-c count
([^ ]* ) To detect a colonne
{2} the colonne number
US your pattern
Using the following 1 line command for changing many files name in linux using phrase specificity:
find -type f -name '*.jpg' | rename 's/holiday/honeymoon/'
For all files with the extension ".jpg", if they contain the string "holiday", replace it with "honeymoon". For instance, this command would rename the file "ourholiday001.jpg" to "ourhoneymoon001.jpg".
This example also illustrates how to use the find command to send a list of files (-type f) with the extension .jpg (-name '*.jpg') to rename via a pipe (|). rename then reads its file list from standard input.
I'm trying to parse a csv file I made with Google Spreadsheet. It's very simple for testing purposes, and is basically:
1,2
3,4
5,6
The problem is that the csv doesn't end in a newline character so when I cat the file in BASH, I get
MacBook-Pro:Desktop kkSlider$ cat test.csv
1,2
3,4
5,6MacBook-Pro:Desktop kkSlider$
I just want to read line by line in a BASH script using a while loop that every guide suggests, and my script looks like this:
while IFS=',' read -r last first
do
echo "$last $first"
done < test.csv
The output is:
MacBook-Pro:Desktop kkSlider$ ./test.sh
1 2
3 4
Any ideas on how I could have it read that last line and echo it?
Thanks in advance.
You can force the input to your loop to end with a newline thus:
#!/bin/bash
(cat test.csv ; echo) | while IFS=',' read -r last first
do
echo "$last $first"
done
Unfortunately, this may result in an empty line at the end of your output if the input already has a newline at the end. You can fix that with a little addition:
!/bin/bash
(cat test.csv ; echo) | while IFS=',' read -r last first
do
if [[ $last != "" ]] ; then
echo "$last $first"
fi
done
Another method relies on the fact that the values are being placed into the variables by the read but they're just not being output because of the while statement:
#!/bin/bash
while IFS=',' read -r last first
do
echo "$last $first"
done <test.csv
if [[ $last != "" ]] ; then
echo "$last $first"
fi
That one works without creating another subshell to modify the input to the while statement.
Of course, I'm assuming here that you want to do more inside the loop that just output the values with a space rather than a comma. If that's all you wanted to do, there are other tools better suited than a bash read loop, such as:
tr "," " " <test.csv
cat file |sed -e '${/^$/!s/$/\n/;}'| while IFS=',' read -r last first; do echo "$last $first"; done
If the last (unterminated) line needs to be processed differently from the rest, #paxdiablo's version with the extra if statement is the way to go; but if it's going to be handled like all the others, it's cleaner to process it in the main loop.
You can roll the "if there was an unterminated last line" into the main loop condition like this:
while IFS=',' read -r last first || [ -n "$last" ]
do
echo "$last $first"
done < test.csv
Say you have a simple loop
while read line
do
printf "${line#*//}\n"
done < text.txt
Is there an elegant way of printing the current iteration with the output? Something like
0 The
1 quick
2 brown
3 fox
I am hoping to avoid setting a variable and incrementing it on each loop.
To do this, you would need to increment a counter on each iteration (like you are trying to avoid).
count=0
while read -r line; do
printf '%d %s\n' "$count" "${line*//}"
(( count++ ))
done < test.txt
EDIT: After some more thought, you can do it without a counter if you have bash version 4 or higher:
mapfile -t arr < test.txt
for i in "${!arr[#]}"; do
printf '%d %s' "$i" "${arr[i]}"
done
The mapfile builtin reads the entire contents of the file into the array. You can then iterate over the indices of the array, which will be the line numbers and access that element.
You don't often see it, but you can have multiple commands in the condition clause of a while loop. The following still requires an explicit counter variable, but the arrangement may be more suitable or appealing for some uses.
while ((i++)); read -r line
do
echo "$i $line"
done < inputfile
The while condition is satisfied by whatever the last command returns (read in this case).
Some people prefer to include the do on the same line. This is what that would look like:
while ((i++)); read -r line; do
echo "$i $line"
done < inputfile
You can use a range to go through, it can be an array, a string, a input line or a list.
In this example, i use a list of numbers [0..10] is used with an increment of 2, as well.
#!/bin/bash
for i in {0..10..2}; do
echo " $i times"
done
The output is:
0 times
2 times
4 times
6 times
8 times
10 times
To print the index regardless of the loop range, you have to use a variable "COUNTER=0" and increase it in each iteration "COUNTER+1".
my solution prints each iteration, the FOR traverses an inputline and increments by one each iteration, also shows each of words in the inputline:
#!/bin/bash
COUNTER=0
line="this is a sample input line"
for word in $line; do
echo "This i a word number $COUNTER: $word"
COUNTER=$((COUNTER+1))
done
The output is:
This i a word number 0: this
This i a word number 1: is
This i a word number 2: a
This i a word number 3: sample
This i a word number 4: input
This i a word number 5: line
to see more about loops: enter link description here
to test your scripts: enter link description here
n=0
cat test.txt | while read line; do
printf "%7s %s\n" "$n" "${line#*//}"
n=$((n+1))
done
This will work in Bourne shell as well, of course.
If you really want to avoid incrementing a variable, you can pipe the output through grep or awk:
cat test.txt | while read line; do
printf " %s\n" "${line#*//}"
done | grep -n .
or
awk '{sub(/.*\/\//, ""); print NR,$0}' test.txt
Update: Other answers posted here are better, especially those of #Graham and #DennisWilliamson.
Something very like this should suit:
tr -s ' ' '\n' <test.txt | nl -ba
You can add a -v0 flag to the nl command if you want indexing from 0.