I have a function:
something() {
if [ something ]; then
echo "Something.";
return 0;
else
echo "Not something.";
return 1;
fi
}
I call it in a loop, it actually validates some files and counts how many files were valid:
find . -type l | while read line ; do something "$line"; done
I need to count how many files were invalid, therefore how many times the function has returned 0. I was thinking about this:
INVALID=0;
find . -type l | while read line ; do INVALID=$(($INVALID + something "$line")); done
Needless to say, bash doesn't buy it. Note a few things:
The info within something must be print in stdout.
The info print does not always correlate with file validity in my code. It's just info for the user.
The return value isn't directly available for arithmetic like that. You can either call the function then access $?, or branch based on the result of the function, like so:
INVALID=0
while IFS= read -r line; do
something "$line" || ((++INVALID))
done < <(find . -type l)
Also note that you can't change variables inside a pipeline. Pipelines run in subshells and have their own copies of variables. You'll need to restructure the loop to run without a pipeline to have the changes to $INVALID stick: change find | loop to loop < <(find).
It's also a good idea to use read -r to disable backslash escapes and clear out $IFS to handle lines with leading whitespace better.
Related
I have a file with a structured list of filenames (file1.sh, file2.sh, ...) and would like to read loop the file names inside a bash script.
cat /home/flora/logs/9681-T13:17:07.091363777.org
%rec: dynamic
Ptrn: Gnu
File: /home/flora/comint.rc
+ /home/flora/engine.rc
+ /home/flora/playa.rc
+ /home/flora/edva.rc
+ /home/flora/dyna.rc
+ /home/flora/lin.rc
Have started with
while read -r fl; do
echo "$fl" | grep -oE '[/].+'
done < "$logfl"
But I want to be more specific by matching the File: , then continue reading the rest using + as a continuation character.
bash doesn't have impose a limit on variables (other than memory). That said, I would start by processing the list of lines one by one:
#!/bin/bash
while read _ f
do
process "$f"
done
where process is whatever function you need to implement.
If you want a variables use an array like this:
#!/bin/bash
while read _ f
do
files+=("$f")
done
In either case pass the input file to script with:
your_script < /home/flora/logs/27043-T13:09:44.893003954.log
I have the following script and I want to overwrite the progress var with the value from the loop but I'm struggling to get this working. I've read many threads but all of them seem to suggest something like writing to output to a file and then read from that again but I don't want to do that.
#!/bin/bash
function Upload {
find . -type f -name "*.test" -printf "%f\n"| sort -k1 | while read fname; do
progressBar=$(echo "scale=2 ; $progress + $percentage" | bc)
echo "bar: $progressBar"
progress=$progressBar
done
}
progress=0
totalFiles=$(find . -name "*.test" | wc -l)
totalCalc=$(($totalFiles + 1))
percentage=$(echo "scale=2 ; 100 / $totalCalc" | bc)
Upload
echo $progress
How can I get the var outside the loop/subshell and overwrite the main var?
As you already correctly point out in your question, you are running the loop body in a child process, so even if you would export the variable, a change can't be seen in the parent process.
Writing to a file is and retrieve it in your "main" program is easy if you do it properly:
In your main program (before calling your function), set up the file like this:
export progressBarFile=/tmp/progressBar.$$
This ensures that several processes of your script use their own file for it.
In your loop, do a
echo $progressBar >$progressBarFile
Then, after your function, fetch the value using
progressBar=$(<$progressBarFile)
In the end, you can erase this file like this:
rm $progressBarFile
If you are really paranoid of having a temporary file left over, you could use trap to catch a premature abort of your script, and erase the file inside the trap function.
Another possibility (without using a file) would be to use an array:
files=( $(find . -type f -name "*.test" -printf "%f\n"| sort -k1) )
for fname in "${files[#]}"
do
...
done
In this case, your loop body is not a child process.
I have one queston about nested loop with bash.
I have an input files with one file name per line (full path)
I read this file and then i make a nest loop:
for i in $filelines ; do
echo $i
for j in $filelines ; do
./program $i $j
done
done
The program I within the loop is pretty low.
Basically it compare the file A with the file B.
I want to skip A vs A comparison (i.e comparing one file with itslef) AND
I want to avoid permutation (i.e. for file A and B, only perform A against B and not B against A).
What is the simplest to perform this?
Version 2: this one takes care of permutations
#!/bin/bash
tmpunsorted="/tmp/compare_unsorted"
tmpsorted="/tmp/compare_sorted"
>$tmpunsorted
while read linei
do
while read linej
do
if [ $linei != $linej ]
then
echo $linei $linej | tr " " "\n" | sort | tr "\n" " " >>$tmpunsorted
echo >>$tmpunsorted
fi
done <filelines
done <filelines
sort $tmpunsorted | uniq > $tmpsorted
while read linecompare
do
echo "./program $linecompare"
done <$tmpsorted
# Cleanup
rm -f $tmpunsorted
rm -f $tmpsorted
What is done here:
I use the while loop to read each line, twice, i and j
if the value of the lines is the same, forget them, no use to consider them
if they are different, output them into a file ($tmpunsorted). And they are sorted in alphebetical order before going tothe $tmpunsorted file. This way the arguments are always in the same order. So a b and b a will be same in the unsorted file.
I then apply sort | uniq on $tmpunsorted, so the result is a list of individual argument pairs.
finally loop on the $tmpsorted file, and call the program on each individual pair.
Since I do not have your program, I did an echo, which you should remove to use the script.
I need to write bash script that converts a string of only integers "intString" to :id. intString always exists after /, may never contain any other types (create_step2 is not a valid intString), and may end at either a second / or end of line. intString may be any 1-8 characters. Script needs to be repeated for every line in a given file.
For example:
/sample/123456/url should be converted to /sample/:id/url
and /sample_url/9 should be converted to /sampleurl/:id however /sample_url_2/ should remain the same.
Any help would be appreciated!
It seems like the long way around the problem to go recursive but then I don't know what problem you are solving. It seems like a good sed command like
sed -E 's/\/[0-9]{1,}/\/:id/g'
could do it in one shot, but if you insist on being recursive then it might go something like this ...
#!/bin/bash
function restring()
{
s="$1"
s="$(echo $s | sed -E 's/\/[0-9]{1,}/\/:id/')"
if ( echo $s | grep -E '\/[0-9]{1,}' > /dev/null ) ; then
restring $s
else
echo $s
exit
fi
echo $s
}
restring "$1"
now run it
$ ./restring.sh "/foo/123/bar/456/baz/45435/andstuff"
/foo/:id/bar/:id/baz/:id/andstuff
There's a getStrings() function that calls getPage() function that returns some html page. That html is piped through egrep and sed combination to get only 3 strings. Then I try to put every string into separate variable link, profile, gallery respectively using while read.. construction. But it works only inside the while...done loop because it runs in subprocess. What should I do to use those variables outside the getStrings() function?
getStrings() {
local i=2
local C=0
getPage $(getPageLink 1 $i) |
egrep *some expression that results in 3 strings* |
while read line; do
if (( (C % 3) == 0 )); then
link=$line
elif (( (C % 3) == 1 )); then
profile=$line
else
gallery=$line
fi
C=$((C+1)) #Counter
done
}
Simple: don't run the loop in a subprocess :)
To actually accomplish that, you can use process substitution.
while read line; do
...
done < <(getPage $(getPageLink 1 $i) | egrep ...)
For the curious, a POSIX-compatible way is to use a named pipe (and its possible that bash uses named pipes to implement process substitution):
mkfifo pipe
getPage $(getPageLink 1 $i) | egrep ... > pipe &
while read line; do
...
done < pipe
Starting in bash 4.2, you can just set the lastpipe option, which causes the last command in a pipeline to run in the current shell, rather than a subshell.
shopt -s lastpipe
getPage $(getPageLink 1 $i) | egrep ... | while read line; do
...
done
However, using a while loop is not the best way to set the three variables. It's easier to just call read three times within a command group, so that they all read from the same stream. In any of the three scenarios above, replace the while loop with
{ read link; read profile; read gallery; }
If you want to be a little more flexible, put the names of the variables you might want to read in an array:
fields=( link profile gallery )
then replace the while loop with this for loop instead:
for var in "${fields[#]}"; do read $var; done
This lets you easily adjust your code, should the pipeline ever return more or fewer lines, by just editing the fields array to have the appropriate field names.
One more solving using array:
getStrings() {
array_3=( `getPage | #some function
egrep | ...` ) #pipe conveyor
}