Value "4" below is the number of CPU threads. Idea is to run the tasks in batch of 4 and wait until the current batch is finished before starting the next batch.
Syntax 1:
while read something; do
((++i%4==0)) && wait
(
task using something as input;
)
done < input_file.txt
Syntax 2:
while read something; do
((i=i%4)); ((i++==0)) && wait
(
task using something as input;
)
done < input_file.txt
To me they both work the same except the second one is longer. But when running in the cloud (AWS ubuntu 14.04), only syntax 1 worked. The syntax2 threw a generic syntax error at "((i=i%4));" step and it became a mystery.
"The second one is longer" doesn't help since you used pseudocode.
Maybe this will help:
while read x; do ((i=++i%4)) || wait; sleep $x & done < input_file.txt
My input_file.txt:
10
9
8
7
6
5
4
3
2
1
Related
for ( i=3; i<5; i++)
do
execute some command 1
if command 2 is successful then do not run the command 1 (the for loop should continue)
if command 2 is not successful then run command 1 only once (like retry command 1 only once, after this the for loop should continue)
done
This is to note that command 2 is dependent on command 1 and command 2 can only be executed after command 1
for example:
for ( i=3; i<5; i++)
do
echo "i" >> mytext.txt ---> command 1
if "check the content of mytext.txt file to see if the value of i is actually added" ---> command 2
if it is not added then execute echo "i" >> mytext.txt (command 1) again and only once.
if i value is added to the file .. then exit and continue the loop
done
Since the "command 1" is quite big and not just an example echo statement here.I do not want to add "command 1" twice .. once outside and once inside the if condition. I want this logic in an optimized way with no redundancy of code.
Per a comment it sounds like the OP may need to invoke command 1 up to 2 times for a given $i value, but only wants to type command 1 once in the script.
Siddhartha's suggestion to use a function is probably good enough but depending on the actual command 1 (OP mentions that it's 'quite big') I'm going to play devil's advocate and assume there could be additional issues with passing some args to the function (eg, a need to escape some characters ... ??).
The general idea is to have an internal loop that can be executed at most 2 times, with logic in the loop that will allow for an 'early' exit (eg, after just one pass through the loop).
Since we're using pseudo-code I'll use the same ...
for ( i=3; i<5; i++ )
do
pass=1 # reset internal loop counter
while ( pass -le 2 )
do
echo "i" >> mytext.txt # command 1
if ( pass -eq 1 ) # after first 'command 1' execution
&& ( value of 'i' is in mytext.txt ) # command 2
then
break # break out of inner loop; alternatively ...
# pass=10 # ensure pass >= 2 to force loop to exit on this pass
fi
pass=pass+1 # on 1st pass set pass=2 => allows another pass through loop
# on 2nd pass set pass=3 => will force loop to exit
done
done
you can declare functions like
function command
{
your_command -f params
}
for ( i=3; i<5; i++)
do
if command ; then
echo "success"
else
echo "retry"
command
fi
done
I’ve got this bash script to download 52k files:
for i in {1..52000};
do wget -c "download.hebrewbooks.org/downloadhandler.ashx?req=$i" ;
done
However the server gives me an 429 error.
How can I pause the loop for X amount of time after every 5 files that are downloaded?
If i is a multiple of five, sleep.
for i in {1..52000}; do
wget -c "download.hebrewbooks.org/downloadhandler.ashx?req=$i"
((i % 5)) || sleep $X
done
Note that ((expr)) returns the Boolean value of expr, where false=0 and true=1, which is the opposite of normal Bash return codes. That's why you have to use OR || instead of AND &&. If that's too confusing, use this instead: ((i % 5 == 0)) && ...
print:
#for number in 10 11 12 13 14 15; do \
( echo "Number: $$number" ); \
( break ); \
( echo Break not working ); \
done
The output that i am getting
Number: 10
Break not working
Number: 11
Break not working
Number: 12
Break not working
Number: 13
Break not working
Number: 14
Break not working
Number: 15
Break not working
The output that i need :
Number: 10
In the posted code the 'for' loop is executed in the shell (bash ?), not by the 'make' utility.
The problem with the bash script is that the 'break' statement is running in a sub-process - as it is placed inside parenthesis '( break '). As per bash manual, control-flow commands (while, if, for, return, ...) should executed in the "main" process. It is not possible to split them across processes.
Removing the '(' should solve the problem - the loop will stop after printing 'Number: 10'.
print:
#for number in 10 11 12 13 14 15; do \
( echo "Number: $$number" ); \
break ; \
( echo Break not working ); \
done
As a side note, no need to put 'echo' statements in '( ... )'. It make the script fork unnecessary instances o bash.
I am quite new in bash, but I need to create a simple script which will do below steps:
Wait 1 minute
A) bash script will use CM to generate result file
B) check row 8 in result file (to know if Administrator is running any jobs or not)
if NO jobs:
C) bash script will use CM to start cube refresh
D) wait 1 minute
D1) Remove result file
E) generate result file
E1) Read row 8
no jobs:
F) remove result file G) EXIT
yes:
I) Go to D)
YES:
E) Wait 1 minute
F) Remove result file
Go to A)
As bash doesn't have goto (or should not be use), I tried few loops, but I not sure which I should choose.
I know how to:
- start cube(step C)
- generate result file (step A & E):
- check line 8:
sed '8!d' /abc_uat/cmlogs/adm_jobs_u1.log
condition for loops will be probably similar to this: !='Owner = Administrator'
but how to avoid goto ?
I tried with while do loop, but I am not sure what should I add in case of false condition, I added else, but not sure of it:
sleep 60
Generate result file with admin jobs (which admin runs inside of 3rd party tool)
while [ sed '8!d' admin_jobs_result_file.log !="Owner = Administrator" ];
do
--NO Admin jobs
START CUBE REFRESH (it will start admin job)
sleep 60
REMOVE RESULT FILE (OLD)
GENERATE RESULT FILE
while [ sed '8!d' admin_jobs_result_file.log = "Owner = Administrator" ];
--Admin is still running cube refresh
do
sleep 60
REMOVE RESULT FILE (OLD)
GENERATE RESULT FILE
-- it should continue checking every 1 minute if admin is still running cube refresh job, so I hope it will go back to while condition
else
done
else
-- Admin is running something
sleep 60
REMOVE RESULT FILE (OLD)
GENERATE RESULT FILE
-it should check result file again but I think it will finish loop
done
You can replace goto with a loop. while loop, for example.
Syntax
while <condition>
do
action
done
Check out cron jobs. Delegate, if possible, "waiting for a minute" task to cron. Cron should worry about running your script on a timely fashion.
You may consider writing two scripts instead of one.
Do you really need to create a result file? Do you know piping ? (no offense, just mentioning it because you said you were fairly new to bash)
Hopefully this is self explanatory.
result_file=admin_jobs_result_file.log
function generate {
logmsg sleeping
sleep 60
rm -f "$result_file"
logmsg generating
# use CM to generate result file
}
function owner_is_administrator {
# if line 8 contains "Owner = Administrator", exit success
# else exit failure
sed -n '8 {/Owner = Administrator/ q 0; q 1}' "$result_file"
}
function logmsg { date "+%Y-%m-%d %T -- $*"; }
##############
generate
while owner_is_administrator; do
generate
done
# at this point, line 8 does NOT contain "Owner = Administrator"
logmsg start cube refresh
# use CM to start cube refresh
generate
while owner_is_administrator; do
generate
done
logmsg Done
Looks like AIX's sed can't exit with a specified status. Try this instead:
function owner_is_administrator {
# if line 8 contains "Owner = Administrator", exit success
# else exit failure
awk 'NR == 8 {if (/Owner = Administrator/) {exit 0} else {exit 1}}' "$result_file"
}
So I have a project in my cyber security class to make a bash game. I like to make one of those medieval games where you make farms and mines to get resources. Well I like to make something like that. To do that I have to have two while loops running. Like this
while [ blah ]; do
blah
done
while [ blah ]; do
blah
done
Is it possible to run two while loops at the same time and if I am writing it wrong how do I write it?
If you put a & after each done, like done&, you will create new processes in the background that run the while loops. You will have to be careful to realize what this means though, since the bash script will continue executing commands after creating those new processes even if they are not finished. You might use the wait command to prevent this from happening, but I'm not too used to using that so I cannot vouch for it.
Yes, but you will have to fork a new process for each while loop to be executing in. Technically, they won't both run at the same time (unless you consider multiple cores, but this isn't even garaunteed).
Below is a link to how to fork multiple processes using bash.
Forking / Multi-Threaded Processes | Bash
Since you mention this is a school project, I'll stop here lest I help you "not learn".
R
First things first, wrap the loop into a function and then fork it.
This is done when you want to split a process, for example, if I'm processing a CSV with 160,000+ lines, single process/"thread" will take hours. If you wrap the loop into a function and simply fork it, you will have x amount of processes running, then add wait/kill defunct process loop and you are done. here what you are looking at.
while loop with nested loop:
function jobA() {
while read STR;
do
touch $1_temp
key=$(IFS="|";set -- $STR; echo $1)
for each in ${blah[#]};
do
#echo "$each"
done
done <$1;
}
for i in ${blah[#]};
do
echo "$i"
$(jobRDtemp $i) &
child_pid=$!
parent_pid=$$
PIDS+=($child_pid)
echo "forked process $child_pid with parent $parent_pid"
done
for pid in ${PIDS[#]};
do
wait $pid
done
echo "all jobs done"
sleep 1
Now this is wrapped, here is example of a FORKED loop. this means you will have parallel processes run in the background, WAIT will wait for ALL to complete before proceeding. This is important for some type of scripts.
Also, DO NOT use nested FOR loops written C style like presented above, example:
for (( i = 1; i <= 5; i++ )) ### Outer for loop ###
This is VERY slow. use THIS type:
for each in ${blah[#]};
do
#echo "$each"
if [ "$key" = "$each" ]; then
# echo "less than $keyValNeed..."
echo $STR >> $1_temp
fi
done
You could also use nested for loops
for (( i = 1; i <= 5; i++ )) ### Outer for loop ###
do
for (( j = 1 ; j <= 5; j++ )) ### Inner for loop ###
do
echo -n "$i "
done
echo "" #### print the new line ###
done
EDIT: I thought you meant Nested Loop but reading again you said running both loops "at the same time". I will leave my answer here though.