I have several (idle) jobs scheduled on a cluster that I want to move to another queue.
I can move a single job like this (where 1234 is the job id):
qmove newQueue 1234
But now I have hundreds of jobs that I want to move to newQueue. Is it possible to move them all? Using * as a wildcard operator does not work.
If the job ids are in sequential order, you could use Bash's brace extension. For example:
$ echo {0..9}
0 1 2 3 4 5 6 7 8 9
Transferred to moving all jobs ranging from 1000 to 2000, the qmove command would be:
qmove newQueue {1000..2000}
This might even work if there are job ids that you are not allowed to move (from other users or in running state). They should be simply ignored. (not tested)
Related
We have shared server with multiple GPU nodes without resource manager. We make agreements that: "this week you can use nodes ID1,ID2 and ID5". I have a program that gets this IDÂ as a parameter.
When I need to run my program ten times with ten different sets of parameters $ARGS1, $ARGS2, ..., $ARGS10, I run first three commands
programOnGPU $ARGS1 -p ID1 &
programOnGPU $ARGS2 -p ID2 &
programOnGPU $ARGS3 -p ID5 &
Then I must wait for any of those to finish and if e.g ID2 finishes first, I then run
programOnGPU $ARGS4 -p ID2 &
As this is not very convenient when you have a lot of processes I would like to automatize the process. I can not use parallel as I need to reuse IDs.
First use case is a script that needs to execute apriori known 10 commands of the type
programOnGPU $PARAMS -p IDX
when any of them finishes to assign its ID to another one in the queue. Is this possible using bash without too much overhead of the type of the SLURM? I don't need to check the state of physical resource.
General solution would be if I can make a queue in the bash or simple command line utility to which I will submit commands of the type
programABC $PARAMS
and it will add the GPU IDÂ parameter to it and manage the queue that will be preconfigured to be able to use just given IDs and one ID at once. Again I don't want this layer to touch physical GPUs, but to ensure that it executes consistently over allowed ID's.
This is very simple with Redis. It is a very small, very fast, networked, in-memory data-structure server. It can store sets, queues, hashes, strings, lists, atomic integers and so on.
You can access it across a network in a lab, or across the world. There are clients for bash, C/C++, Ruby, PHP, Python and so on.
So, if you are allocated nodes 1, 2 and 5 for the week, you can just store those in a Redis "list" with LPUSH using the Redis "Command Line Interface"* for bash:
redis-cli lpush VojtaKsNodes 1 2 5
If you are not on the Redis host, add its hostname/IP-address into the command like this:
redis-cli -h 192.168.0.4 lpush VojtaKsNodes 1 2 5
Now, when you want to run a job, get a node with BRPOP. I specify an infinite timeout with the zero at the end, but you could wait a different amount of time:
# Get a node with infinite timeout
node=$(redis-cli brpop VojtaKsNodes 0)
run job on "$node"
# Give node back
redis-cli lpush VojtaKsNodes "$node"
I would:
I have a list of IDS=(ID1 ID2 ID5)
I would make 3 files, one with each IDs.
Run <arguments xargs -P3 programOnGPUFromLockedFile so run 3 processes for each of your argument.
Each of the processes will nonblockingly try to flock the 3 files in a loop, endlessly (ie. you can run more processes then 3, if you wanna).
When they succeed to flock,
they read the ID from the file
run the action on that ID
When they terminate, they will free flock, so the next process may flock the file and use the ID.
Ie. it's a very, very basic mutex locking. There are also other ways you can do it, like with an atomic fifo:
Create a fifo
Spawn one process for each argument you want to run with that will:
Read one line from the fifo
That line will be the ID to run on
Do the job on that ID
Output one line with the ID to the fifo back
And then write one ID per line to the fifo (in 3 separate writes! so that it's hopefully atomic), so 3 processes may start.
wait until all except 3 child processes exit
read 3 lines from fifo
wait until all child processes exit
I have 8 threads in JMeter, which i am executing for every 5 minutes using Task scheduler.
Now i have included 2 threads which want to run for 5 times per day only (ex: at 12am, 5am,10am...)
when the moment comes, the execution shall be 8+2 & remaining time, it shall be only 8 threads.
Is it possible to configure such usecase in Jmeter..
If you're going to use the same .jmx script and want to execute either 8 or 10 "threads" (whatever it is), you can go for:
If Controller - for conditional execution of this or that test elements
__groovy() function to check current time, an example condition which trigger the test at i.e. 5 AM would be:
${__groovy(Calendar.getInstance().get(Calendar.HOUR_OF_DAY) == 5 && Calendar.getInstance().get(Calendar.MINUTE) == 0,)}
I have a shell script that collects some data and send it to destination. Part of the data should be copied every 5 minutes and other every 20 minutes. How can this be achieved in a single script? As of now i'm collecting the data every 5 minutes by scheduling with cron.
Best practice would be to use to separate files with two different cron entries. If you need to reutilize part of your code consider using functions.
If you still want to do it in only one file, you should run it every 5 minutes and on each run check whether or not you should execute the other part (every 20 min) or not.
modulo=$((`date +%_M)` % 20))
//do whatever has to be done every 5min
[...]
//check for modulo of current minute / 20
if [ $modulo -eq 0 ]; then
echo Current minute is `date +%_M)`, must execute part 2
//whatever has to be done every 20min
else
//do nothing
fi;
The reason why the variable modulo is defined in the first line is because what has to be done every 5min can potentially take longer than 1min to execute so by the time it is done minute is no longer 20 but 21. Defining the variable before is a safeguard to overcome this.
How can I get a running count of total number of spot fleet instances?
If the target capacity is 5 instances and sometime after if 2 of them somehow terminates and aws spot fleet automatically relaunch 2 instances to sustain the target capacity and I want to keep count of total instances launched as being 7 ( 5 target + 2 new instances because 2 of them terminated)
I can use this command to get the index value:
curl -s http://169.254.169.254/latest/meta-data/ami-launch-index
However, once the spot fleet is met and if one of the instances terminates for some reason, the spot fleet will relaunch another instance automatically to fulfill the TargetCapacity, but the launch index will revert to "0" vs incrementing by 1 from the very last number assigned.
thanks!
I have trouble implementing the following scenario and Google did't help - may be I am missing something obvious?
Scenario is :
Step 1. 9 sesssions simultaneously running 3 different JDBC queries, i.e
3*Q1,3*Q2,3*Q3 all starting and running at the same time
Clarification: In the beginning of step 1, the following queries will start in 9 different sessions - Q1,Q1,Q1,Q2,Q2,Q2,Q3,Q3,Q3
Step 2. 27 sessions like
above (9 times each query)
Step 3. 54 sessions (18 times each query)
Steps must run sequentially.
To do so:
Step 1)
3 thread groups, each one with 3 threads, each thread group calling a different Qi
Step2)
3 thread groups, each one with 9 threads, each thread group calling a different Qi with scheduler delayed so that it starts after step1 has finished
Step3)
Same as step2 with 18 threads and delayed so that it starts after step 2
But I must say I don't understand why you need such behaviour