I want to run a batch job with two nodes. One node has a GPU and the other one does not. I can get the list of all nodes assigned by:
nodes=$(scontrol show hostnames "$SLURM_JOB_NODELIST")
How can U check in the batch script which node is the one with the GPU or which is the one without GPU?
If you run a command such as scontrol show -d job $SLURM_JOB_ID, the output will contain lines such as
Nodes=nodexxx CPU_IDs=2-5,24-31 Mem=98304 GRES=gpu:1(IDX:2)
Nodes=nodeyyy CPU_IDs=8,14,27 Mem=12000 GRES=
which is probably the information you are looking for.
So you can try something like:
GPUNODE=$(scontrol show -d job $SLURM_JOB_ID | awk '/Nodes=.*GRES=gpu/ {print $1}' | cut -d= -f2)
(the parsing awk ... | cut could certainly be improved but it seems to work)
Related
I have 3000 text files on a linux cluster, each ending in something like
Run Time of 4.533 mins
I am wondering if there was an easy way in Bash scripting to run a loop or something similar on all these files *.txt and to extract the times and find what the maximum was?
Given a file myFile.txt formatted like this
some
content
goes
here
...
Run Time of 4.533 mins
You can get the time with tail -n1 myFile.txt | cut -f4 -d' '
tail -n1 returns the last line
cut -d' ' cut the columns by white space
-f4 select the forth entry
follow #Mark comment to apply it to your 3000 files
I adore taskwarrior, it seems to be the only management app that you can dial in what you decide is the most urgent. Not just due today, or overdue, but a combination of values.
I want to put the top urgency task in a bunch of scripts and widgets (tmux, top bar etc), but this is the best I can do:
task next limit:1 | head -n 4 |tail -n1
Which displays the whole line, due dates, cruft and all, like this:
1 2d H Make widgets 16.5
I know about task _get, the DOM access, but I can't find the way to use it, or any filter.
How can I just display the description of the top task? Thanks!
Make a new report named 'desrc' by adding the following lines to your .taskrc file:
report.desrc.columns = description
report.desrc.labels = Description
report.desrc.sort = urgency-
and then ask for the desrc list:
task rc.verbose: limit:1 desrc
The rc.verbose: removes headlines and everything else, so no need for head and tail.
You can use a one line script to get the ID of the latest, this is assuming that taskwarrior does not change its output format.
task _get $(task next limit:1 | tail -n +4 | head -n 1 | sed 's/^ //' | cut -d ' ' -f1).description
You could sure use cut to get only certain elements of that line, separated by tabs. It's a fast coreutil.
Nothing against creating a new report though, just wanted to mention it.
Awk would also work:
$ task _get $(task next limit:1 | awk 'NR==4{print $1}').description
Ive seen the same question asked on linux and windows but not mac (terminal). Can anyone tell me how to get the current processor utilization in %, so an example output would be 40%. Thanks
This works on a Mac (includes the %):
ps -A -o %cpu | awk '{s+=$1} END {print s "%"}'
To break this down a bit:
ps is the process status tool. Most *nix like operating systems support it. There are a few flags we want to pass to it:
-A means all processes, not just the ones running as you.
-o lets us specify the output we want. In this case, it all we want to the cpu% column of ps's output.
This will get us a list of all of the processes cpu usage, like
0.0
1.3
27.0
0.0
We now need to add up this list to get a final number, so we pipe ps's output to awk. awk is a pretty powerful tool for parsing and operating on text. We just simply add up the numbers, then print out the result, and add a "%" on the end.
Adding up all those CPU % can give a number > 100% (probably multiple cores).
Here's a simpler method, although it comes with some problems:
top -l 2 | grep -E "^CPU"
This gives 2 samples, the first of which is nonsense (because it calculates CPU load between samples).
Also, you need to use RegEx like (\d+\.\d*)% or some string functions to extract values, and add "user" and "sys" values to get the total.
(From How to get CPU utilisation, RAM utilisation in MAC from commandline)
Building on previous answers from #Jon R. and #Rounak D, the following line prints the sum of user and system values, with the added percent. I've have tested this value and I like that it roughly tracks well with the percentages shown in the macOS Activity Monitor.
top -l 2 | grep -E "^CPU" | tail -1 | awk '{ print $3 + $5"%" }'
You can then capture that value in a variable in script like this:
cpu_percent=$(top -l 2 | grep -E "^CPU" | tail -1 | awk '{ print $3 + $5"%" }')
PS: You might also be interested in the output of uptime, which shows system load.
Building upon #Jon R's answer, we can pick up the user CPU utilization through some simple pattern matching
top -l 1 | grep -E "^CPU" | grep -Eo '[^[:space:]]+%' | head -1
And if you want to get rid of the last % symbol as well,
top -l 1 | grep -E "^CPU" | grep -Eo '[^[:space:]]+%' | head -1 | sed s/\%/\/
top -F -R -o cpu
-F Do not calculate statistics on shared libraries, also known as frameworks.
-R Do not traverse and report the memory object map for each process.
-o cpu Order by CPU usage
Answer Source
You can do this.
printf "$(ps axo %cpu | awk '{ sum+=$1 } END { printf "%.1f\n", sum }' | tail -n 1),"
So I want to automate a manual task using shell scripting, but I'm a little lost as to how to parse the output of a few commands. I would be able to this in other languages without a problem, so I'll just explain what I'm going for in psuedo code and provide an example of the cmd output I'm trying to parse.
Example of output:
Chg 2167467 on 2012/02/13 by user1234#filename 'description of submission'
What I need to parse out is '2167467'. So what I want to do is split on spaces and take element 1 to use in another command. The output of my next command looks like this:
Change 2167463 by user1234#filename on 2012/02/13 18:10:15
description of submission
Affected files ...
... //filepath/dir1/dir2/dir3/filename#2298 edit
I need to parse out '//filepath/dir1/dir2/dir3/filename#2298' and use that in another command. Again, what I would do is remove the blank lines from the output, grab the 4th line, and split on space. From there I would grab the 1st element from the split and use it in my next command.
How can I do this in shell scripting? Examples or a point to some tutorials would be great.
Its not clear if you want to use the result from the first command for processing the 2nd command. If that is true, then
targString=$( cmd1 | awk '{print $2}')
command2 | sed -n "/${targString}/{n;n;n;s#.*[/][/]#//#;p;}"
Your example data has 2 different Chg values in it, (2167467, 2167463), so if you just want to process this output in 2 different ways, its even simpler
cmd1 | awk '{print $2}'
cmd2 | sed -n '/Change/{n;n;n;s#.*[/][/]#//#;p;}'
I hope this helps.
I'm not 100% clear on your question, but I would use awk.
http://www.cyberciti.biz/faq/bash-scripting-using-awk/
Your first variable would look something like this
temp="Chg 2167467 on 2012/02/13 by user1234#filename 'description of submission'"
To get the number you want do this:
temp=`echo $temp | cut -f2 -d" "`
Let the output of your second command be saved to a file something like this
command $temp > file.txt
To get what you want from the file you can run this:
temp=`tail -1 file.txt | cut -f2 -d" "`
rm file.txt
The last block of code gets the last nonwhite line of the file and delimits on the second set of white spaces
In my bash script I have the following (for concreteness I preserve the original names;
sometimes people ask about the background etc., and then the original names make more sense):
tail -n +2 Data | while read count phi npa; do
cat Instances/$phi >> $nF
done
That is, the first line of file Data is skipped, and then all lines, which are of
the form "r c p n", are read, and the content of files Instances/p is appended
to file $nF (in the order given by Data).
In typical examples, Data has millions of lines. So perhaps I should write a
C++ application for that. However I wondered whether somebody knew a faster
solution just using bash?
Here I use cut instead of your while loop, but you could re-introduce that if it provides some utility to you. The loop would have to output the phy variable once per iteration.
tail -n +2 Data | cut -d' ' -f 2 | xargs -I{} cat Instances/{} >> $nF
This reduces the number of cat invocations to as few as possible, which should improve efficiency. I also believe that using cut here will improve things further.