I'm trying to create a simple bash script to monitor the following: CPU Utilization, outbound network bandwidth, and inbound network bandwidth. The kicker, I have to use information from /proc/loadavg for the CPU and information from /proc for the bandwidth.
For the CPU Utilization, because it is supposed to be on a short time interval, I can use the first value from /proc/loadavg. Thing is, I'm not sure how to just get that one value so what I have so far is this:
CPU=sudo cat /proc/loadavg | sed 's///'
echo "CPU Utilization: $CPU %"
Where I'm not sure what the sed operation should be. Also I'm not sure how to format what I would get from that so that it would print as "16.5%"
For the bandwidth monitors I haven't the slightest clue of what I could use in /proc to get that sort of information so I'm open to all suggestions.
Load average
You don't need sudo to read /proc/loadavg
In addition, sed is the wrong tool here, try using cut, for example:
$ cut -d' ' -f1 < /proc/loadavg
0.04
cut will cut lines by a delimiter (given with -d), in this case a space, and you can then use -f to select a field, in this case the first one.
Now, converting it to percentages is actually fairly meaningless, since you'll often end up above 100% (see comment below), I've seen load averages in excess of 50 (that would be 5000% percent?).
In all my years of UNIX/Linux experience, I can't recall ever seeing the load average being expressed as a percentage, and if I would encounter such a thing, I would find it very odd.
But if you really want to (you don't!), just multiply by 100 with dc, like so:
$ dc -e "`cut -d' ' -f1 < /proc/loadavg` 100 * p"
29.00
For the CPU Utilization, because it is supposed to be on a short time
interval, I can use the first value from /proc/loadavg.
The load average is not the same thing as CPU usage.
A load average of 1 means there is one process waiting for something (usually the CPU or disk).
A load average of 2 means there are two processes waiting.
A load average of 0.5 (over the last minute), can mean that for 30 seconds, there was one process waiting, and for 30 seconds, there were no processes waiting. It can also mean that for 15 seconds there were two processes waiting, and for 45 seconds there were no processes waiting. The keyword here is average.
If you want to get the CPU utilization, then this is probably the most portable way:
$ top -bn2 | grep "Cpu(s)" | \
tail -n1 | \
sed "s/.*, *\([0-9.]*\)%* id.*/\1/" | \
awk '{print 100 - $1"%"}'
Note you need to use -n2 to get fairly accurate results.
I've adapted this from this answer, which also lists some other possibilities, some simpler, but most tools mentioned aren't installed by default on most systems.
Network
For the bandwidth monitors I haven't the slightest clue of what I
could use in /proc to get that sort of information so I'm open to all
suggestions.
You can use the output of ifconfig, for example, on my system:
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.178.28 netmask 255.255.255.0 broadcast 192.168.178.255
inet6 2001:980:82cd:1:20c:29ff:fe9e:c84b prefixlen 128 scopeid 0x0<global>
inet6 fe80::20c:29ff:fe9e:c84b prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:9e:c8:4b txqueuelen 1000 (Ethernet)
RX packets 45891 bytes 36176865 (34.5 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 20802 bytes 2603821 (2.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
It's the RX packets & TX packets we want. Let's extract just those values:
$ ifconfig ens33 | grep -E '(R|T)X packets' | grep -Eo '\([0-9].*\)' | tr -d '()'
34.5 MiB
2.5 MiB
First we grep all the lines starting with RX or TX
With those lines, we then grep for a parenthesis \(, followed by a number [0-9], followed by any characters .*, followed by a closing parenthesis \). With the -o flag we show only the matching part, instead of the whole line.
With tr, we remove the unwanted parentheses.
This should be what you want. If you want to get a number of bytes, you can use a different grep pattern in the second grep. I'll leave it as an exercise to you what exactly that is.
Here's how you can print the first number output by cat /proc/loadavg as a percent value (but see #Carpetsmoker's caveat regarding whether that makes sense), rounded to 1 decimal place:
printf "1-minute load average: %.1f%%\n" \
$(bc <<<"$(cut -d ' ' -f 1 /proc/loadavg) * 100")
Related
I would like to write a bash script that writes the current CPU utilization to a file "logfile". I am using Intel® Core™ i7-4500U CPU # 1.80GHz × 4 and Ubuntu 15.10.
I have seen similar questions asked already in this forum, however not all of my questions were answered to 100 percent. By my research I came up with two possible ways of achieving my goal. First one is
mpstat | grep "all" | awk '{ print $3 + $5; }' >> logfile
(adding user CPU and system CPU) and my second candidate is
mpstat | grep "all" | awk '{ print 100 - $12; }' >> logfile
(100 - %idle CPU). Which one of those two is the right one for me if I am interested in the total CPU utilization (so all components that count in some form as CPU should be included).
Another question: By what I have learned by reading other threads, I think my second candidate
mpstat | grep "all" | awk '{ print 100 - $12; }' >> logfile
should be quite accurate. However, when I open the "System Monitor" and monitor the "CPU History" I observe significantly different CPU utilization. Another thing is that the values in the System Monitor are very dynamic (CPU varies between 4% and 18%) whereas over the same period the outcome of the second command remains almost constant. Has someone an explanation for that?
Many thanks for all comments!
This happens because mpstat's first line shows an average value calculated since the system booted (which will be much more "stable" - will tend to change less and less as time goes by ).
Quote from mpstat man page:
The interval parameter specifies the amount of time in seconds
between each report. A value of 0 (or no parameters at all)
indicates that processors statistics are to be reported for the time
since system startup (boot).
If you add an interval parameter, you will start to get back live numbers, which should more closely match your System Monitor output (try executing mpstat 1 vs. the plain mpstat).
Therefore, this Bash line should do the trick:
mpstat 1 1 | grep "all" | awk '{ print 100 - $NF; exit; }' >> logfile
and, to do it without grep (saving the extra process spawn):
mpstat 1 1 | awk '/all/{ print 100 - $NF; exit; }' >> logfile
(changed $12 to $NF for the case when the first line has a time and shifts the arguments over; with $NF we consistently get the last value, which is the idle value)
How can I get interrupts per sec during a test and record the information to a file? Do I use itop or what would I use to collect the information? Script in bash.
Example of what I want to do:
# ./itop -t -f eth0
Output
Device (IRQ) CPU0 CPU1 IRQs/Second
TOTAL
eth0 ( 59): 0 0 20
I would like to report the number under IRQs/Second TOTAL no matter how many CPUs. SO pretty much I would like to strip everything except what's under "IRQs a second total" like below example.
Output for example above should be simply:
# 20
pipe out to file
You can use something like
$ awk '!/^Device/{print $NF}' inputFile
20
I need to write a bash script that would consume a maximum of RAM of my ESXi and potentially generate a memory shortage.
I already checked here and try to run the given script several times so that I can consume more thant 500Mb of RAM.
However I get a "sh: out of memory" error (of course) and I'd like to know if there is any possibility to configuration the amount of memory allocated to my shell ?
Note1 : Another requirement is that I cannot enter a VM a run a greedy task.
Note2 : I tried to script the creation of greedy new VMs with huge RAM however I cannot get to ESXi state where there is a shortage of memory.
Note3 : I cannot use a C compiler and I only have very limited python library.
Thank you in advance for your help :)
From an earlier answer of mine: https://unix.stackexchange.com/a/254976/30731
If you have basic GNU tools (head and tail) or BusyBox on Linux, you can do this to fill a certain amount of free memory:
</dev/zero head -c BYTES | tail
# Protip: use $((1024**3*7)) to calculate 7GiB easily
</dev/zero head -c $((1024**3*7)) | tail
This works because tail needs to keep the current line in memory, in case it turns out to be the last line. The line, read from /dev/zero which outputs only null bytes and no newlines, will be infinitely long, but is limited by head to BYTES bytes, thus tail will use only that much memory. For a more precise amount, you will need to check how much RAM head and tail itself use on your system and subtract that.
To just quickly run out of RAM completely, you can remove the limiting head part:
tail /dev/zero
If you want to also add a duration, this can be done quite easily in bash (will not work in sh):
cat <( </dev/zero head -c BYTES) <(sleep SECONDS) | tail
The <(command) thing seems to be little known but is often extremely useful, more info on it here: http://tldp.org/LDP/abs/html/process-sub.html
Then for the use of cat: cat will wait for inputs to complete until exiting, and by keeping one of the pipes open, it will keep tail alive.
If you have pv and want to slowly increase RAM use:
</dev/zero head -c BYTES | pv -L BYTES_PER_SEC | tail
For example:
</dev/zero head -c $((1024**3)) | pv -L $((1024**2)) | tail
Will use up to a gigabyte at a rate of a megabyte per second. As an added bonus, pv will show you the current rate of use and the total use so far. Of course this can also be done with previous variants:
</dev/zero head -c BYTES | pv | tail
Just inserting the | pv | part will show you the current status (throughput and total by default).
Credits to falstaff for contributing a variant that is even simpler and more broadly compatible (like with BusyBox).
Bash has the ulimit command, which can be used to set the size of the virtual memory which may be used by a bash process and the processes started by it.
There are two limits, a hard limit and a soft limit. You can lower both limits, but only raise the soft limit up to the hard limit.
ulimit -S -v unlimited sets the virtual memory size to unlimited (if the hard limit allows this).
If there is a hard limit set (see ulimit -H -v), check any initialisation scripts for lines setting it.
grep -i -A 5 -B 5 'db_pd.Clients' eightygigsfile.sql
This has been running for an hour on a fairly powerful linux server which is otherwise not overloaded.
Any alternative to grep? Anything about my syntax that can be improved, (egrep,fgrep better?)
The file is actually in a directory which is shared with a mount to another server but the actual diskspace is local so that shouldn't make any difference?
the grep is grabbing up to 93% CPU
Here are a few options:
1) Prefix your grep command with LC_ALL=C to use the C locale instead of UTF-8.
2) Use fgrep because you're searching for a fixed string, not a regular expression.
3) Remove the -i option, if you don't need it.
So your command becomes:
LC_ALL=C fgrep -A 5 -B 5 'db_pd.Clients' eightygigsfile.sql
It will also be faster if you copy your file to RAM disk.
If you have a multicore CPU, I would really recommend GNU parallel. To grep a big file in parallel use:
< eightygigsfile.sql parallel --pipe grep -i -C 5 'db_pd.Clients'
Depending on your disks and CPUs it may be faster to read larger blocks:
< eightygigsfile.sql parallel --pipe --block 10M grep -i -C 5 'db_pd.Clients'
It's not entirely clear from you question, but other options for grep include:
Dropping the -i flag.
Using the -F flag for a fixed string
Disabling NLS with LANG=C
Setting a max number of matches with the -m flag.
Some trivial improvement:
Remove the -i option, if you can, case insensitive is quite slow.
Replace the . by \.
A single point is the regex symbol to match any character, which is also slow
Two lines of attack:
are you sure, you need the -i, or do you habe a possibility to get rid of it?
Do you have more cores to play with? grep is single-threaded, so you might want to start more of them at different offsets.
< eightygigsfile.sql parallel -k -j120% -n10 -m grep -F -i -C 5 'db_pd.Clients'
If you need to search for multiple strings, grep -f strings.txt saves a ton of time. The above is a translation of something that I am currently testing. the -j and -n option value seemed to work best for my use case. The -F grep also made a big difference.
Try ripgrep
It provides much better results compared to grep.
All the above answers were great. What really did help me on my 111GB file was using the LC_ALL=C fgrep -m < maxnum > fixed_string filename.
However, sometimes there may be 0 or more repeating patterns, in which case calculating the maxnum isn't possible. The workaround is to use the start and end patterns for the event(s) you are trying to process, and then work on the line numbers between them. Like so -
startline=$(grep -n -m 1 "$start_pattern" file|awk -F":" {'print $1'})
endline=$(grep -n -m 1 "$end_pattern" file |awk -F":" {'print $1'})
logs=$(tail -n +$startline file |head -n $(($endline - $startline + 1)))
Then work on this subset of logs!
hmm…… what speeds do you need ? i created a synthetic 77.6 GB file with nearly 525 mn rows with plenty of unicode :
rows = 524759550. | UTF8 chars = 54008311367. | bytes = 83332269969.
and randomly selected rows at an avg. rate of 1 every 3^5, using rand() not just NR % 243, to place the string db_pd.Clients at a random position in the middle of the existing text, totaling 2.16 mn rows where the regex pattern hits
rows = 2160088. | UTF8 chars = 42286394. | bytes = 42286394.
% dtp; pvE0 < testfile_gigantic_001.txt|
mawk2 '
_^(_<_)<NF { print (__=NR-(_+=(_^=_<_)+(++_)))<!_\
?_~_:__,++__+_+_ }' FS='db_pd[.]Clients' OFS=','
in0: 77.6GiB 0:00:59 [1.31GiB/s] [1.31GiB/s] [===>] 100%
out9: 40.3MiB 0:00:59 [ 699KiB/s] [ 699KiB/s] [ <=> ]
524755459,524755470
524756132,524756143
524756326,524756337
524756548,524756559
524756782,524756793
524756998,524757009
524757361,524757372
And mawk2 took just 59 seconds to extract out a list of row ranges it needs. From there it should be relatively trivial. Some overlapping may exist.
At throughput rates of 1.3GiB/s, as seen above calculated by pv, it might even be detrimental to use utils like parallel to split the tasks.
I am looking for a command(s) that could cause my Linux computer to use lot of RAM ?
Any pointers?
I want to run that in the back ground and do some other task that needs already high RAM usage.
Thanks!
That one will cause growing memory usage by bash:
while read f; do arr[$((i++))]=$f; done < /dev/urandom
Another hard way might be (be careful):
while read f; do arr="$arr:$arr:$arr:$arr:$f"; done < /dev/urandom
Softer version:
while read f; do arr="$arr:$f"; done < /dev/urandom
to reclaim parts of the memory call:
unset arr
You could also mix it with a fork bomb... but I would avoid it
The current answer will try to use more RAM until it's simply completely out of RAM. For a more controlled approach, if you have basic GNU tools (head and tail) or BusyBox on Linux, you can do this to fill a certain amount of free memory:
</dev/zero head -c BYTES | tail
</dev/zero head -c 5000m | tail #~5GB, portable
</dev/zero head -c 5G | tail #5GiB on GNU (not busybox)
cat /dev/zero | head -c 5G | tail #Easier notation; does the same thing
This works because tail needs to keep the current line in memory, in case it turns out to be the last line. The line, read from /dev/zero which outputs only null bytes and no newlines, will be infinitely long, but is limited by head to BYTES bytes, thus tail will use only that much memory. For a more precise amount, you will need to check how much RAM head and tail itself use on your system and subtract that.
To just quickly run out of RAM completely, you can remove the limiting head part:
tail /dev/zero
If you want to also add a duration, this can be done quite easily in bash (will not work in sh):
cat <( </dev/zero head -c 500m) <(sleep SECONDS) | tail
The <(command) thing seems to be little known but is often extremely useful, more info on it here: http://tldp.org/LDP/abs/html/process-sub.html
The cat command will wait for inputs to complete until exiting, and by keeping one of the pipes open, it will keep tail alive.
If you have pv and want to slowly increase RAM use:
</dev/zero head -c TOTAL | pv -L BYTES_PER_SEC | tail
</dev/zero head -c 1000m | pv -L 10m | tail
The latter will use up to one gigabyte at a rate of ten megabytes per second. As an added bonus, pv will show the current rate of use and the total use so far. Of course this can also be done with previous variants:
</dev/zero head -c 500m | pv | tail
Just inserting the | pv | part will show you the current status (throughput and total by default).
If you do not have a /dev/zero device, the standard yes and tr tools might substitute: yes | tr \\n x | head -c BYTES | tail (yes outputs an infinite amount of "yes"es, tr substitutes the newline such that everything becomes one huge line and tail needs to keep all that in memory).
Another, simpler alternative is using dd: dd if=/dev/zero bs=1G of=/dev/null uses 1GB of memory on GNU and BusyBox, but also 100% CPU on one core.
Finally, if your head does not accept a suffix, you can calculate an amount of bytes inline, for example 50 megabytes: head -c $((1024*1024*50))
Cross-posted from my answer on the Unix StackExchange
ulimit -m can be used to impose an artificial memory limit on a process.
Use this at your own risk :
:(){ :|:& };:
=> Explosion in your ram