I'm trying to write a script that monitors real time CPU% load in AIX 6.1-servers by process(PID), and have been searching for this both in IBMs documentation and all over stackoverflow.
I only find examples of people using, for example
ps aux
Needless to say, thats not what I need, since it only monitors how much CPU% the process has been using over the session time, which in my case is quite long.
The information I need is contained in topas and nmon, but I don't know how to grab a snapshot of this information for each individual moment.
top
Does not exist in AIX systems.
Solved this by making a script that generates 30second tprof logs and iterates through them adding up the process threads by PID and reaching a sum that equals more or less a real-time CPU load% process list.
Here is a function I use to search and grab CPU load from nmon data log
function fetch_UARG_process_by_pid ()
{
#Check_Argument $1 $2
#parameter initialization
filesource=$1
cpuvalue=$2
readarray -t X <<< "$(grep TOP $filesource)"
length=${#X[#]}
#echo " this is the length of my array : $length"
#you have to start from 2 to avoid the first lines that describe the content of the file
#TOP,%CPU Utilisation
#TOP,+PID,Time,%CPU,%Usr, Sys,Size,ResSet,ResText,ResData,ShdLib,MinorFault,MajorFault,Command
for ((i = 2; i != length; i++)); do
echo ${X[i]} | awk -F "," '{print $2 , $4}' | while read processid n
do
if (( $(echo "$n > $cpuvalue " |bc -l) ));
then
echo "the value of CPU usage is: $n"
echo "the Linux PID is : $processid "
echo "And the short desciption of the process:"
echo ${X[i]} | awk -F "," '{print $14}'
echo -e "And the long desciption of the process:"
grep UARG $1 | grep $processid | awk -F "," '{print $5}'
echo -e "\n"
fi
done
done
}
Related
Log file abc.log
PID:6543 ……
…………………
PID:4325 ……
……………………
PID:6543 ……
Log file xyz.log
PID:8888 ……
…………………
PID:9992 ……
……………………
PID:6543 ……
Note: The PID numbers can repeat in a file. And also one PID number can appear in multiple log files.
This question is asked in an interview today that I have to return output with each PID number and the count of each PID number logged today.
Here is the script that I have written. Can you confirm if that would work or not. The interviewer didn't said if my answer is correct or not. Can someone review this for me. They want me to print each unique PID:number and its count with tab space like PID:5674 10
— If today’s and previous day’s log files are in same folder
#!/bin/ksh
cd /A/B/
for a in `ls -lrt | grep "Mar 24" | awk '{print $9}'`; — list of files generated today
do
grep "^PID:" $a | cut -d " " f1 >> /tmp/abc.log — saving first column which look like PID:23456
done
for b in `cat /tmp/abc.log | sort -u`;
do
x=grep $b /tmp/abc.log | grep -v grep | wc;
echo $b" "$x — will print like PID:23456 56(count)
done
#!/bin/ksh
— If today’s log files are in different folder
cd /A/B/
for a in `ls /A/B/*.log`
do
grep "^PID:" $a | cut -d " " f1 >> /tmp/abc.log
done
for b in `cat /tmp/abc.log | sort -u`;
do
x=grep $b /tmp/abc.log | grep -v grep | wc;
echo $b" "$x
done
You could use awk
grep -h 'PID:' *.log > all_pids.log
#put only PID lines to a file.This can also be done
#with pattern in awk and multiple files. I'm just separating it for clarity
awk '
{ a[$1]++ } #increments by 1 for corresponding PID
END {
for (i in a) {
printf "%s %s\n", i, a[i];
}
}
' all_pids.log
Code sample :
declare -i a=1
echo "The number of NMON instances running in Performance VM"
ps -ef | grep nmon | awk '{ print $2 }' | wc -l
echo "---------------------------------------------"
num2= ps -ef | grep nmon | awk '{ print $2 }' | wc -l
#num1=${num2%%.*}
#num2 = $(ps -ef | grep nmon | awk '{ print $2 }' | wc -l)
echo "---------------------------------------------"
echo "${num2}"
while [ "$a" -lt "$num2" ]
do
kill -USR2 $(ps -ef | grep nmon | awk '{ print $2 }' | head -1)
a=`expr $a + 1`
done
In the Output i am getting the following error
[: : integer expression expected
in the debug it shows
++ '[' 1 -lt '' ']'
that num2 is empty but when i echo the num2 value i am getting the value correctly.
Output:
The number of NMON instances running in Performance VM
1
1
thanks in advance
The 1 you see in the output is not from echo "${num2}". Like the diagnostics already tell you, this variable is empty.
The general syntax of shell scripts is
[ variable=value ...] command parameters ...
which will assign value to variable for the duration of command, then restore its original value. So the pipeline you are running temporarily sets num2 to the empty string (which apparently it already contained anyway), then runs the pipeline without storing the output anywhere (such as, I imagine you expected, in num2).
Here is a fixed version of your script, with the additional change that the Awk scripts handle stuff you used grep and head and wc for. Because the functionality of these commands is easily replaced within Awk, using external utilities is doubtful (especially so for grep which really is useless when you just run it as a preprocessor for a simple Awk script).
countnmon () {
ps -ef | awk '/[n]mon/ { ++n } END { print n }'
}
declare -i a=1
echo "The number of NMON instances running in Performance VM"
countnmon
echo "---------------------------------------------"
num2=$(countnmon)
#num1=${num2%%.*}
#num2 = $(countnmon)
echo "---------------------------------------------"
echo "${num2}"
while [ "$a" -lt "$num2" ]
do
kill -USR2 $(ps -ef | awk '/[n]mon/ { print $2; exit }')
a=`expr $a + 1`
done
The repeated code could be refactored even further to avoid all code duplication but that will somewhat hamper the readability of this simple script so I have not done that.
Whitespaces matter in bash.
The syntax for command execution is:
command arg1 arg2 ...
So,
var = value # command: var, arg1: =, arg2: value
There's are two exceptions to this rule
exporting variables to an executed command (the variables vanish after the command finishes):
var1=value1 var2=value2 .. command arg1 arg2 ...
assigning a variable (you want this one):
var=value
I was experimenting with a bash script that would recursively fork and call itself. The terminating condition was subtle and I got it a wrong a few times, the result being a script that called itself ad infinitum. What's a safe way to sandbox a script like this while debugging it so that every time there's a mistake, I don't have to deal with stopping the infinite tower that fills up the process table?
You could use ulimit
ulimit -u 20
Will limit the maximum number of processes runned by your user
You could simply count the numbers of processes with your script name and terminate if the number gets too high.
This blog post introduces a function to achieve this:
count_process(){
return $(ps -ef | grep -v grep | grep -c $1)
}
Explanation (taken from blog post):
ps -ef will return a list of all running processes (in detail),
and the process list will then be filtered first to exclude any instances of grep
and second to count the processes specified with $1
For re-usability the author provides the function as little script:
#!/bin/sh
#
# /usr/bin/processCount
# Source: http://samcaldwell.net/index.php/technical-articles/3-how-to-articles/68-how-do-i-count-the-number-of-linux-instances-of-a-given-process-using-a-bash-script
#
[ -z $1 ] && {
echo " "
echo "Missing expected input."
echo " "
echo "USAGE:"
echo " "
echo " $0 <executable file>"
echo " "
echo " NOTE: When executing this script use the path and filename of the"
echo " program. Using only the process name can potentially return inaccurate"
echo " results."
echo " "
exit 1
}
echo $(ps -ef | grep -v grep | grep -v $0 | grep -c $1)
#script ends here.
I have this code:
total=0;
ps -u $(whoami) --no-headers | awk {'print $1'} | while read line;
do vrednost=$(pmap $line | tail -n1 | column -t | cut -d" " -f3 | tr "K" " ");
total=$(( vrednost + total ))
echo $total
done
echo total: $total
As you can see, my code sums usage of all my processes. When I echo my total every time in while, it is working ok, but at the end... When i want total to be a value (echo total: $total) it is still zero. but before (in while) has right value.
BASH FAQ #24: "I set variables in a loop that's in a pipeline. Why do they disappear after the loop terminates? Or, why can't I pipe data to read?"
#!/bin/bash
while read ...
do
...
done < <(ps ...)
Okay, pick and choose. You can either do it in BASH or AWK, but don't do both. You've seen a BASH example, here's an AWK example:
ps -e -o user -o vsz | awk -v USER="$(whoami)" '
BEGIN {TOTAL = 0}
END {print "Total is " TOTAL}
{
if ($1 == USER) {
TOTAL += $2
}
}
'
Awk is like a programming language that assumes a loop (like perl -n) and processes each line in the file. Each field (normally separated by whitespace) is given a $ variable. The first is $1, the second is $2, etc.
The -v option allows me to define an awk variable (in this case USER) before I run awk.
The BEGIN line is what I want to do before I run my awk script. In this case, initialize TOTAL to zero. (NOTE: This really isn't necessary since undefined variables automatically are given a value of zero). The END line is what I want to do afterwards. In this case, print out my total.
So, if the first field ($1) is equal to my user, I'll add the second field (the vsize) to my total.
All Awk programs are surrounded by {...} and they usually have single quotes around them to prevent shell interpolation of $1, etc.
Try this
#!/bin/bash
total=0;
for line in `ps -u $(whoami) --no-headers | awk {'print $1'}`;
do
vrednost=$(pmap $line | tail -n1 | column -t | cut -d" " -f3 | tr "K" " ");
total=$(( $vrednost + $total ))
echo $total
done
echo "total: $total"
Ignacio's answer is fine, but process substitution is not portable. And there is a simpler solution. You need to echo the total in the same subshell in which it is calculated:
... | { while read line; do ...; done; echo total: $total; }
Let's cut down on the number of extra processes you need :)
declare -i total=0
for size in $( ps -u $(whoami) --no-header -o vsz ); do
total+=$size
done
echo $total
First, use various options for ps to generate the desired list of process sizes, in kilobytes. Iterate over that list using a bash for-loop, keeping a running total in a parameter declared with the 'integer' attribute for easy arithmetic. The desired sum is now in total, ready for whatever use you need. The sum includes the memory used by the ps process itself.
Using while (Dennis' suggestion) and avoiding process substitution (William's suggestion):
ps -u $(whoami) --no-header -o vsz | {
while read -r var; do
((total+=$var))
done
echo $total
}
(For real one-liner, here's a dc command that I borrowed from https://stackoverflow.com/a/453290/1126841:
ps -u $(whoami) --no-header -o vsz | dc -f - -e '[+z1<r]srz1<rp'
This sum includes the memory used by the ps and dc commands themselves.)
I have a problem that I can't solve, so I've come to you.
I need to write a program that will read all processes and a program must sort them by users and for each user it must display how much of a memory is used.
For example:
user1: 120MB
user2: 300MB
user3: 50MB
total: 470MB
I was thinking to do this with ps aux command and then get out pid and user with awk command. Then with pmap I just need to get total memory usage of a process.
it's just a little update, users are automatically selected
#!/bin/bash
function mem_per_user {
# take username as only parameter
local user=$1
# get all pid's of a specific user
# you may elaborate the if statement in awk obey your own rules
pids=`ps aux | awk -v username=$user '{if ($1 == username) {print $2}}'`
local totalmem=0
for pid in $pids
do
mem=`pmap $pid | tail -1 | \
awk '{pos = match($2, /([0-9]*)K/, mem); if (pos > 0) print mem[1]}'`
# when variable properly set
if [ ! -z $mem ]
then
totalmem=$(( totalmem + $mem))
fi
done
echo $totalmem
}
total_mem=0
for username in `ps aux | awk '{ print $1 }' | tail -n +2 | sort | uniq`
do
per_user_memory=0
per_user_memory=$(mem_per_user $username)
if [ "$per_user_memory" -gt 0 ]
then
total_mem=$(( $total_mem + $per_user_memory))
echo "$username: $per_user_memory KB"
fi
done
echo "Total: $total_mem KB"
Try this script, which may solve your problem:
#!/bin/bash
function mem_per_user {
# take username as only parameter
local user=$1
# get all pid's of a specific user
# you may elaborate the if statement in awk obey your own rules
pids=`ps aux | awk -v username=$user '{if ($1 == username) {print $2}}'`
local totalmem=0
for pid in $pids
do
mem=`pmap $pid | tail -1 | \
awk '{pos = match($2, /([0-9]*)K/, mem); if (pos > 0) print mem[1]}'`
# when variable properly set
if [ ! -z $mem ]
then
totalmem=$(( totalmem + $mem))
fi
done
echo $totalmem
}
total_mem=0
for i in `seq 1 $#`
do
per_user_memory=0
eval username=\$$i
per_user_memory=$(mem_per_user $username)
total_mem=$(( $total_mem + $per_user_memory))
echo "$username: $per_user_memory KB"
done
echo "Total: $total_mem KB"
Best regards!
You can access the shell commands in python using the subprocess module. It allows you to spawn subprocesses and connect to the out/in/error. You can execute the ps -aux command and parse the output in python.
check out the docs here
Here is my version. I think that Tim's version is not working correctly, the values in KB are too large. I think the RSS column from pmap -x command should be used to give more accurate value. But do note that you can't always get correct values because processes can share memmory. Read this A way to determine a process's "real" memory usage, i.e. private dirty RSS?
#!/bin/bash
if [ "$(id -u)" != "0" ]; then
echo "WARNING: you have to run as root if you want to see all users"
fi
echo "Printing only users that current memmory usage > 0 Kilobytes "
all=0
for username in `ps aux | awk '{ print $1 }' | tail -n +2 | sort | uniq`
do
pids=`ps aux | grep $username | awk -F" " '{print $2}'`
total_memory=0
for pid in $pids
do
process_mem=`pmap -x $pid | tail -1 | awk -F" " '{print $4}'`
if [ ! -z $process_mem ]
then #don't try to add if string has no length
total_memory=$((total_memory+$process_mem))
fi
done
#print only those that use any memmory
if [ $total_memory -gt 0 ]
then
total_memory=$((total_memory/(1024)))
echo "$username : $total_memory MB"
all=$((all+$total_memory))
fi
done
echo "----------------------------------------"
echo "Total: $all MB"
echo "WARNING: Use at your own risk"