Count string occurences in file - shell

I am trying to count the occurence of a particular line in a file using shell script. The content of the file is:
Mon Dec 23 06:21:00 2019 [pid 3294] [ftpuser] FTP response: Client "127.0.0.1", "230 Login successfull."
Mon Dec 23 06:21:00 2019 [pid 3294] [ftpuser] FTP response: Client "127.0.0.1", "230 Login successfull."
Mon Dec 23 06:21:00 2019 [pid 3294] [ftpuser] FTP response: Client "127.0.0.1", "230 Login successfull."
Mon Dec 23 06:21:00 2019 [pid 3294] [ftpuser] FTP response: Client "127.0.0.1", "230 Login successfull."
The shell script code is:
file="vsftpd1.log"
count=0
while read str; do
echo $str
if [[ $str == *"230 Login successfull."* ]];
then
count=$((count+1))
fi
done < $file
echo $count
The output of the last line should be four. I have noticed that if condition is executing only one time. But it should run four times. Please help me to find mistake in the code, if any.
Thankxx in advance!

You could simplify that with grep (match occurences) and its -c (count occurences).
#!/bin/sh
file=vsftpd1.log
grep -c '230 Login successfull.' "$file"
If you need to capture the number into a variable:
count=$(grep -c '230 Login successfull.' "$file")
echo $count

That entire (rather inefficient) snippet of code can be replaced with the much simpler:
count=$(grep -c '230 Login successfull.' vsftpd1.log)
echo ${count}
As more explanation, grep -c returns the line count of the lines that match your search pattern. There is no need for a bash-based loop which reads one line at a time and checks the pattern - this is likely to be much faster in the grep executable.
The following transcript shows this in action:
pax:~$ grep '230 Login successfull.' qq.in
Mon Dec 23 06:21:00 2019 [pid 3294] [ftpuser] FTP response: Client "127.0.0.1", "230 Login successfull."
Mon Dec 23 06:21:00 2019 [pid 3294] [ftpuser] FTP response: Client "127.0.0.1", "230 Login successfull."
Mon Dec 23 06:21:00 2019 [pid 3294] [ftpuser] FTP response: Client "127.0.0.1", "230 Login successfull."
Mon Dec 23 06:21:00 2019 [pid 3294] [ftpuser] FTP response: Client "127.0.0.1", "230 Login successfull."
pax:~$ count=$(grep -c '230 Login successfull.' qq.in) ; echo ${count}
4

Related

unbound variable when run a crontab task in mac os x

when I run script in mac os x like this:
*/1 * * * * /Users/dolphin/Library/"Mobile Documents"/com~apple~CloudDocs/Document/source/dolphin/dolphin-scripts/bash/cron/latex_compile_alive_monitor.sh >> /Users/dolphin/shell.log
the output is:
From dolphin#dolphins-MacBook-Pro.local Tue Jan 21 20:47:01 2020
Return-Path: <dolphin#dolphins-MacBook-Pro.local>
X-Original-To: dolphin
Delivered-To: dolphin#dolphins-MacBook-Pro.local
Received: by dolphins-MacBook-Pro.local (Postfix, from userid 501)
id 7430417C4C19; Tue, 21 Jan 2020 20:47:00 +0800 (CST)
From: dolphin#dolphins-MacBook-Pro.local (Cron Daemon)
To: dolphin#dolphins-MacBook-Pro.local
Subject: Cron <dolphin#dolphins-MacBook-Pro> /Users/dolphin/Library/"Mobile Documents"/com~apple~CloudDocs/Document/source/dolphin/dolphin-scripts/bash/cron/latex_compile_alive_monitor.sh >> /Users/dolphin/shell.log
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=dolphin>
X-Cron-Env: <USER=dolphin>
X-Cron-Env: <HOME=/Users/dolphin>
Message-Id: <20200121124701.7430417C4C19#dolphins-MacBook-Pro.local>
Date: Tue, 21 Jan 2020 20:47:00 +0800 (CST)
+ BOOK_PATH='/Users/dolphin/Library/Mobile Documents/com~apple~CloudDocs/Document/source/dolphin/summary/'
+ COMMAND='/Library/TeX/texbin/latexmk -pdfxe -pvc -xelatex -interaction=nonstopmode '
+ PROCESS_NAME_KEYWORDS_MAP=(["dolphin-book-2020.tex"]="${COMMAND} ./dolphin-book-2020/dolphin-book-2020.tex" ["the-book-of-mine.tex"]="${COMMAND} ./the-books-of-mine/the-book-of-mine.tex" ["kubelet-learn.tex"]="${COMMAND} ./kubelet-learn/kubelet-learn.tex")
/Users/dolphin/Library/Mobile Documents/com~apple~CloudDocs/Document/source/dolphin/dolphin-scripts/bash/cron/latex_compile_alive_monitor.sh: line 15: dolphin: unbound variable
which variable was unboud? COMMAND? I am already defined in script.this is my script:
#!/usr/bin/env bash
# 当使用未初始化的变量时,程序自动退出
set -u
# 当任何一行命令执行失败时,自动退出脚本
set -e
# 在运行结果之前,先输出执行的那一行命令
set -x
BOOK_PATH="/Users/dolphin/Library/Mobile Documents/com~apple~CloudDocs/Document/source/dolphin/summary/"
COMMAND="/Library/TeX/texbin/latexmk -pdfxe -pvc -xelatex -interaction=nonstopmode "
declare -A PROCESS_NAME_KEYWORDS_MAP=(
["dolphin-book-2020.tex"]="${COMMAND} ./dolphin-book-2020/dolphin-book-2020.tex"
["the-book-of-mine.tex"]="${COMMAND} ./the-books-of-mine/the-book-of-mine.tex"
["kubelet-learn.tex"]="${COMMAND} ./kubelet-learn/kubelet-learn.tex"
)
cd "${BOOK_PATH}"
for key in ${!PROCESS_NAME_KEYWORDS_MAP[#]}
do
PID_COUNT=`ps -ef | grep "${key}" | grep -v "grep" | wc -l`
if [[ ${PID_COUNT} -lt 1 ]]; then
nohup `${PROCESS_NAME_KEYWORDS_MAP[${key}]}` &
else
echo "process already exists..."
fi
done
maybe remove the space in the end of COMMAND

Updated Ubuntu and pcntl_fork stopped working (php)

Things that changed recently on my server:
I'm almost sure it's because the dist-upgrade.(few days ago)
I added a new user and added him a library in var/www/html/banana.
so it might be from that too (?) - (2 weeks ago)
Tried installing FastCGI without any success - but this didn't disrupt any regular processing and flow. (2 months ago)
I usually run API queries from my PHP code using forking, and in some point it stopped working for me (it does work, but when getting to heavy query results it stopps).
error.log:
[Sun Aug 28 12:15:03.201994 2016] [:notice] [pid 1882] FastCGI: process manager initialized (pid 1882)
[Sun Aug 28 12:15:03.278176 2016] [mpm_prefork:notice] [pid 1879] AH00163: Apache/2.4.18 (Ubuntu) mod_fastcgi/mod_fastcgi-SNAP-0910052141 configured -- resuming normal operations
running cat /var/mail/root outputs:
From root#banana Sun Aug 28 12:39:01 2016
Return-Path: <root#banana>
X-Original-To: root
Delivered-To: root#banana
Received: by banana (Postfix, from userid 0)
id ABC281005BA; Sun, 28 Aug 2016 12:39:01 +0300 (IDT)
From: root#banana (Cron Daemon)
To: root#banana
Subject: Cron <root#banana> [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
Message-Id: <20160828093901.ABC281005BA#YHserver>
Date: Sun, 28 Aug 2016 12:39:01 +0300 (IDT)
Can someone help me debug the problem better and solve it?
Running this script gets true results:
<?php
echo "Is fork? <br/>";
var_dump (extension_loaded('pcntl'));
echo "<br><br> more checks: <br>";
$supports = array();
if (function_exists("pcntl_fork")) $supports[] = "ispcntl";
echo implode(",", $supports);
for ($i = 1; $i <= 5; ++$i) {
$pid = pcntl_fork();
if (!$pid) {
sleep(1);
print "In child $i\n";
exit;
}
}
?>
EDIT: I Tried running that same script on the server without forking and I got all the results right (after waiting a lot and getting my website stuck for a while..)

Convert epoch to local time in perl on windows

We are trying to convert the output received from the below code
The current output is in this form
testingwindows,1446727960,1446728560,kkulka11,testingwin
testingwindows1,1446727160,141228560,kkulka11,testingwin
testingwindows2,1446727120,1446728560,kkulka11,testingwin
testingwindows3,1446727960,1446728560,kkulka11,testingwin
The output required is something like
testingwindows from Fri Oct 3 13:51:05 2015 GMT to Mon Nov 9 13:51:05 2015 GMT by kkulka11 for testingwin.
testingwindows1 from Fri Oct 2 13:51:05 2015 GMT to Mon Nov 9 13:51:05 2015 GMT by kkulka11 for testingwin.
testingwindows2 from Fri Oct 2 13:51:05 2015 GMT to Mon Nov 9 13:51:05 2015 GMT by kkulka11 for testingwin.
testingwindows3 from Fri Oct 12 13:51:05 2015 GMT to Mon Nov 9 13:51:05 2015 GMT by kkulka11 for testingwin.
This is my current code
if ( $COMMAND eq 'queryone' ) {
my $msend_query = "$MCELL_HOME\\bin\\mquery";
my #args_query = (
$msend_query,
"-q",
"-c", "$MCELL_HOME\\etc\\mclient.conf",
"-n", "$CS_BLACKOUT_CELL",
"-d",
"-f", "csv",
"-a", "CS_EMB_GBF_BLACKOUTS" ,
"-s", "blackout_host,start_timestamp,stop_timestamp,userid,reason",
"-w", "blackout_host: == '${BLACKOUTHOST}'"
);
system(#args_query);
We tried using perl -pe 's/(\d{10})/gmtime($1)/e'; but not able to convert and it gives this error
'o~}go⌂⌂t⌂x⌂w' is not recognized as an internal or external command,
operable program or batch file.
when we used the code as
if ( $COMMAND eq 'queryone' ) {
my $msend_query = "$MCELL_HOME\\bin\\mquery";
my $mqt = "$MCELL_HOME\\mqt.pl";
my #args_query = (
$msend_query,
"-q",
"-c", "$MCELL_HOME\\etc\\mclient.conf",
"-n", "$CS_BLACKOUT_CELL",
"-d",
"-f", "csv",
"-a", "CS_EMB_GBF_BLACKOUTS",
"-s", "blackout_host,start_timestamp,stop_timestamp,userid,reason",
"-w", "blackout_host: == '${BLACKOUTHOST}'"
) | $mqt;
system(#args_query);
Needed experts quick help and guidance to achieve the output in human-readable format.
Edit:
Updated the code as per Jacob comments but still not received the output as desired. Please suggest
if ( $COMMAND eq 'queryone' ) {
my $msend_query = "$MCELL_HOME\\bin\\mquery";
my #args_query = (
$msend_query,
"-q",
"-c", "$MCELL_HOME\\etc\\mclient.conf",
"-n", "$CS_BLACKOUT_CELL",
"-d",
"-f", "csv",
"-a", "CS_EMB_GBF_BLACKOUTS" ,
"-s", "blackout_host,start_timestamp,stop_timestamp,userid,reason",
"-w", "blackout_host: == '${BLACKOUTHOST}'"
);
chomp;
my #parts = split(/,/, system(#args_query));
$parts[1] = localtime($parts[1]);
$parts[2] = localtime($parts[2]);
printf("%s from %s to %s by %s for %s\n", #parts);
}
Output:
M:\AbhayBackup\PerlKK>test.pl -q -h testingwin
testingwin
sub: testingwin
testingwin,1446727960,1446728560,kkulka11,testingwin
0 from Thu Jan 1 05:30:00 1970 to Thu Jan 1 05:30:00 1970 by for
while (<DATA>) {
chomp;
my #parts = split(/,/, $_);
$parts[1] = localtime($parts[1]);
$parts[2] = localtime($parts[2]);
printf("%s from %s to %s by %s for %s\n", #parts);
}
__DATA__
testingwindows,1446727960,1446728560,kkulka11,testingwin
testingwindows1,1446727160,141228560,kkulka11,testingwin
testingwindows2,1446727120,1446728560,kkulka11,testingwin
testingwindows3,1446727960,1446728560,kkulka11,testingwin
Output:
testingwindows from Thu Nov 5 05:52:40 2015 to Thu Nov 5 06:02:40 2015 by kkulka11 for testingwin
testingwindows1 from Thu Nov 5 05:39:20 2015 to Sun Jun 23 07:09:20 1974 by kkulka11 for testingwin
testingwindows2 from Thu Nov 5 05:38:40 2015 to Thu Nov 5 06:02:40 2015 by kkulka11 for testingwin
testingwindows3 from Thu Nov 5 05:52:40 2015 to Thu Nov 5 06:02:40 2015 by kkulka11 for testingwin
Edit: based on your most recent update to the question, I now believe that you're trying to capture the output of the command and process it. Since you haven't provided a minimal, complete, and verifiable example, and because I have no idea what mquery is and you haven't provided an explanation for that, either, I present to you this guess:
if ($COMMAND eq 'queryone') {
my #lines = `$MCELL_HOME\\bin\\mquery -q -c $MCELL_HOME\\etc\\mclient.conf -n $CS_BLACKOUT_CELL -d -f csv -a CS_EMB_GBF_BLACKOUTS -s blackout_host,start_timestamp,stop_timestamp,userid,reason -w blackout_host: == '${BLACKOUTHOST}'`;
for (#lines) {
chomp;
my #parts = split(/,/, $_);
$parts[1] = localtime($parts[1]);
$parts[2] = localtime($parts[2]);
printf("%s from %s to %s by %s for %s\n", #parts);
}
}

cron task wouldn't work, why?

I want to write a cron task to record the ntpdate synchronization info into the system log, but there's no such info printed in the /var/log/messages after this cron task is done, where did I do wrong?
The followings are what my crontab looks like.
*/1 * * * * ntpdate 192.168.100.97 | logger -t "NTP"
*/1 * * * * echo "log test" | logger -t "TEST"
*/1 * * * * whoami | logger -t "WHO"
When I do tailf /var/log/messages and wait some time I only got the following lines, the NTP lines are missing.
Oct 29 15:22:01 localhost TEST: log test
Oct 29 15:22:01 localhost WHO: root
Oct 29 15:23:01 localhost TEST: log test
Oct 29 15:23:01 localhost WHO: root
Oct 29 15:24:01 localhost TEST: log test
Oct 29 15:24:01 localhost WHO: root
Oct 29 15:25:01 localhost TEST: log test
Oct 29 15:25:01 localhost WHO: root
Oct 29 15:26:01 localhost TEST: log test
Oct 29 15:26:01 localhost WHO: root
But when I do the ntpdate 192.168.100.97 | logger -t "NTP" in the command line, I could see there's message Oct 29 15:28:39 localhost NTP: 29 Oct 15:28:39 ntpdate[11101]: adjust time server 192.168.100.97 offset 0.000043 sec print out in the system log. What am I missing here?
Thanks in advance for your kind help.

Cronjob missing content from mail body, manually works fine

Edit: Solution found via Barmar's answer. Added full smartctl command path and it works via crontab now.
I have the below script:
#!/bin/bash
#set -x
EMAIL="admin#domain.co.uk"
FILE="/root/scripts/hddreport.txt"
HOST=`hostname`
HDD01="/dev/sda"
P=`ping -c 1 $HOST | sed '1 ! d' | awk '{print $3}'`
cd /root/scripts/
echo -en "HDD health check on the server hosting" $HOST $P > $FILE
echo -e "\n" >> $FILE
smartctl -H $HDD01 >> $FILE
# The above commands do correctly write the content to $FILE (proved by removing the rm command at the bottom and doing cat on the file after)
smartctl -H $HDD01
echo "\nEmailed you the health of the Hard Drive $HDD01\n"
mailx -s "HDD health check complete on `date`" $EMAIL < $FILE
rm $FILE
which runs fine by doing bash /root/scripts/diskhealth.sh as it shows this in my mailbox:
HDD health check on the server hosting domain.co.uk (0.0.0.0)
smartctl 5.40 2010-07-12 r3124 [x86_64-unknown-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net
SMART Health Status: OK
But when I let it run via crontab using any of the following syntax:
X 20 * * * /bin/bash /root/scripts/diskhealth.sh
X 20 * * * /bin/sh /root/scripts/diskhealth.sh
X 20 * * * /root/scripts/diskhealth.sh
it puts everything but the smartctl disk check:
HDD health check on the server hosting domain.co.uk (0.0.0.0)
Here's what it shows if I add extra echo lines:
This is a test
HDD health check on the server hosting domain.co.uk (0.0.0.0)
Amended script for "This is a test" below:
#!/bin/bash
#set -x
EMAIL="admin#domain.co.uk"
FILE="/root/scripts/hddreport.txt"
HOST=`hostname`
HDD01="/dev/sda"
P=`ping -c 1 $HOST | sed '1 ! d' | awk '{print $3}'`
cd /root/scripts/
echo "This is a test" > $FILE
echo -en "HDD health check on the server hosting" $HOST $P >> $FILE
echo -e "\n" >> $FILE
smartctl -H $HDD01 >> $FILE
smartctl -H $HDD01
echo "\nEmailed you the health of the Hard Drive $HDD01\n"
mailx -s "HDD health check complete on `date`" $EMAIL < $FILE
rm $FILE
Here is the /var/log/syslog output from cron:
Jun 6 20:25:01 hostname /USR/SBIN/CRON[1018112]: (root) CMD (bash /root/scripts/diskhealth.sh)
Jun 6 20:25:01 hostname postfix/pickup[1016576]: 5740356613F: uid=0 from=<root>
Jun 6 20:25:01 hostname postfix/cleanup[1018125]: 5740356613F: message-id=<20130606192501.5740356613F#hostname>
Jun 6 20:25:01 hostname postfix/qmgr[292015]: 5740356613F: from=<root#hostname>, size=465, nrcpt=1 (queue active)
Jun 6 20:25:01 hostname postfix/pickup[1016576]: 631F156613E: uid=0 from=<root>
Jun 6 20:25:01 hostname postfix/cleanup[1018125]: 631F156613E: message-id=<20130606192501.631F156613E#hostname>
Jun 6 20:25:01 hostname postfix/qmgr[292015]: 631F156613E: from=<root#hostname>, size=759, nrcpt=1 (queue active)
Jun 6 20:25:01 hostname pvemailforward[1018132]: forward mail to <root#localhost.localdomain>
Jun 6 20:25:01 hostname postfix/pickup[1016576]: B597B566148: uid=65534 from=<nobody>
Jun 6 20:25:01 hostname postfix/cleanup[1018125]: B597B566148: message-id=<20130606192501.631F156613E#hostname>
Jun 6 20:25:01 hostname postfix/local[1018131]: 631F156613E: to=<root#hostname>, orig_to=<root>, relay=local, delay=0.39, delays=0.16/0/0/0.23, dsn=2.0.0, status=sent (delivered to command: /usr/bin/pvemailforward)
Jun 6 20:25:01 hostname postfix/qmgr[292015]: 631F156613E: removed
Jun 6 20:25:01 hostname postfix/qmgr[292015]: B597B566148: from=<nobody#hostname>, size=963, nrcpt=1 (queue active)
Jun 6 20:25:01 hostname postfix/smtp[1018135]: B597B566148: to=<root#localhost.localdomain>, relay=none, delay=0.16, delays=0.12/0/0.04/0, dsn=5.4.4, status=bounced (Host or domain name not found. Name service error for name=localhost.localdomain type=A: Host not found)
Jun 6 20:25:01 hostname postfix/qmgr[292015]: B597B566148: removed
Jun 6 20:25:01 hostname postfix/cleanup[1018125]: D6570566147: message-id=<20130606192501.D6570566147#hostname>
Jun 6 20:25:01 hostname postfix/smtp[1018130]: 5740356613F: to=<admin#domain.co.uk>, relay=ASPMX.L.GOOGLE.COM[173.194.67.27]:25, delay=0.68, delays=0.12/0/0.19/0.36, dsn=2.0.0, status=sent (250 2.0.0 OK 1370546701 iy4si8635735wic.1 - gsmtp)
Jun 6 20:25:01 ds9453 postfix/qmgr[292015]: 5740356613F: removed
The email is received, just missing the smartctl output.
Cron jobs don't run your .profile. So if smartctl is in a directory you add to $PATH in your profile, it won't be found when you run via cron. Try using the full pathname to the command.

Resources