I need to cut something from stream, but it seems, that cut cant process last line before EOL or EOF appear.
My bash command:
tail -n 5 -F /data/apache/log/error/error-log | cut -d ' ' -f1-5
log file contains for example:
[Wed Jan 14 09:00:00 2015] [error] Error 1
[Wed Jan 14 09:10:00 2015] [error] Error 2
[Wed Jan 14 09:20:00 2015] [error] Error 3
[Wed Jan 14 09:30:00 2015] [error] Error 4
[Wed Jan 14 09:40:00 2015] [error] Error 5
but my output is:
[Wed Jan 14 09:00:00 2015]
[Wed Jan 14 09:10:00 2015]
[Wed Jan 14 09:20:00 2015]
[Wed Jan 14 09:30:00 2015]
is there a way to make cut process (and output) last, unfinished line if there are all needed columns (first five) streamed already?
My desired output should contain last line from stream:
[Wed Jan 14 09:00:00 2015]
[Wed Jan 14 09:10:00 2015]
[Wed Jan 14 09:20:00 2015]
[Wed Jan 14 09:30:00 2015]
[Wed Jan 14 09:40:00 2015]
//edit
Answers "remove -F" are useless, because I need it for live stream. Whenever apache add new entry into log, I need immediately process it with cut. Without user interaction, without running script again etc. Just process stream in real time in the way, that 'cut' don't wait for EOL before it process (and output) line.
If that is not possible, fine, I try to find workaround, but please, don't tell me to not use live stream when I need live stream!
Finally, after some tests, I found that the problem is not the tail but the cut. Replacing cut by awk fixes the problem:
tail -n 5 -F /data/apache/log/error/error-log | awk '{print $1, $2, $3, $4, $5}'
Is it really necessary -F (follow & retry) option?
Remove it, and it'll work.
Related
I have these lines in a file:
Oct 29 23:14:39
Oct 30 19:45:15
Oct 31 13:15:19
Nov 1 10:34:15
Nov 2 18:39:20
Nov 3 12:34:59
Nov 4 16:34:59
Nov 5 20:34:59
When I run sort -r -k2 it gives me the following:
Nov 5
Nov 4
Nov 3
Oct 31
Oct 30
Oct 29
Nov 2
Nov 1
How do I get it like so:
Nov 5
Nov 4
Nov 3
Nov 2
Nov 1
Oct 31
Oct 30
Oct 29
Would appreciate any pointers, comments, advices at all. Do I also need to sort on months in reverse order? How? -M -r?
Assuming the dates are sorted in the file, just reverse the line order:
tac file
This works well!
sort -k1Mr -r -k2nr file.txt
file.txt content
Oct 6 20:34:59
Oct 29 23:14:39
Oct 30 19:45:15
Oct 31 13:15:19
Nov 1 10:34:15
Nov 2 18:39:20
Nov 3 12:34:59
Nov 4 16:34:59
Nov 5 20:34:59
Output:
Nov 5 20:34:59
Nov 4 16:34:59
Nov 3 12:34:59
Nov 2 18:39:20
Nov 1 10:34:15
Oct 31 13:15:19
Oct 30 19:45:15
Oct 29 23:14:39
Oct 6 20:34:59
NOTE:
The following answers fail to sort the entry Oct 6 20:34:59 if exists.
sort -k1,1 -k2r,2
sort -rM
This works for me:
sort -k1Mr,1 -k2r,2
month-sorted by field #1 reversely, then sorted by field #2 reversely
Nov 5 20:34:59
Nov 4 16:34:59
Nov 3 12:34:59
Nov 2 18:39:20
Nov 1 10:34:15
Oct 31 13:15:19
Oct 30 19:45:15
Oct 29 23:14:39
Edit:
A key specification like -k2 means to take all the fields from 2 to the end of the line into account.
To sort by a single column, use -k2,2 as the key specification. This means to use the fields from #2 to #2, i.e. only the second field.
man sort
-k, --key=KEYDEF
sort via a key; KEYDEF gives location and type
...
KEYDEF is F[.C][OPTS][,F[.C][OPTS]] for start and stop position, where
F is a field number and C a character position in the field; both are
origin 1, and the stop position defaults to the line's end.
The arrangement of the fields is lucky, in that the primary one comes first, and the secondary one second (and the rest of the line doesn't matter because you have at most one entry per day). But your example has only Oct and Nov, and I don't think you ought to rely on that. If you add e.g. Aug 7 you'll need a more robust solution. I suggest:
sort -rM
That is, sort in the default way (by the first field, then by the second), treat any month abbreviation found as a month abbreviation, and reverse the whole thing.
EDIT: This is incorrect. #SaSkY's answer is the correct one.
I have 3 variables, $commonName, $expiryDate and $DaysRemInUnixEpoch. Each variable has 3 lines as below output. I want to display output of all 3 variables in to 3 different columns. I tried looking for solution using printf but no luck. Can anyone please advise if they have done this in the past using printf and how? Any help will be much appreciated.
Below are 3 variables output together in one column. I want to split in to 3 columns having 3 rows in each column.
bash-4.1$ echo -e "$commonName\n$expiryDate\n$daysRemInUnixEpoch"
mycertificate_mycert.mycomp.net
PSIN0P551
ROOTROOTCA
Feb 6 2022 11:57:32 GMT
Jan 9 2023 18:51:25 GMT
Mar 12 2035 18:24:54 GMT
682
1020
5465
bash-4.1$
desired output I am looking for is something like below
mycertificate_mycert.mycomp.net Feb 6 2022 11:57:32 GMT 682
PSIN0P551 Jan 9 2023 18:51:25 GMT 1020
ROOTROOTCA Mar 12 2035 18:24:54 GMT 5465
With bash (Process Substitution), paste and column:
paste -d ';' <(echo "$commonName") <(echo "$expiryDate") <(echo "$daysRemInUnixEpoch") | column -s ';' -t
Output:
mycertificate_mycert.mycomp.net Feb 6 2022 11:57:32 GMT 682
PSIN0P551 Jan 9 2023 18:51:25 GMT 1020
ROOTROOTCA Mar 12 2035 18:24:54 GMT 5465
I assume that your variables do not contain ;.
see: man paste and man column
I am trying to get the date "+%a %b %d %R:%S %Y" in bash.
here's the sample command and output
$ xscreensaver-command --time
XScreenSaver 5.32: screen non-blanked since Thu Oct 29 12:15:05 2015 (hacks: #184, #60)
I am trying to get the the value Thu Oct 29 12:15:05 2015 on the string.
How can I achieve this?
Try to append with GNU grep:
2>&1 | grep -Po 'since \K.*(?= \()'
Output:
Thu Oct 29 12:15:05 2015
I get the following from spamdb, where the third field represents the time in seconds since the Epoch.
Cns# spamdb | fgrep TRAPPED
TRAPPED|113.163.117.129|1360836903
TRAPPED|113.171.216.201|1360837481
TRAPPED|122.177.159.61|1360844596
TRAPPED|36.231.9.231|1360865649
TRAPPED|37.146.207.209|1360832096
TRAPPED|212.156.98.210|1360837015
TRAPPED|59.99.160.62|1360839785
TRAPPED|86.127.116.162|1360840492
TRAPPED|92.83.139.194|1360843056
TRAPPED|219.71.12.150|1360844704
I want to sort this table by the time, and print the time field with date -r, such that it's presentable and clear when the event has occurred.
How do I do this in tcsh on OpenBSD?
Sorting with sort is easy, and so is editing with sed; but how do I make sed execute date -r or equivalent?
There are indeed a few obstacles here: first, you basically have to separate the data, and then one part of it is presented as-is, whereas another part has to be passed down to date -r for date formatting, prior to being presented to the user.
Another obstacle is making sure the output is aligned: apparently, it's quite difficult to handle the tab character in the shell, possibly only on the BSDs:
sed replace literal TAB
Replacing / with TAB using sed
Also, as we end up piping this to sh for execution, we have to use a different separator for the fields other than the pipe character, |.
So far, this is the best snippet I could come up with, it seems to work great in my tcsh:
Cns# spamdb | fgrep TRAPPED | sort -n -t '|' -k 3 | sed -E -e 's#\|###g' \
-e 's#^([A-Z]+)#([0-9.]+)#([0-9]+)$#"echo -n \2_"; "date -r \3"#g' | \
xargs -n1 sh -c | awk '{gsub("_","\t",$0); print;}'
37.146.207.209 Thu Feb 14 00:54:56 PST 2013
113.163.117.129 Thu Feb 14 02:15:03 PST 2013
212.156.98.210 Thu Feb 14 02:16:55 PST 2013
113.171.216.201 Thu Feb 14 02:24:41 PST 2013
59.99.160.62 Thu Feb 14 03:03:05 PST 2013
86.127.116.162 Thu Feb 14 03:14:52 PST 2013
92.83.139.194 Thu Feb 14 03:57:36 PST 2013
122.177.159.61 Thu Feb 14 04:23:16 PST 2013
219.71.12.150 Thu Feb 14 04:25:04 PST 2013
36.231.9.231 Thu Feb 14 10:14:09 PST 2013
please advice how to match the ERROR strings that comes only after "]" char
by awk or sed
[Mon Jan 30 21:14:01 IST 2012] ERROR file /etc/ghy.txt not found
My target is to count all ERROR words that appears only after "]” character in the log file
remark - between “]” and ERROR must be one space or more
My target is to count all ERROR words that appears only after "]”
character in the log file
remark - between “]” and ERROR must be one space or more
then you don't need those nuclear heads like awk, sed even perl. grep does it for you like this:
grep -Pc ']\s+ERROR' yourLogFile
small test:
kent$ echo "[Mon Jan 30 21:14:01 IST 2012] ERROR file /etc/ghy.txt not found "|grep -Pc ']\s+ERROR'
1
AWK:
awk -F"] " '/ERROR/{print $2}' inputfile
Test:
[jaypal:~] echo "[Mon Jan 30 21:14:01 IST 2012] ERROR file /etc/ghy.txt not found " | awk -F"] " '/ERROR/{print $2}'
ERROR file /etc/ghy.txt not found
Perl:
perl -pe 's/.*(?<=] )(.*)/$1/' inputfile
Test:
echo "[Mon Jan 30 21:14:01 IST 2012] ERROR file /etc/ghy.txt not found " | perl -pe 's/.*(?<=] )(.*)/$1/'
ERROR file /etc/ghy.txt not found
Count no. of occurrences:
[jaypal:~/Temp] cat file
[Mon Jan 30 21:14:01 IST 2012] ERROR file /etc/ghy.txt not found
[Mon Jan 30 21:14:01 IST 2012] ERROR file /etc/ghy.txt not found
[Mon Jan 30 21:14:01 IST 2012] ERROR file /etc/ghy.txt not found
[Mon Jan 30 21:14:01 IST 2012] ERROR file /etc/ghy.txt not found
[Mon Jan 30 21:14:01 IST 2012] ERROR file /etc/ghy.txt not found
[Mon Jan 30 21:14:01 IST 2012] ERROR file /etc/ghy.txt not found
[Mon Jan 30 21:14:01 IST 2012] ERROR file /etc/ghy.txt not found
[Mon Jan 30 21:14:01 IST 2012] ERROR file /etc/ghy.txt not found
[Mon Jan 30 21:14:01 IST 2012] ERROR file /etc/ghy.txt not found
[jaypal:~/Temp] awk -F"] " '/ERROR/{a[NR]=$2}END{print "count is " length(a)}' file
count is 9
This might work for you:
grep -c '\] \+ERROR' file
Or
grep -c '\][[:space:]]\+ERROR' file
Or
sed '/\]\s\+ERROR/!d' file | wc -l
Here is a shell only snippet, that should be faster than using any external programs (for small file reads) since it only uses shell builtins. It can be modified to handle errors case by case while running in a daemon mode (by tail-ing the log file to a fifo instead of reading it directly and modifying the case conditionals)
not the intended use of echo, but it does reduce spaces/tabs to 1 space
FILE="logfile"
ERRORS=0
while read LINE || [ "$LINE" ]; do
case "`echo $LINE`" in
*\]" "ERROR*)ERRORS=$(($ERRORS+1));;
esac
done < "${FILE}"
echo $ERRORS
You can do
sed -n '/] ERROR/p' infile