How to use startmsg.regex in Rsyslog - rsyslog

The following is my conf file. I want to add config for startmsg.regex.
I added the following line in my config file
startmsg.regex="^[[:digit:]]{4}\/[[:digit:]]{2}\/[[:digit:]]{2} [[:digit:]]{2}:[[:digit:]]{2}:[[:digit:]]{2}"
ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup proxy
$WorkDirectory /var/spool/rsyslog
$InputFileName /var/log/app/cache.log
$InputFileTag app-error:
$InputFileStateFile stat-app-error
$InputFileSeverity error
$InputFilePersistStateInterval 20000
$InputRunFileMonitor
startmsg.regex="^[[:digit:]]{4}\/[[:digit:]]{2}\/[[:digit:]]{2} [[:digit:]]{2}:[[:digit:]]{2}:[[:digit:]]{2}"
$template AppError,"error %msg%\n"
if $programname == 'app-error' then ##0.0.0.0:12345;AppError
if $programname == 'app-error' then ~
And when I check my config file using rsyslogd -N1, I am getting error.
What is the correct way to use this feature ?

Following is my conf file which I am using to send my logs via rsyslog.
I am also adding sample logs.
This configuration will match that each event starts with YYYY/MM/DD HH:MM:SS
and then send to my TCP endpoint.
This configuration can be used with multi line logs
module(load="imfile" PollingInterval="10") #needs to be done just once
# File 1
input(type="imfile"
File="/var/log/app/my.log"
Tag="app-error"
Severity="error"
startmsg.regex="^[[:digit:]]{4}/[[:digit:]]{2}/[[:digit:]]{2} [[:digit:]]{1,2}:[[:digit:]]{1,2}:[[:digit:]]{1,2}"
)
$PrivDropToGroup proxy
$WorkDirectory /var/spool/rsyslog
$template AppError,"error %msg%\n"
if $programname == 'app-error' then ##0.0.0.0:12345;AppError
if $programname == 'app-error' then ~
Sample Log :
2017/10/24 09:14:06 id1| Took 0.00 seconds ( 0.00 entries/sec).
CPU Usage: 0.052 seconds = 0.032 user + 0.020 sys
Maximum Resident Size: 104944 KB
Page faults with physical i/o: 0
2017/10/24 09:14:06 id1| found error
Now rsyslog will send my multi line logs as a single event to my tcp end point as follows :
2017/10/24 09:14:06 id1| Took 0.00 seconds ( 0.00 entries/sec). \nCPU Usage: 0.052 seconds = 0.032 user + 0.020 sys \nMaximum Resident Size: 104944 KB \nPage faults with physical i/o: 0
2017/10/24 09:14:06 id1| found error

Related

bash - count, process, and increment thru multiple "Tasks" in log file

I have log files that are broken down into between 1 and 4 "Tasks". In each "Task" there are sections for "WU Name" and "estimated CPU time remaining". Ultimately, I want to the bash script output to look like this 3 Task example;
Task 1 Mini_Protein_binds_COVID-19_boinc_ 0d:7h:44m:28s
Task 2 shapeshift_pair6_msd4X_4_f_e0_161_ 0d:4h:14m:22s
Task 3 rep730_0078_symC_reordered_0002_pr 1d:1h:38m:41s
So far; I can count the Tasks in the log. I can isolate x number of characters I want from the "WU Name". I can convert the "estimated CPU time remaining" in seconds to days:hours:minutes:seconds. And I can output all of that into 'pretty' columns. Problem is that I can only process 1 Task using;
# Initialize counter
counter=1
# Count how many iterations
cnt_wu=`grep -c "WU name:" /mnt/work/sec-conv/bnc-sample3.txt`
# Iterate the loop for cnt-wu times
while [ $counter -le ${cnt_wu} ]
do
core_cnt=$counter
wu=`cat /mnt/work/sec-conv/bnc-sample3.txt | grep -Po 'WU name: \K.*' | cut -c1-34`
sec=`cat /mnt/work/sec-conv/bnc-sample3.txt | grep -Po 'estimated CPU time remaining: \K.*' | cut -f1 -d"."`
dhms=`printf '%dd:%dh:%dm:%ds\n' $(($sec/86400)) $(($sec%86400/3600)) $(($sec%3600/60)) \ $(($sec%60))`
echo "Task ${core_cnt}" $'\t' $wu $'\t' $dhms | column -ts $'\t'
counter=$((counter + 1))
done
Note: /mnt/work/sec-conv/bnc-sample3.txt is a static one Task sample only used for this scripts dev.
What I can't figure out is the next step which is to be able to process x number of multiple Tasks. I can't figure out how to leverage the while/counter combination properly, and can't figure out how to increment through the occurrences of Tasks.
Adding bnc-sample.txt (contains 3 Tasks)
1) -----------
name: Rosetta#home
master URL: https://boinc.bakerlab.org/rosetta/
user_name: XXXXXXX
team_name:
resource share: 100.000000
user_total_credit: 10266.993660
user_expavg_credit: 512.420495
host_total_credit: 10266.993660
host_expavg_credit: 512.603669
nrpc_failures: 0
master_fetch_failures: 0
master fetch pending: no
scheduler RPC pending: no
trickle upload pending: no
attached via Account Manager: no
ended: no
suspended via GUI: no
don't request more work: no
disk usage: 0.000000
last RPC: Wed Jun 10 15:55:29 2020
project files downloaded: 0.000000
GUI URL:
name: Message boards
description: Correspond with other users on the Rosetta#home message boards
URL: https://boinc.bakerlab.org/rosetta/forum_index.php
GUI URL:
name: Your account
description: View your account information
URL: https://boinc.bakerlab.org/rosetta/home.php
GUI URL:
name: Your tasks
description: View the last week or so of computational work
URL: https://boinc.bakerlab.org/rosetta/results.php?userid=XXXXXXX
jobs succeeded: 117
jobs failed: 0
elapsed time: 2892439.609931
cross-project ID: 3538b98e5f16a958a6bdd2XXXXXXXXX
======== Tasks ========
1) -----------
name: shapeshift_pair6_msd4X_4_f_e0_161_X_0001_0001_fragments_abinitio_SAVE_ALL_OUT_946179_730_0
WU name: shapeshift_pair6_msd4X_4_f_e0_161_X_0001_0001_fragments_abinitio_SAVE_ALL_OUT_946179_730
project URL: https://boinc.bakerlab.org/rosetta/
received: Mon Jun 8 09:58:08 2020
report deadline: Thu Jun 11 09:58:08 2020
ready to report: no
state: downloaded
scheduler state: scheduled
active_task_state: EXECUTING
app version num: 420
resources: 1 CPU
estimated CPU time remaining: 26882.771040
slot: 1
PID: 28434
CPU time at last checkpoint: 3925.896000
current CPU time: 4314.761000
fraction done: 0.066570
swap size: 431 MB
working set size: 310 MB
2) -----------
name: rep730_0078_symC_reordered_0002_propagated_0001_0001_0001_A_v9_fold_SAVE_ALL_OUT_946618_54_0
WU name: rep730_0078_symC_reordered_0002_propagated_0001_0001_0001_A_v9_fold_SAVE_ALL_OUT_946618_54
project URL: https://boinc.bakerlab.org/rosetta/
received: Mon Jun 8 09:58:08 2020
report deadline: Thu Jun 11 09:58:08 2020
ready to report: no
state: downloaded
scheduler state: scheduled
active_task_state: EXECUTING
app version num: 420
resources: 1 CPU
estimated CPU time remaining: 26412.937920
slot: 2
PID: 28804
CPU time at last checkpoint: 3829.626000
current CPU time: 3879.975000
fraction done: 0.082884
swap size: 628 MB
working set size: 513 MB
3) -----------
name: Mini_Protein_binds_COVID-19_boinc_site3_2_SAVE_ALL_OUT_IGNORE_THE_REST_0aw6cb3u_944116_2_0
WU name: Mini_Protein_binds_COVID-19_boinc_site3_2_SAVE_ALL_OUT_IGNORE_THE_REST_0aw6cb3u_944116_2
project URL: https://boinc.bakerlab.org/rosetta/
received: Mon Jun 8 09:58:47 2020
report deadline: Thu Jun 11 09:58:46 2020
ready to report: no
state: downloaded
scheduler state: scheduled
active_task_state: EXECUTING
app version num: 420
resources: 1 CPU
estimated CPU time remaining: 27868.559616
slot: 0
PID: 30988
CPU time at last checkpoint: 1265.356000
current CPU time: 1327.603000
fraction done: 0.032342
swap size: 792 MB
working set size: 668 MB
Again, I appreciate any guidance!

Configuring 3proxy Socks5 behind NAT network - error 12

I'm trying to configure 3proxy server using this guide (I've already used it on OHV hosting and it works just nice!), now trying to start 3proxy behind NAT, and have error 12 of 3proxy which means 12 - failed to bind()
Where is mistake and what I'm doing wrong?
Internal IP:
172.16.20.50
External IP:
82.118.227.155
NAT Ports:
5001-5020
Here are my entire config:
######################
##3Proxy.cfg Content##
######################
##Main##
#Starting 3proxy as a service/daemon
daemon
#DNS Servers to resolve domains and for the local DNS cache
#that providers faster resolution for cached entries
nserver 8.8.8.8
nserver 1.1.1.1
nscache 65536
#Authentication
#CL = Clear Text, CR = Encrypted Passswords (MD5)
#Add MD5 users with MD5 passwords with "" (see below)
#users "user:CR:$1$lFDGlder$pLRb4cU2D7GAT58YQvY49."
users 3proxy:CL:hidden
#Logging
log /var/log/3proxy/3proxy.log D
logformat "- +_L%t.%. %N.%p %E %U %C:%c %R:%r %O %I %h %T"
#logformat "-""+_L%C - %U [%d/%o/%Y:%H:%M:%S %z] ""%T"" %E %I"
rotate 30
#Auth type
#auth strong = username & password
auth strong
#Binding address
external 82.118.227.155
internal 172.16.20.50
#SOCKS5
auth strong
flush
allow 3proxy
maxconn 1000
socks -p5011
User 3proxy created, access to 3proxy granted.
Logs, which means connection established, but no traffic transfered (0/0):
[root#bgvpn113 ~]# tail -f /var/log/3proxy/3proxy.log.2018.05.14
1526329023.448 SOCK5.5011 00012 3proxy MY_LOCAL_IP:21151 88.212.201.205:443 0 0 0 CONNECT_88.212.201.205:443
1526329023.458 SOCK5.5011 00012 3proxy MY_LOCAL_IP:21154 88.212.201.205:443 0 0 0 CONNECT_88.212.201.205:443
1526329023.698 SOCK5.5011 00012 3proxy MY_LOCAL_IP:21158 88.212.201.205:443 0 0 0 CONNECT_88.212.201.205:443
1526329037.419 SOCK5.5011 00012 3proxy MY_LOCAL_IP:21162 195.201.201.32:443 0 0 0 CONNECT_195.201.201.32:443
1526329037.669 SOCK5.5011 00012 3proxy MY_LOCAL_IP:21164 195.201.201.32:443 0 0 0 CONNECT_195.201.201.32:443
Mistake was in outside IP.
I set both ips to 172.16.20.50 and it started to work!

icinga2 disk space check or with three arguments

I am trying to configure icinga2 to monitor my linux server disk space using check_nrpe. my configuraiton is given below
nrpe.cfg:
command[check_root]=/usr/lib/nagios/plugins/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$
icinga configuration
object CheckCommand "nrpe-check-2arg" {
import "plugin-check-command"
command = [PluginDir + "/check_nrpe" ]
arguments = {
"-H" = "$host_name$"
"-c" = "$check$"
"-a" = "$loads$"
}
}
object Service "testing-haproxy-master: / disk space" {
import "generic-service"
host_name = "tmahaprx01.verizon.com"
check_command = "nrpe-check-2arg"
vars.address = "192.168.1.104"
vars.check = "check_root"
vars.loads = "80%!90%!/"
}
Now the out put i am getting is
root#icinga:/etc/icinga2/hosts# /usr/lib/nagios/plugins/check_nrpe -H 192.168.1.104 -c check_root -a '80%C!90%!/'
DISK OK - free space: /sys/fs/cgroup 0 MB (100% inode=99%); /dev 1457 MB (99%
inode=99%); /run 293 MB (99% inode=99%); /run/lock 5 MB (100% inode=99%);
/run/shm 1468 MB (100% inode=99%); /run/user 100 MB (100% inode=99%);|
/sys/fs/cgroup=0MB;0;0;0;0 /dev=0MB;291;145;0;1457 /run=0MB;58;29;0;293
/run/lock=0MB;0;0;0;5 /run/shm=0MB;293;146;0;1468 /run/user=0MB;19;9;0;100
The expecting output when I execute from my remote Linux machine is
root#tmahaprx01:~# /usr/lib/nagios/plugins/check_disk -w 80% -c 90% -p /
DISK OK - free space: / 43144 MB (96% inode=97%);| /=1743MB;9462;4731;0;47314
Could you please guide me how i can pass the third argument (/) ?
The problem with NRPE is that you're writing a command that executes another command. Assuming that the nrpe.cfg includes something like this:
command[check_disk]=/usr/lib/nagios/plugins/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$
you know that the path must be the 3rd argument:
object CheckCommand "nrpe-disk" {
import "nrpe"
vars.nrpe_arguments = [ "$disk_wfree$%", "$disk_cfree$%", "$disk_partition$" ]
vars.nrpe_command = "check_disk"
//variables should be propagated from host/group definition
vars.disk_wfree = 20
vars.disk_cfree = 10
vars.disk_partition = "/"
}
variable names might be dependent on Icinga version, check the original nrpe command definition on your system, it might be located in:
/usr/share/icinga2/include/command-plugins.conf

Sphinx: Permission denied/Broken pipe on deltas merge

When i launch this batch command for create and merge deltas:
D:\Sphinx\bin\indexer.exe --config D:\Sphinx\project\product.conf idx_product_delta --rotate
D:\Sphinx\bin\indexer.exe --config D:\Sphinx\project\product.conf --merge idx_product_main idx_product_delta --rotate
In searchd.log found this error and deltas are not merged into main
[Fri Sep 25 15:34:42.549 2015] [ 2312] WARNING: rotating index 'idx_product_main': cur to old rename failed: rename D:\Sphinx\project\data\product.spa to D:\Sphinx\project\data\product.old.spa failed: Broken pipe
Console output is:
using config file 'D:\Sphinx\project\product.conf'...
merging index 'idx_product_delta' into index 'idx_product_main'...
read 7.2 of 7.2 MB, 100.0% done
merged 11.5 Kwords
merged in 0.127 sec
ERROR: index 'idx_product_main': failed to delete 'D:\Sphinx\project\data\product.new.spa': Permission deniedtotal 671 reads, 0.006 sec, 15.3 kb/call avg, 0.0 msec/call avg total 36 writes, 0.004 sec, 277.8 kb/call avg, 0.1 msec/call avg
My product.conf is:
source src_product_main
{
type = mysql
sql_host = localhost
sql_user = root
sql_pass =
sql_db = database
sql_port = 3306 # optional, default is 3306
sql_query_pre = REPLACE INTO sphinx_index_meta(index_name, last_update) \
VALUES('idx_prodotti_main', current_timestamp())
sql_query_range = SELECT MIN(id),MAX(id) \
FROM product \
WHERE deleted = 0 AND visible= 1
sql_range_step = 1000
sql_query = SELECT id, text, last_update \
FROM product \
WHERE id>=$start AND id<=$end AND deleted = 0 AND visible = 1
sql_attr_timestamp = last_update
}
index idx_product_main
{
source = src_product_main
path = D:\Sphinx\project\data\product
ondisk_attrs = 1
stopwords = D:\Sphinx\project\stopwords.txt
min_word_len = 2
min_prefix_len = 0
min_infix_len = 3
ngram_len = 1
}
source src_product_delta : src_product_main
{
sql_query_range = SELECT MIN(id),MAX(id) \
FROM product \
WHERE deleted = 0 AND visible= 1
sql_range_step = 1000
sql_query = SELECT id, text, last_update \
FROM product \
WHERE id>=$start AND id<=$end AND deleted = 0 AND visible = 1
}
index idx_product_delta : idx_product_main
{
source = src_product_delta
path = D:\Sphinx\project\delta\product
ondisk_attrs = 1
stopwords = D:\Sphinx\project\stopwords.txt
min_word_len = 2
min_prefix_len = 0
min_infix_len = 3
ngram_len = 1
}
indexer
{
mem_limit = 128M
max_iosize = 1M
}
searchd
{
listen = 9312
listen = 9306:mysql41
log = D:\Sphinx\project\log\searchd.log
query_log = D:\Sphinx\project\log\query.log
read_timeout = 5
client_timeout = 300
max_children = 30
pid_file = D:\Sphinx\project\log\searchd.pid
seamless_rotate = 1
preopen_indexes = 0
unlink_old = 1
workers = threads # for RT to work
binlog_path = D:\Sphinx\project\data
}
I have also tried on Windows 7 and Windows 8, with both stable 2.2.10 and beta
2.3.1-id64-beta (r4926) with same error.
indexer running with a cron (windows scheduler) as SYSTEM user
searchd service running as SYSTEM user
D:\Sphinx\project\data\ folder permission has full control for SYSTEM
How can I solve this issue
UPDATE for Eugene Soldatov answer
I have also tried (first command less --rotate)
D:\Sphinx\bin\indexer.exe --config D:\Sphinx\project\product.conf idx_product_delta
D:\Sphinx\bin\indexer.exe --config D:\Sphinx\project\product.conf --merge idx_product_main idx_product_delta --rotate
but in console output found this error
Sphinx 2.2.10-id64-release (2c212e0)
Copyright (c) 2001-2015, Andrew Aksyonoff
Copyright (c) 2008-2015, Sphinx Technologies Inc (http://sphinxsearch.com)
using config file 'D:\Sphinx\project\product.conf'...
indexing index 'idx_prodotti_delta'...
FATAL: failed to lock D:\Sphinx\project\delta\prodotti.spl: No error, will not index. Try --rotate option.
Sphinx 2.2.10-id64-release (2c212e0)
Copyright (c) 2001-2015, Andrew Aksyonoff
Copyright (c) 2008-2015, Sphinx Technologies Inc (http://sphinxsearch.com)
using config file 'D:\Sphinx\project\product.conf'...
merging index 'idx_prodotti_delta' into index 'idx_prodotti_main'...
read 7.2 of 7.2 MB, 100.0% done
merged 11.5 Kwords
merged in 0.214 sec
ERROR: index 'idx_prodotti_main': failed to delete 'D:\Sphinx\project\data\prodotti.new.spa': Permission deniedtotal 20136 reads, 0.071 sec, 30.9 kb/call avg, 0.0 msec/call avg
total 36 writes, 0.012 sec, 283.3 kb/call avg, 0.3 msec/call avg
In searchd.log found this error
[Wed Sep 30 09:09:29.371 2015] [ 4244] rotating index 'idx_prodotti_main': started
[Wed Sep 30 09:09:29.381 2015] [ 4244] WARNING: rotating index 'idx_prodotti_main': cur to old rename failed: rename D:\Sphinx\project\data\prodotti.spa to D:\Sphinx\project\data\prodotti.old.spa failed: Broken pipe
[Wed Sep 30 09:09:29.381 2015] [ 4244] rotating index: all indexes done
UPDATE 2
Also try to insert sleep between two commands
D:\Sphinx\bin\indexer.exe --config D:\Sphinx\project\product.conf idx_product_delta --rotate
timeout /t 60
D:\Sphinx\bin\indexer.exe --config D:\Sphinx\project\product.conf --merge idx_product_main idx_product_delta --rotate
Console output:
ERROR: index 'idx_prodotti_main': failed to delete 'D:\Sphinx\project\data\prodotti.new.spa': Permission deniedtotal 20137 reads, 0.072 sec, 30.9 kb/c
UPDATE 3: Issue solved
Issue solved by sphinx guys here
http://sphinxsearch.com/bugs/view.php?id=2335
The reason of such behavior is that --rotate command is asynchronous, so when you run second command:
D:\Sphinx\bin\indexer.exe --config D:\Sphinx\project\product.conf --merge idx_product_main idx_product_delta --rotate
first may continue to work with index idx_product_delta:
D:\Sphinx\bin\indexer.exe --config D:\Sphinx\project\product.conf idx_product_delta --rotate
, so it's locked.
If it's possible, remove --rotate option on first command.
UPDATE:
Seems that you need --rotate option in first command. So you could measure average time that need to make it done and insert sleep between two commands. For example, for 30 seconds:
D:\Sphinx\bin\indexer.exe --config D:\Sphinx\project\product.conf idx_product_delta --rotate
timeout /t 30
D:\Sphinx\bin\indexer.exe --config D:\Sphinx\project\product.conf --merge idx_product_main idx_product_delta --rotate

Say my input file is table delimited, how do i get the class name and the executable time using AWK

Say my input file is table delimited, how can get the class name and the executable time?
- class Apple.java executed by user ABC34 executable time 980.9 ms
- class ABC.java executed by user ABC34 executable time is in sec 45.9 ss
- class TAS.java executed by user ABC34 executable time 75.9 ms
- class GW.java executed by user ABC34 executable time is in sec 0.9 ss
- class MYClass.java executed by user ABC34 executable time is in sec 7.9 ss
- class Times.java executed by user ABC34 executable time is in sec 9.9 ss
- class Sunday.java executed by user ABC34 executable time is in sec 85.9 ss
- class Week.java executed by user ABC34 executable time is in sec 1000.9 ss
I want the out put like this
- Apple.java 980.9 ms
- ABC.java 45.9 ss
- TAS.java 75.9 ms
- GW.java 0.9 ss
- MYClass.java 7.9 ss
- Times.java 9.9 ss
- Sunday.java 85.9 ss
- Week.java 1000.9 ss
please see my code. i can't able to get out put
#!/bin/bash
LOGFILE=$1
SEARCH=$2
echo $SEARCH
if [ -f $LOGFILE ];
then
awk '/ABC34 / && /is in sec/ {print $2, $13, $14;}' $LOGFILE>Output.txt
awk '/ABC34 / {print $2, $8, $9;}' $LOGFILE>Output.csv
else
echo "No File Found"
fi
But when i check my Output.txt is empty. Many Thanks.
Using awk its a one-liner:
awk '{print $1,$3,$(NF-1),$NF}' your.file
NF contains the number of fields in a line. So the command prints the 1., the 3. the next to last and the last field.
Output:
- Apple.java 980.9 ms
- ABC.java 45.9 ss
- TAS.java 75.9 ms
- GW.java 0.9 ss
- MYClass.java 7.9 ss
- Times.java 9.9 ss
- Sunday.java 85.9 ss
- Week.java 1000.9 ss
Code for GNU sed:
sed -r 's/.*class\s(\S+)\s.*(\s[0-9.]+\s[ms][ms])$/- \1\2/' file
$ cat file
- class Apple.java executed by user ABC34 executable time 980.9 ms
- class ABC.java executed by user ABC34 executable time is in sec 45.9 ss
- class TAS.java executed by user ABC34 executable time 75.9 ms
- class GW.java executed by user ABC34 executable time is in sec 0.9 ss
- class MYClass.java executed by user ABC34 executable time is in sec 7.9 ss
- class Times.java executed by user ABC34 executable time is in sec 9.9 ss
- class Sunday.java executed by user ABC34 executable time is in sec 85.9 ss
- class Week.java executed by user ABC34 executable time is in sec 1000.9 ss
$ sed -r 's/.*class\s(\S+)\s.*(\s[0-9.]+\s[ms][ms])$/- \1\2/' file
- Apple.java 980.9 ms
- ABC.java 45.9 ss
- TAS.java 75.9 ms
- GW.java 0.9 ss
- MYClass.java 7.9 ss
- Times.java 9.9 ss
- Sunday.java 85.9 ss
- Week.java 1000.9 ss

Resources