I wrote some liblognormalize rules to parse postgresql logs.
My rule file contains two rules and each rule has some tags like:
rule=POSTGRESQL,CHECKPOINT: ....
rule=POSTGRESQL,SLOWQUERY: ....
After running mmnormalize in my rsyslog configuration, I would like to know which rule actually matched the log line being processed. The simplest solution would be to get the tags. I know that mmnormalize export some variables like $parsesuccess. Is there any variable containing the tags of the rule used ?
I do not know where I found it in the docs but in a rsyslog config file that I wrote some time ago I found that I can access the list of tags that liblognorm assigns to a message via event.tags in rsyslog.
I have for example
template( # ...
property(name="$!event.tags")
)
or
if $!event.tags != "" then { # ...
The solution seems to be the annotate feature:
rule=POSTGRESQL,CHECKPOINT: ....
annotate=CHECKPOINT:+checkpoint="complete"
Basically the annotate line will add a field checkpoint containing the value complete to all the log lines matching the rules with a tag CHECKPOINT
Found it here
Related
I try to write a map in y yaml file and want to use it in yaml file itself with a key. for example ->
in my yaml file I wrote -
pod_env_map:
pod1: "env1"
pod2: "env2"
pod3: "env3"
and in this yaml file itself I'm getting a var $pod_name.
So now i want to write stages with some command for all pod and correspond env.
So instead of putting multiple checks of $pod_name == 'pod1'....
I want to use above dict somehow in code like pod_env_map[$pod_name], please help me to know what should be the syntax for this? I tried to find but didn't get relavent info anywhere.
I'm trying to create a temporary file in my pipeline, then use that file in another rule.
For example, I have two rules in a .smk file:
#Unzip adapter trimmed fastq file
rule unzip_fastq:
input:
'{sample}.adapterTrim.round2.fastq.gz',
output:
temp('{sample}.adapterTrim.round2.fastq')
conda:
'../envs/rep_element.yaml'
shell:
'gunzip -c {input[0]} > {output[0]}'
#Run bowtie2 to align to rep elements and parse output
rule parse_bowtie2_output_realtime:
input:
'{sample}.adapterTrim.round2.fastq'
output:
'rep_element_pipeline/{sample}.fastq.gz.mapped_vs_' + config["ref"]["bt2_index"] + '.sam'
params:
bt2=config["ref"]["bt2_index_path"], eid=config["ref"]["enst2id"]
conda:
'../envs/rep_element.yaml'
shell:
'perl ../scripts/parse_bowtie2_output_realtime_includemultifamily.pl '
'{input[0]} {params.bt2} {output[0]} {params.eid}'
{sample}.adapterTrim.round2.fastq is used once and should ultimately be deleted upon completion. However, I'm finding that this file is uploaded to Amazon S3, even with the addition of temp(). I'm also finding that this file is removed locally, but still persists on S3.
Am I doing this correctly? '{sample}.adapterTrim.round2.fastq' is not currently written in the rule-all of the Snakefile.
We ultimately need to prevent this file from being uploaded to S3, so if there is a way to specify not to upload this file in the rule, that would be useful.
It seems that the snippet in the question is not consistent with actual use, since for S3 files one would need to wrap file names in remote.
However, as a general solution, documentation contains the following:
The remote() wrapper is mutually-exclusive with the temp() and protected() wrappers.
Hence, if you intend to use a temp file, make sure it's not wrapped in remote, or explicitly wrap the file in local.
I have a few lines of configurations that I need in my rsyslgog.
if $programname == 'project' then /var/log/file.log
When added to the end of the main rsyslog configuration file, /etc/rsyslog.conf, this configuration appears to be valid and functional.
However, when using the rsyslog.d directory I get a syntax error.
error during parsing file /etc/rsyslog.d/project.conf, on or before line 2: syntax error on token '==' [v8.32.0 try http://www.rsyslog.com/e/2207 ]
Is there anything in the main config that has to be parsed in advance, or is this a bug that should be reported to Fedora 27 developers?
As rsyslog author, I would assume that there is some include right in front of it that somehow renders your (valid) construct invalid. Red Hat unfortunately tends to stick to obsolete legacy format, and things like these can easily happen when this is used (after all, this was why we obsoleted it).
To hunt this down, I would check the config include that is included immediately in front of your own. If included via wildcards, the include order is sorted by filename.
Sorry, it was my bad. The configuration for my rsyslog config file was rewritten by my installer bash script, and that interpreted the $ sign as variable within the string. I should have double-checked the correctness of my config file.
I'd like to change set various configuration options for Marathon, and I'm unsure how to do this. For example, I'd like to add --event_subscriber http_callback to the launch command.
For each configuration you can create a file in the directory /etc/default/marathon where the filename is the name of the option and a single line in the file containing the value for that option.
Eg. Make a file /etc/default/marathon/event_subscriber that contains the line "http_callback"
This also works for mesos.
Actually the directory has to be /etc/marathon/conf. This is valid for 0.11.1.
I'm using Sphinx on Windows.
Most of my documentation is for regular users, but there are some sub-pages with content for administrators only.
So I want to build two versions of my documentation: a complete version, and a second version with the "admin" pages excluded.
I used the exclude_patterns in the build configuration for that.
So far, it works. Every file in every subfolder whose name contains "admin" is ignored when I put this into the conf.py file:
exclude_patterns = ['**/*admin*']
The problem is that I'd like to run the build once to get both versions.
What I'm trying to do right now is running make.bat twice and supply different parameters on each run.
According to the documentation, I can achieve this by setting the BUILDDIR and SPHINXOPTS variables.
So now I have a build.bat that looks like this:
path=%path%;c:\python27\scripts
rem BUILD ADMIN DOCS
set SPHINXOPTS=
set BUILDDIR=c:\build\admin
call make clean
call make html
rem BUILD USER DOCS
set SPHINXOPTS=-D exclude_patterns=['**/*admin*']
set BUILDDIR=c:\build\user
call make clean
call make html
pause
The build in the two different directories works when I delete the line set BUILDDIR=build from the sphinx-generated make.bat file.
However, the exclude pattern does not work.
The batch file listed above outputs this for the second build (the one with the exclude pattern):
Making output directory...
Running Sphinx v1.1.3
loading translations [de]... done
loading pickled environment... not yet created
Exception occurred:
File "C:\Python27\lib\site-packages\sphinx-1.1.3-py2.7.egg\sphinx\environment.
py", line 495, in find_files
['**/' + d for d in config.exclude_dirnames] +
TypeError: coercing to Unicode: need string or buffer, list found
The full traceback has been saved in c:\users\myusername\appdata\local\temp\sphinx-err-kmihxk.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
Either send bugs to the mailing list at <http://groups.google.com/group/sphinx-dev/>,
or report them in the tracker at <http://bitbucket.org/birkenfeld/sphinx/issues/>.
What am I doing wrong?
Is the syntax for exclude_patterns in the sphinx-build command line different than in the conf.py file?
Or is there a better way to build two different versions in one step?
My first thought was that this was a quoting issue, quoting being notoriously difficult to get right on the Windows command line. However, I wasn't able to come up with any combination of quoting that changed the behavior at all. (The problem is easy to replicate)
Of course it could still just be some quoting issue I'm not smart enough to figure out, but I suspect this is a Sphinx bug of some kind, and hope you will report it to the Sphinx developers.
In the meantime, here's an alternate solution:
quoting from here:
There is a special object named tags available in the config file. It can be used to query and change the tags (see Including content based on tags). Use tags.has('tag') to query, tags.add('tag') and tags.remove('tag') to change
This allows you to essentially pass flags into the conf.py file from the command line, and since the conf.py file is just Python, you can use if statements to set the value of exclude_patterns conditionally based on the tags you pass in.
For example, you could pass Sphinx options like:
set SPHINXOPTS=-t foradmins
to pass the "foradmins" tag, and then check for it in your conf.py like so:
exclude_patterns = blah
if tags.has('foradmins'):
exclude_patterns = []
That should allow you to do what you want. Good Luck!