I have a log that i want to filter out and drop messages that do not contain two strings
It works well when i use one string with the filter
if ([message] !~ "Low Volts ") {
drop{}
}
I am trying to drop lines that do not contain either of two strings "Low Volts" and 'Charge". I am trying this;
if ([message] !~ "Low Volts " or [message] !~ "Charge Fail") {
drop{}
}
the above does not work
Any help?
That conditional says if the message does not contain "Low Volts" or if it does not contain "Charge Fail". Thus it is only true if [message] contains both. Try
if ([message] !~ "Low Volts " and [message] !~ "Charge Fail") {
That will be true only if [message] contains neither.
Related
if ($input eq "TEST/TEST")
then "My Value"
else if ($input ne "")
then "My Value2"
else ()
Is there a way to set up the above using conditionals with Regex and the replace() function? Basically if my input equals "SLT/SLT", I want my new output to be "My Value". For any other non-empty field value, I would want the output to be "My Value 2". For all empty string inputs I would want it to stay empty.
Split the text values in a column based on the data type of the first character in each record.
I need to have the new (custom) column return the text before the first " " delimiter if the first character of the text is a number, otherwise return "0,".
If Value.Is(Text.Start([ConsumerAddress],1), type number) Then
Text.BeforeDelimiter([ConsumerAddress]," ") else "0,"
I need to have the new (custom) column return the text before the first " " delimiter if the first character of the text is a number, otherwise return "0,".
I don't think Value.Is is quite what you want. I would recommend a try otherwise construction along with Number.FromText like this:
= Table.AddColumn(#"Previous Step", "Custom",
each try Number.FromText(Text.BeforeDelimiter([ConsumerAddress], " "))
otherwise 0
)
If the text before the first space can be converted to a number, then that's what you get. If it can't the Number.FromText throws an error and you get the 0 from the otherwise specification.
Edit: If you want the criterion for the first character only, try this:
= Table.AddColumn(#"Previous Step", "Custom",
each if (try Number.FromText(Text.Start([ConsumerAddress], 1)) otherwise 0) <> 0
then Text.BeforeDelimiter([ConsumerAddress], " ")
else "0"
)
This will return "12b" from "12b Maple St" whereas the first version would return 0 since "12b" can't be converted into a number.
I can distinguish the "msg" field in logstash in the following format
filter {
kv {
field_split => "|"
source => "msg"
}
}
Properly seperated.
But then the reserved area "latitude" is not processed
Adding as string
" deviceValue" => "null ",
**"test1" => "%{latitude}"**,
" timeLabel" => "NOON ",
" appllicationName" => "null ",
" longitude" => "29.08222 ",
Thank you for your help
Take a closer look to the parsed values. I believe they are not in fact properly separated. You have spaces in the source data surrounding your split character "|", so when it is parsed you actually don't get a field named "latitude" but " latitude".
From your post:
" longitude" => "29.08222 ",
Do you see the leading space on " longitude" and the trailing one in the value?
I assume you don't need those, so one way to resolve the problem would be to clean the whitespace from the source data and then use your existing scripts.
Alternatively, if you cannot modify the source data, you can set your filter to split on " | ":
filter {
kv {
field_split => " | "
source => "msg"
}
}
And finally, if you indeed need those spaces and cannot change that, you can change "%{latitude}" to "%{ latitude}".
I have an elasticsearch index which I am using to index a set of documents.
These documents are originally in csv format and I am looking parse these using logstash.
My problem is that I have something along the following lines.
field1,field2,field3,xyz,abc
field3 is something like 123456789 and I want to parse it as 4.56(789) using ruby code filter.
My try:
I tried with stdin and stdout with the following logstash.conf .
input {
stdin {
}
}
filter {
ruby {
code => "
b = event["message"]
string2=""
for counter in (3..(num.size-1))
if counter == 4
string2+= '_'+ num[counter]
elsif counter == 6
string2+= '('+num[counter]
elsif counter == 8
string2+= num[counter] +')'
else
string2+= num[counter]
end
end
event["randomcheck"] = string2
"
}
}
output {
stdout {
codec=>rubydebug
}
}
I am getting syntax error using this.
My final aim is to use this with my csv file , but first I was trying this with stdin and stdout.
Any help will be highly appreciated.
The reason you're getting a syntax error is most likely because you have unescaped double quotes inside the double quoted string. Either make the string single quoted or keep it double quoted but use single quotes inside. I also don't understand how that code is supposed to work.
But that aside, why use a ruby filter in the first place? You can use a csv filter for the CSV parsing and a couple of standard filters to transform 123456789 to 4.56(789).
filter {
# Parse the CSV fields and then delete the 'message' field.
csv {
remove_field => ["message"]
}
# Given an input such as 123456789, extract 4, 56, and 789 into
# their own fields.
grok {
match => [
"column3",
"\d{3}(?<intpart>\d)(?<fractionpart>\d{2})(?<parenpart>\d{3})"
]
}
# Put the extracted fields together into a single field again,
# then delete the temporary fields.
mutate {
replace => ["column3", "%{intpart}.%{fractionpart}(%{parenpart})"]
remove_field => ["intpart", "factionpart", "parenpart"]
}
}
The temporary fields have really bad names in the example above since I don't know what they represent. Also, depending on what the input can look like you may have to adjust the grok expression. As it stands now it assumes nine-digit input.
I have the following linq expression to do my ordering but was wondering how do I change this so that it will orderby name but ignore the first word if it is "the"
CaseStudies.OrderBy(a => a.Name)
Simplest way (if there is always lower-case the and no more than one space between words):
CaseStudies.OrderBy(a => a.Name.StartsWith("the ") ? a.Name.Substring(4) : a.Name)
You can create method with nice descriptive name and move this logic as well as null check and ignore case comparison there:
private string RemoveDefiniteArticle(string s)
{
if (String.IsNullOrEmpty(s))
return s;
if (s.StartsWith("the ", StringComparison.CurrentCultureIgnoreCase))
return s.Substring(4).TrimStart();
return s;
}
And use it
CaseStudies.OrderBy(a => RemoveDefiniteArticle(a.Name))
There are a surprising number of edge cases here. Suppose your list is
List<string> strings = new List<string> { "The aardvark", "the bear", "The cat", " dog", " elephant"};
Then the starting point is handling "the" at the start
strings.OrderBy(w => w.StartsWith("the ") ? w.Substring(4) : w);
Which gives:
elephant
dog
the bear
The cat
The aardvark
Ignoring case is better
strings.OrderBy(w => w.StartsWith("the ", StringComparison.CurrentCultureIgnoreCase) ? w.Substring(4) : w);
Giving:
elephant
The cat
dog
The aardvark
the bear
Handling multiple spaces after the leading "the" is even better, but not perfect:
strings.OrderBy(w => w.StartsWith("the ", StringComparison.CurrentCultureIgnoreCase) ? w.Substring(4).TrimStart() : w);
elephant
dog
The aardvark
the bear
The cat
Handling leading spaces before the leading "the" looks correct
strings.OrderBy(w => w.TrimStart().StartsWith("the ", StringComparison.CurrentCultureIgnoreCase) ? w.TrimStart().Substring(4).TrimStart() : w.TrimStart());
Gives:
The aardvark
the bear
The cat
dog
elephant
But there may be other edge cases around null/empty/whitespace checking at multiple points...
CaseStudies.OrderBy(a => a.Name.TrimStart().StartsWith("the ", StringComparison.CurrentCultureIgnoreCase) ? a.Name.TrimStart().Substring(4).TrimStart() : a.Name)