I'm having these json parsing errors from time to time:
2022-01-07T12:15:19,872][WARN ][logstash.filters.json ] Error parsing json
{:source=>"message", :raw=>" { the invalid json }", :exception=>#<LogStash::Json::ParserError: Unrecognized character escape 'x' (code 120)
Is there a way to get the :exception field in the logstash config file?
I opened the exact same thread on the elastic forum and got a working solution there. Thanks to #Badger on the forum, I ended up using the following raw ruby filter:
ruby {
code => '
#source = "message"
source = event.get(#source)
return unless source
begin
parsed = LogStash::Json.load(source)
rescue => e
event.set("jsonException", e.to_s)
return
end
#target = "jsonData"
if #target
event.set(#target, parsed)
end
'
}
which extracts the info I needed:
"jsonException" => "Unexpected character (',' (code 44)): was expecting a colon to separate field name and value\n at [Source: (byte[])\"{ \"baz\", \"oh!\" }\r\"; line: 1, column: 9]",
Or as the author of the solution suggested, get rid of the #target part and use the normal json filter for the rest of the data.
Related
I am using Sphinx 4.4.0 with napoleon extension (Google Doc String). I have this two problems
ARNING: Block quote ends without a blank line; unexpected unindent.
ERROR: Unexpected indentation.
I found something about it on the internet but can not fit this two my code. My problem is I even do not understand the messages. I do not see where the problem could be.
This is the code:
def read_and_validate_csv(basename, specs_and_rules):
"""Read a CSV file with respect to specifications about format and
rules about valid values.
Hints: Do not use objects of type type (e.g. str instead of "str") when
specificing the column type.
specs_and_rules = {
'TEMPLATES': {
'T1l': ('Int16', [-9, ' '])
},
'ColumnA': 'str',
'ColumnB': ('str', 'no answer'),
'ColumnC': None,
'ColumnD': (
'Int16',
-9, {
'len': [1, 2, (4-8)],
'val': [0, 1, (3-9)]
}
}
Returns:
(pandas.DataFrame): Result.
"""
This are the original messages:
.../bandas.py:docstring of buhtzology.bandas.read_and_validate_csv:11: WARNING: Block quote ends without a blank line; unexpected unindent.
.../bandas.py:docstring of buhtzology.bandas.read_and_validate_csv:15: ERROR: Unexpected indentation.
.../bandas.py:docstring of buhtzology.bandas.read_and_validate_csv:17: ERROR: Unexpected indentation.
.../bandas.py:docstring of buhtzology.bandas.read_and_validate_csv:19: WARNING: Block quote ends without a blank line; unexpected unindent.
.../bandas.py:docstring of buhtzology.bandas.read_and_validate_csv:20: WARNING: Block quote ends without a blank line; unexpected unindent.
reStructuredText is not Markdown, and indentation alone is not enough to demarcate the code block. reStructuredText calls this a literal block. Although the use of :: is one option, you might want to explicitly specify the language (overriding the default) with the use of the code-block directive.
Also I noticed that you have invalid syntax in your code block—a missing ) and extra spaces in your indentation—which could have caused those errors.
Try this.
def read_and_validate_csv(basename, specs_and_rules):
"""Read a CSV file with respect to specifications about format and
rules about valid values.
Hints: Do not use objects of type type (e.g. str instead of "str") when
specificing the column type.
.. code-block:: python
specs_and_rules = {
'TEMPLATES': {
'T1l': ('Int16', [-9, ' '])
},
'ColumnA': 'str',
'ColumnB': ('str', 'no answer'),
'ColumnC': None,
'ColumnD': (
'Int16',
-9, {
'len': [1, 2, (4-8)],
'val': [0, 1, (3-9)]
}
)
}
Returns:
(pandas.DataFrame): Result.
"""
When I run this code:
$client->evaluate('
box.session.settings.error_marshaling_enabled = false
box.error{code = 42, reason = "Foobar", type = "MyError"}
');
regardless of the value of error_marshaling_enabled I always get a response with a new (extended) error format:
[
49 => 'Foobar',
82 => [
0 => [
0 => [
0 => 'CustomError',
2 => 3,
1 => 'eval',
3 => 'Foobar',
4 => 0,
5 => 42,
6 => [
'custom_type' => 'MyError',
],
],
],
],
],
Why is that?
Short answer.
error_marshaling_enabled option affects only how error objects are encoded in response body (48, IPROTO_DATA). It does not affect how they are returned as exceptions, in the response header (82, IPROTO_ERROR).
Long answer.
In Tarantool an error object can be returned in 2 ways: as an exception and as an object. For example, this is how to throw an error as exception:
function throw_error()
box.error({code = 1000, reason = "Error message"})
-- Or
error('Some error string')
end
This is how to return it as an object:
function return_error()
return box.error.new({code = 1000, reason = "Error message"})
end
If the function was called remotely, using IPROTO protocol via a connector like netbox, or PHP connector, or any other one, the error return way affects how it is encoded into MessagePack response packet. When the function throws, and the error reaches the top stack frame without being caught, it is encoded as IPROTO_ERROR (82) and IPROTO_ERROR_24 (49).
When the error object is returned as a regular value, not as an exception, it is encoded also as a regular value, inside IPROTO_DATA (48). Just like a string, a number, a tuple, etc.
With encoding as IPROTO_ERROR/IPROTO_ERROR_24 there is no much of a configuration space. Format of these values can't be changed. IPROTO_ERROR is always returned as a MessagePack map, with a stack of errors in it. IPROTO_ERROR_24 is always an error message. The IPROTO_ERROR_24 field is kept for compatibility with connectors to Tarantool versions < 2.4.1.
With encoding as a part of IPROTO_DATA you can choose serialization way using error_marshaling_enabled option. When it is true, errors are encoded as MessagePack extension type MP_EXT, and contain the whole error stack, encoded exactly like IPROTO_ERROR value. When the option is false (default behaviour in 2.4.1), the error is encoded as a string, MP_STR, which is the error's message. If there is a stack of errors, only the newest error is encoded.
error_marshaling_enabled option exists for backward compatibility, in case your application on Tarantool wants to be compatible with old connectors, which don't support MP_EXT encoded errors.
In Tarantool < 2.4.1 errors were encoded into result MessagePack as a string with error message, and error stacks didn't exist at all. So when the new format and the error stacks feature were introduced, making the new format default would be a too radical change breaking the old connectors.
Consider these examples of how error marshaling affects results. I use Tarantool 2.4.1 console here, and built-in netbox connector. The code below can be copy pasted into the console.
First instance:
box.cfg{listen = 3313}
box.schema.user.grant('guest', 'super')
function throw_error()
box.error({code = 1000, reason = "Error message"})
end
function return_error()
return box.error.new({code = 1000, reason = "Error message"})
end
Second instance:
netbox = require('net.box')
c = netbox.connect(3313)
Now I try to call the function on the second instance:
tarantool> c:call('throw_error')
---
- error: Error message
...
The c:call('throw_error') threw an exception. If I catch it using pcall() Lua function, I will see the error object.
tarantool> ok, err = pcall(c.call, c, 'throw_error')
tarantool> err:unpack()
---
- code: 1000
base_type: ClientError
type: ClientError
message: Error message
trace:
- file: '[string "function throw_error()..."]'
line: 2
...
As you can see, I didn't set error_marshaling_enabled, but got the full error. Now I will call the other function, without exceptions. But the error object won't be full.
tarantool> err = c:call('return_error')
tarantool> err
---
- Error message
...
tarantool> err:unpack()
---
- error: '[string "return err:unpack()"]:1: attempt to call method ''unpack'' (a nil
value)'
...
The error was returned as a mere string, error message. Not as an error object. Now I will turn on the marshaling:
tarantool> c:eval('box.session.settings.error_marshaling_enabled = true')
---
...
tarantool> err = c:call('return_error')
---
...
tarantool> err:unpack()
---
- code: 1000
base_type: ClientError
type: ClientError
message: Error message
trace:
- file: '[C]'
line: 4294967295
...
Now the same function returned the error in the new format, more featured.
On the summary: error_marshaling_enabled affects only returned errors. Not thrown errors.
I am using Telegraf to get logs information from Apache NiFi, for this task I am using this config:
[[inputs.tail]]
## files to tail.
files = ["/var/log/nifi/nifi-app.log"]
## Read file from beginning.
from_beginning = true
#name_override = "nifi_app"
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
data_format = "grok"
grok_patterns = [ "%{DATE:date} %{TIME:time} %{WORD:EventType} \[%{GREEDYDATA:NifiTask} %{NOTSPACE:Thread}\] %{NOTSPACE:NifiEventType} %{GREEDYDATA:EventText} %{NUMBER:EventDuration} %{WORD:EventDurationUnits}" ]
When I try to start telegraf it give me this error:
Error parsing /etc/telegraf/telegraf.conf, toml: line 10: parse error
The pattern I wrote was tested in a Grok debugger with this text:
2018-08-02 10:53:16,976 INFO [Heartbeat Monitor Thread-1]
o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats
in 11863 nanos
These are the results of some testing:
grok_patterns = ["\[%{GREEDYDATA:NifiTask}\]"] ==> toml: line 10: parse error
grok_patterns = ["[%{GREEDYDATA:NifiTask}]"] ==> Invalid data format: grok
grok_patterns = ['\[%{GREEDYDATA:NifiTask}\]'] ==> Invalid data format: grok
grok_patterns = ["\\[%{GREEDYDATA:NifiTask}\\]"] ==> Invalid data format: grok
grok_patterns = ['[%{GREEDYDATA:NifiTask}]'] -> Invalid data format: grok
The first option for me is the right one, but doesn't works, and the problem seems to be the way the bracket is being escaped.
How is possible to solve this issue?
To fix the issue about escaping bracket, a "partial" solution is to change double quote by simple quote, with this way, in my case (telegraph version 1.13.4) the bracket is correctly escaped by \
There was more than one problem:
First problem: the grok dataformat is added to Telegraf in the 1.8 release (ref), so I must use a nightly install until this version is released.
Second problem: how to escape the brackets, there are problems doing it in a regular way, so what I finally did was to put this part in a custom pattern file, this way it works perfectly.
I am trying to parse error logs using Logstash to capture few fields especially errormessage. But unable to capture errormessage in Logstash. Below is the actual error message and parser which I wrote
12345 http://google.com 2017-04-17 09:02:43.065 ERROR 10479 --- [http-nio-8052-exec-2] com.utilities.TokenUtils : Error
org.xml.SAXParseException: An invalid XML character (Unicode: 0xe) was found in the value of attribute "ID" and element is "saml".
at org.apache.parsers.DOMParser.parse(Unknown Source)
at org.apache.jaxp.DocumentBuilderImpl.parse(Unknown Source)
at javax.parsers.DocumentBuilder.parse(DocumentBuilder.java:121)
at com.utilities.TokenUtils.validateSignature(TokenUtils.java:99)
Parser:
`%{NOTSPACE:stnum}\s*%{NOTSPACE:requestURL}\s*%{TIMESTAMP_ISO8601:log_timestamp}\s*%{LOGLEVEL:loglevel}\s*%{NUMBER:pid}\s*---\s*\[(?<thread>[A-Za-z0-9-]+)\]\s*%{DATA:class}\s*:\s%{NOTSPACE:level}\s*(?<errormessage>.[^\n]*).[^\n]*`
I am trying to capture this message from the log:
org.xml.SAXParseException: An invalid XML character (Unicode: 0xe) was found in the value of attribute "ID" and element is "saml".
Which logstash parser you are using? Please provide while conf file which can give us more info. Here's the sample to parse exception type from your logs (Using grok filter).
filter {
grok {
match => ["message", "%{DATA:errormessage} %{GREEDYDATA:EXTRA}"]
}
}
I'm building a new XML document using the following code:
doc = Nokogiri::XML::Builder.new(:encoding => 'UTF-8') do
reginfo {
client_type_id client_type == :associate ? ASSOCIATE : CLIENT
parent_ {
id_ client_type == :associate ? associate_id : customer_id
client_type_id client_type == :associate ? ASSOCIATE : CLIENT
vsn client_type == :associate ? associate_vsn : customer_vsn
}
}
end
The tags above listed as client_type_id show up in the XML file like client_type_id, but that is not the correct XML tag name format. I need them to be client-type-id.
I tried replacing the line with:
:"client-type-id" client_type == :associate ? ASSOCIATE : CLIENT
but I get:
syntax error, unexpected tIDENTIFIER, expecting '}'
:"client-type-id" client_type == :associate ? ASSOCIATE : CLIENT
^
or:
:"client-type-id" (client_type == :associate ? ASSOCIATE : CLIENT)
but I get:
syntax error, unexpected '(', expecting '}'
:"client-type-id" (client_type == :associate ? ASSOCIATE : CLIENT)
^
Is there a way to tell Nokogiri::XML::Builder:
on a per-line basis, to use dashes for a tag name and does it have a different syntax?
that for the entire document, during creation or after, to use dashes for all underscores in element names so that it forms correctly formatted XML element names?
There is a fancy way of doing this:
Nokogiri::XML::Builder.new(:encoding => 'UTF-8') do |xml|
...
xml.send :"client-type-id", (client_type == :associate ? ASSOCIATE : CLIENT)
...
end