I am trying to decode a message coming as part of avro message in my spark2.2 streaming. I have a schema defined for this json and whenever the json message comes with out honoring the json schema, my JsonDecoder fails with below error
Caused by: org.apache.avro.AvroTypeException: Expected field name not found: "some_field"
at org.apache.avro.io.JsonDecoder.doAction(JsonDecoder.java:477)
at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
at org.apache.avro.io.JsonDecoder.advance(JsonDecoder.java:139)
at org.apache.avro.io.JsonDecoder.readString(JsonDecoder.java:219)
at org.apache.avro.io.JsonDecoder.readString(JsonDecoder.java:214)
at org.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:201)
at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:422)
at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:414)
at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:181)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:232)
at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222)
at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:395)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:315)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:258)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:256)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1375)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:261)
I know jackson decoding has a way to ignore the extra as well as absent fields. Is there a way in org.apache.avro.io.JsonDecoder for the same behaviour?
Related
I'm having a user-generated input field that translates to a simple query string query in Elasticsearch. The code is picking up any invalid input in that field and presenting it for the user. But every time a user inputs something that is invalid, an exception is logged in the Elasticsearch log file like this:
Caused by: org.apache.lucene.queryparser.classic.ParseException: Cannot parse 'INVALID LIKE THIS: \");': Lexical error at line 1, column 94. Encountered: <EOF> after : "\");"
at org.apache.lucene.queryparser.classic.QueryParserBase.parse(QueryParserBase.java:114) ~[lucene-queryparser-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:52:15]
at org.elasticsearch.index.search.QueryStringQueryParser.parse(QueryStringQueryParser.java:793) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.query.QueryStringQueryBuilder.doToQuery(QueryStringQueryBuilder.java:1042) ~[elasticsearch-6.4.2.jar:6.4.2]
... 21 more
This is expected since this is an invalid query. My question is, is it possible to ignore these exceptions being logged. Since the user can write anything here, I don't want invalid queries to fill up my log since this is expected.
Disclamer: I know that this is a bit dangerous. Having free-text input is not preferred. And if I disable this exception, I will also miss if I make a mistake in the query generation or similar.
I'm running grails 4.0.1 attempting to use org.grails.plugins:elasticsearch:3.0.0 with elasticsearch 7.9.0. I'm not sure if I've misconfigured a bean along the way that I'd missed but any help in the right direction would be well received! Near ready to drop using the plugin if can't get it wired in my env but would like to keep up with it.
More complete trace contains
Caused by: org.springframework.beans.ConversionNotSupportedException: Failed to convert property value of type 'grails.core.support.proxy.DefaultProxyHandler' to required type 'grails.core.support.proxy.EntityProxyHandler' for property 'proxyHandler'; nested exception is java.lang.IllegalStateException: Cannot convert value of type 'grails.core.support.proxy.DefaultProxyHandler' to required type 'grails.core.support.proxy.EntityProxyHandler' for property 'proxyHandler': no matching editors or conversion strategy found
at org.springframework.beans.AbstractNestablePropertyAccessor.convertIfNecessary(AbstractNestablePropertyAccessor.java:590)
at org.springframework.beans.AbstractNestablePropertyAccessor.convertForProperty(AbstractNestablePropertyAccessor.java:604)
at org.springframework.beans.BeanWrapperImpl.convertForProperty(BeanWrapperImpl.java:219)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.convertForProperty(AbstractAutowireCapableBeanFactory.java:1723)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1679)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1426)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:592)
... 49 common frames omitted
Caused by: java.lang.IllegalStateException: Cannot convert value of type 'grails.core.support.proxy.DefaultProxyHandler' to required type 'grails.core.support.proxy.EntityProxyHandler' for property 'proxyHandler': no matching editors or conversion strategy found
at org.springframework.beans.TypeConverterDelegate.convertIfNecessary(TypeConverterDelegate.java:262)
at org.springframework.beans.AbstractNestablePropertyAccessor.convertIfNecessary(AbstractNestablePropertyAccessor.java:585)
Here is a screenshot of my tomcat spring boot error log. How do I read this?
Specifically,
1.) where do I read there is says "63 more"?
2.) Is the top-most error the most generic, or most specific? I'm thinking most generic because it says "caused by" and then another error.
Thanks.
First, get destination of log (console, file, database, ...) by find information in:
application.properties (for example: spring.log.**** = ...)
or log4j.properties/log4j.yml/log4j.yaml (if using log4j)
or log4j2.properties/log4j2.yml/log4j2.yaml (if using log4j2)
or logback.properties/logback.yml (if using logback)
and then you can get "63 more" there
It's the most specific
most specific error <== root cause is here
^ cause by: generic error
^ cause by: generic error
...
^ cause by: most generic error
The first line contains the exact reason for the error, such as this:
java.lang.NullPointerException
abc.def.ghi.handleRequest(MyController.java:30)
...
The more you go down, the more information you get about the root cause. Usually you can find the cause at the bottom of the log message containing the stack trace. In your case, the cause for the exception is "No converter found capable of converting from type ...".
I'm using ZlibDecompressor in Hadoop, but I'm getting incorrect header check exception.
This is how I instantiate it
ZlibDecompressor inflater = new ZlibDecompressor(ZlibDecompressor.CompressionHeader.DEFAULT_HEADER,1024);
inflater.setInput(bytesCompressed, 0, bytesCompressed.length);
And here is how I use it for decompression
inflater.decompress(bytesDecompressedBuffer,0,bufferSizeInBytes);
I'm using Hadoop 0.20.2.
What could be the problem and how to solve it?
Thanks
d31efcf42e83e76d3df76d38db5d3c141f76135e7417de41d44dc50b507a07b03a07a03ad40f75db7f00038d7df02177db9dbbd01f02e35ef7eb60f6f77dfaebde3a0b7f75036d41dc3dc00c4e40136e3b044e83ec5d35f01044f050841011000c0df4d3ae40ec1079078101f02dfcd40dfbef9df5ec4db8e45d37d85102d350b8001d79f7de8303ce7a045efdd75e35dfc03b036f3c0f5e43034d78dfadb9e7ad7d0750c10c30bce7a103d04ef4000dbde01dfdf7a0c20b907df7def9d80137ef8
The problem reported is that there is no valid zlib header in the first two bytes. The problem with the data is that it does not appear to have any deflate-compressed data anywhere in it, regardless of whether such data could be zlib wrapped, gzip wrapped, or raw.
I'm using tire and elasticsearch. The service has started using port 9200. However, it was returning 2 errors:
"org.elasticsearch.search.SearchParseException: [countries][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"query":{"query_string":{"query":"name:"}}}]]"
and
"Caused by: org.apache.lucene.queryParser.ParseException: Cannot parse 'name:': Encountered "<EOF>" at line 1, column 5."
So, I reinstalled elasticsearch and the service container. Service starts fine.
Now, when I search using tire I get no results when results should appear and I don't receive any error messages.
Does anybody have any idea how I might find out what is wrong, let alone fix it?
first of all, you don't need to reindex anything, in the usual cases. It depends how you installed and configured elasticsearch, but when you install and upgrade eg. with Homebrew, the data are persisted safely.
Second, no need to reinstall anything. The error you're seeing means just what it says on the tin: SearchParseException, ie. your query is invalid:
{"query":{"query_string":{"query":"name:"}}}
Notice that you didn't pass any query string for the name qualifier. You have to pass something, eg:
{"query":{"query_string":{"query":"name:foo"}}}
or, in Ruby terms:
Tire.index('test') { query { string "name:hey" } }
See this update to the Railscasts episode on Tire for an example how to catch errors due to incorrect Lucene queries.