I am new to the Druid.
I have done a local setup of druid in local and I am able to load files of json data in druid.
However when I am trying to upload a parquet file, it is giving Unexpected character exception.
I've installed Parquet and Avro extensions and I get following error in each case
$ >curl -X 'POST' -H 'Content-Type:application/json' -d #examples/wikipedia_hadoop_parquet_job.json http://localhost:8090/druid/indexer/v1/task
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>enter code here
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /druid/indexer/v1/task. Reason:
<pre> javax.servlet.ServletException: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('}' (code 125)): was expecting double-quote to start field name
at [Source: HttpInputOverHTTP#57df7615[c=2032,q=1,[0]=EOF,s=STREAM]; line: 1, column: 493]</pre></p>
<hr />Powered by Jetty:// 9.3.19.v20170502<hr/>
</body>
</html>
Below is the JSON config file
{
"type": "index_hadoop",
"spec": {
"ioConfig": {
"type": "hadoop",
"inputSpec": {
"type": "static",
"inputFormat": "org.apache.druid.data.input.parquet.DruidParquetInputFormat",
"paths": "example/wikipedia_list.parquet"
},
"metadataUpdateSpec": {
"type": "postgresql",
"connectURI": "jdbc:postgresql://localhost/druid",
"user" : "druid",
"password" : "asdf",
"segmentTable": "druid_segments"
},
},
"dataSchema": {
"dataSource": "wikipedia",
"parser": {
"type": "parquet",
"parseSpec": {
"format": "timeAndDims",
"timestampSpec": {
"column": "timestamp",
"format": "auto"
},
"dimensionsSpec": {
"dimensions": [
"page",
"language",
"user",
"unpatrolled"
],
"dimensionExclusions": [],
"spatialDimensions": []
}
}
},
"metricsSpec": [{
"type": "count",
"name": "count"
}, {
"type": "doubleSum",
"name": "deleted",
"fieldName": "deleted"
}, {
"type": "doubleSum",
"name": "delta",
"fieldName": "delta"
}],
"granularitySpec": {
"type": "uniform",
"segmentGranularity": "DAY",
"queryGranularity": "NONE",
"intervals": ["2013-08-30/2013-09-02"]
}
},
"tuningConfig": {
"type": "hadoop",
"workingPath": "tmp/working_path",
"partitionsSpec": {
"targetPartitionSize": 5000000
},
"leaveIntermediate": true
}
}
}
I am not able to figure out the issue. Let me know, if I am missing something.
Related
I try to implement NLog into my .NET Core Api web service.
I want to log to an Oracle database. All works well through an nlog.config XML file.
But the goal is to implement NLog config into appsettings.json and here problem occurs.
I get the error set in title:
LayoutRenderer cannot be found: 'aspnet-user-identity
My config file is like this :
"NLog": {
"autoReload": true,
"throwConfigExceptions": true,
"internalLogLevel": "info",
"internalLogFile": "c:/app/log/dev/internal-appsetting-nlog.txt",
"extensions": {
"NLog.Extensions.Logging": {
"assembly": [
"NLog.Extensions.Logging",
"NLog.Web.AspNetCore"
]
}
},
"variables": {
"var_logdir": "c:/app/log/dev"
},
"default-wrapper": {
"type": "AsyncWrapper",
"overflowAction": "Block"
},
"targets": {
"all-file": {
"type": "File",
"fileName": "${var_logdir}/nlog-all-${shortdate}.log",
"layout": {
"type": "JsonLayout",
"Attributes": [
{
"name": "timestamp",
"layout": "${date:format=o}"
},
{
"name": "level",
"layout": "${level}"
},
{
"name": "logger",
"layout": "${logger}"
},
{
"name": "message",
"layout": "${message:raw=true}"
},
{
"name": "properties",
"encode": false,
"layout": {
"type": "JsonLayout",
"includeallproperties": "true"
}
}
]
}
},
"db": {
"type": "Database",
"commandText": "INSERT INTO logtable (LOGLEVEL,LOGGER,MESSAGE,MACHINENAME,USERNAME,CALLSITE, THREADID,EXCEPTIONMESSAGE,STACKTRACE,SESSIONID) VALUES (:pLEVEL,:pLOGGER,:pMESSAGE,:pMACHINENAME, :pCALLSITE,:pTHREADID,:pEXCEPTIONMESSAGE,:pSTACKTRACE)",
"parameters": [
{
"name": "#pLEVEL",
"layout": "${level}"
},
{
"name": "#pLOGGER",
"layout": "${logger}"
},
{
"name": "#pMESSAGE",
"layout": "${message}"
},
{
"name": "#pMACHINENAME",
"layout": "${machinename}"
},
{
"name": "#pUSERNAME",
"layout": "${aspnet-user-identity}"
},
{
"name": "#pCALLSITE",
"layout": "${callsite:filename=true}"
},
{
"name": "#pTHREADID",
"layout": "${threadid}"
},
{
"name": "#pEXCEPTIONMESSAGE",
"layout": "${exception}"
},
{
"name": "#pSTACKTRACE",
"layout": "${stacktrace}"
},
{
"name": "#pSESSIONID",
"layout": "${aspnet-sessionid}"
}
],
"dbProvider": "Oracle.ManagedDataAccess.Client.OracleConnection, Oracle.ManagedDataAccess",
"connectionString": "xxxxxxxxxxxx"
}
},
"rules": [
{
"logger": "*",
"minLevel": "Trace",
"writeTo": "all-file"
},
{
"logger": "*",
"minLevel": "Trace",
"writeTo": "db"
},
{
"logger": "Microsoft.*",
"maxLevel": "Info",
"final": true
}
]
},
The internal debugger reports:
2019-10-09 16:48:48.6665 Info Adding target AsyncTargetWrapper(Name=all-file)
2019-10-09 16:48:48.7859 Warn Error when setting property 'Layout' on 'NLog.Targets.DatabaseParameterInfo' Exception: System.ArgumentException: LayoutRenderer cannot be found: 'aspnet-user-identity'. Is NLog.Web not included?
at NLog.Config.Factory`2.CreateInstance(String itemName)
at NLog.Layouts.LayoutParser.GetLayoutRenderer(ConfigurationItemFactory configurationItemFactory, String name)
at NLog.Layouts.LayoutParser.ParseLayoutRenderer(ConfigurationItemFactory configurationItemFactory, SimpleStringReader stringReader)
at NLog.Layouts.LayoutParser.CompileLayout(ConfigurationItemFactory configurationItemFactory, SimpleStringReader sr, Boolean isNested, String& text)
at NLog.Layouts.SimpleLayout.set_Text(String value)
at NLog.Internal.PropertyHelper.TryNLogSpecificConversion(Type propertyType, String value, Object& newValue, ConfigurationItemFactory configurationItemFactory)
at NLog.Internal.PropertyHelper.SetPropertyFromString(Object obj, String propertyName, String value, ConfigurationItemFactory configurationItemFactory)
Error occurs on ${aspnet-sessionid}. If I comment out both layout, everything works well.
I found different things on GitHub issue report but all I tried was a fail.
Could someone help?
The unknown aspnet-user-identity is probably an issue with your extensions:
"extensions": [
{ "assembly": "NLog.Extensions.Logging" },
{ "assembly": "NLog.Web.AspNetCore" }
],
Could you try the above suggestion?
P.S. Updated the wiki to include example of multiple "extensions"
When attempting to index the following document:
{
"branch": "master",
"classes": [
{
"content_count": 2,
"documentation": "",
"extends": [],
"generic": "",
"implements": [],
"line": 10,
"line_count": 36,
"modifiers": [
"public"
],
"name": "removeDuplicateFromString"
}
],
"commit_hash": "e53249ba2381d2f20f3d4493ad70e2da0abb3b05",
"contributors": [
{
"id": "7676016",
"name": "varunu28",
"url": "https://github.com/varunu28"
}
],
"enums": [],
"fields": [],
"filename": "removeDuplicateFromString.java",
"imports": [
{
"name": "java.io.BufferedReader",
"wildcard": false
},
{
"name": "java.io.InputStreamReader",
"wildcard": false
}
],
"interfaces": [],
"license": "",
"methods": [
{
"cyclomatic_complexity": 1,
"documentation": "",
"generic": "",
"line": 11,
"line_count": 9,
"modifiers": [
"public",
"static"
],
"name": "main",
"params": [
{
"name": "args",
"type": "String[]"
}
],
"parent": "removeDuplicateFromString",
"type_": "void"
},
{
"cyclomatic_complexity": 5,
"documentation": "",
"generic": "",
"line": 29,
"line_count": 16,
"modifiers": [
"public",
"static"
],
"name": "removeDuplicate",
"params": [
{
"name": "s",
"type": "String"
}
],
"parent": "removeDuplicateFromString",
"type_": "String"
}
],
"number_forks": 1695,
"number_stars": 4000,
"number_watchs": 394,
"package": "",
"path": "Others",
"repository": "TheAlgorithms/Java"
}
I get the following error:
{"error":{"root_cause":[{"type":"settings_exception","reason":"Failed to load settings from [{\"interfaces\":[],\"imports\":[{\"name\":\"java.io.BufferedReader\",\"wildcard\":false},{\"name\":\"java.io.InputStreamReader\",\"wildcard\":false}],\"package\":\"\",\"methods\":[{\"parent\":\"removeDuplicateFromString\",\"line_count\":9,\"line\":11,\"documentation\":\"\",\"name\":\"main\",\"cyclomatic_complexity\":1,\"modifiers\":[\"public\",\"static\"],\"params\":[{\"name\":\"args\",\"type\":\"String[]\"}],\"type_\":\"void\",\"generic\":\"\"},{\"parent\":\"removeDuplicateFromString\",\"line_count\":16,\"line\":29,\"documentation\":\"\",\"name\":\"removeDuplicate\",\"cyclomatic_complexity\":5,\"modifiers\":[\"public\",\"static\"],\"params\":[{\"name\":\"s\",\"type\":\"String\"}],\"type_\":\"String\",\"generic\":\"\"}],\"number_forks\":1695,\"classes\":[{\"implements\":[],\"line_count\":36,\"extends\":[],\"line\":10,\"documentation\":\"\",\"name\":\"removeDuplicateFromString\",\"content_count\":2,\"modifiers\":[\"public\"],\"generic\":\"\"}],\"repository\":\"TheAlgorithms/Java\",\"branch\":\"master\",\"commit_hash\":\"e53249ba2381d2f20f3d4493ad70e2da0abb3b05\",\"enums\":[],\"path\":\"Others\",\"license\":\"\",\"filename\":\"removeDuplicateFromString.java\",\"number_watchs\":394,\"contributors\":[{\"name\":\"varunu28\",\"id\":\"7676016\",\"url\":\"https://github.com/varunu28\"}],\"fields\":[],\"number_stars\":4000}]"}],"type":"settings_exception","reason":"Failed to load settings from [{\"interfaces\":[],\"imports\":[{\"name\":\"java.io.BufferedReader\",\"wildcard\":false},{\"name\":\"java.io.InputStreamReader\",\"wildcard\":false}],\"package\":\"\",\"methods\":[{\"parent\":\"removeDuplicateFromString\",\"line_count\":9,\"line\":11,\"documentation\":\"\",\"name\":\"main\",\"cyclomatic_complexity\":1,\"modifiers\":[\"public\",\"static\"],\"params\":[{\"name\":\"args\",\"type\":\"String[]\"}],\"type_\":\"void\",\"generic\":\"\"},{\"parent\":\"removeDuplicateFromString\",\"line_count\":16,\"line\":29,\"documentation\":\"\",\"name\":\"removeDuplicate\",\"cyclomatic_complexity\":5,\"modifiers\":[\"public\",\"static\"],\"params\":[{\"name\":\"s\",\"type\":\"String\"}],\"type_\":\"String\",\"generic\":\"\"}],\"number_forks\":1695,\"classes\":[{\"implements\":[],\"line_count\":36,\"extends\":[],\"line\":10,\"documentation\":\"\",\"name\":\"removeDuplicateFromString\",\"content_count\":2,\"modifiers\":[\"public\"],\"generic\":\"\"}],\"repository\":\"TheAlgorithms/Java\",\"branch\":\"master\",\"commit_hash\":\"e53249ba2381d2f20f3d4493ad70e2da0abb3b05\",\"enums\":[],\"path\":\"Others\",\"license\":\"\",\"filename\":\"removeDuplicateFromString.java\",\"number_watchs\":394,\"contributors\":[{\"name\":\"varunu28\",\"id\":\"7676016\",\"url\":\"https://github.com/varunu28\"}],\"fields\":[],\"number_stars\":4000}]","caused_by":{"type":"illegal_state_exception","reason":"only value lists are allowed in serialized settings"}},"status":500}
From which I've gathered that the main issues are either described in the part saying that:
{"type":"illegal_state_exception","reason":"only value lists are allowed in serialized settings"}}
Or:
"error":{"root_cause":[{"type":"settings_exception","reason":"Failed to load settings from [{\"interfaces\":[],\"imports\": ........
But I cannot find any information about this error or what it could be caused by. I've tried indexing both using a predefined index with mappings and to a non-existing index. Nothing seems to work.
Why can't I index this document?
It turns out that, as Farid mentioned in the comments section, I was using the wrong command when indexing from the command line.
The correct command to run is
curl -X POST -H 'Content-Type: application/json' [index location] -d [data]
Where the key is that you use POST and not PUT which is what I was doing.
Adding this for the ones using Kibana Dev Tools.
The key is to use an document type after an index name when adding the document
POST /{index name}/{document type}
{
request body (document) goes here.
}
My Structure created in liferay is as shown below:
{
"availableLanguageIds": [
"en_US"
],
"defaultLanguageId": "en_US",
"fields": [
{
"label": {
"en_US": "Image"
},
"predefinedValue": {
"en_US": ""
},
"style": {
"en_US": ""
},
"tip": {
"en_US": "Upload the image to be displayed in the Small Banner"
},
"dataType": "image",
"fieldNamespace": "ddm",
"indexType": "keyword",
"localizable": true,
"name": "smallImage",
"readOnly": false,
"repeatable": true,
"required": true,
"showLabel": true,
"type": "ddm-image",
"nestedFields": [
{
"label": {
"en_US": "Priority of Image"
},
"predefinedValue": {
"en_US": "0"
},
"style": {
"en_US": ""
},
"tip": {
"en_US": "Priority of the uploaded Image. Lower the number, higher the priority."
},
"dataType": "integer",
"fieldNamespace": "ddm",
"indexType": "keyword",
"localizable": true,
"name": "imagePriority",
"readOnly": false,
"repeatable": false,
"required": true,
"showLabel": true,
"type": "ddm-integer"
}
]
}
]
}
Now in the template I want to sort it using the priority field of the image. My template is as shown below:
<div id="example2" class="slider-pro">
<div class="sp-slides">
<#if smallImage.getSiblings()?has_content>
<#list smallImage.getSiblings()?sort_by(["properties","imagePriority"]) as cur_smallImage>
<#if cur_smallImage.getData()?? && cur_smallImage.getData() != "">
<div class="sp-slide">
<img class="sp-image" alt="${cur_smallImage.getAttribute("alt")}" src="${cur_smallImage.getData()}" data-src="${cur_smallImage.getData()}" data-retina="${cur_smallImage.getData()}" />
</div>
</#if>
</#list>
</#if>
</div>
</div>
I tried the sorting as ?sort_by("cur_smallImage.imagePriority") also by ?sort_by("imagePriority") but it didnt work. I want to display the images in the sequence of the priority specified.
Please help.
If the subvariable that you want to use for the sorting is on a deeper level (that is, if it is a subvariable of a subvariable and so on), then you can use a sequence as parameter, that specifies the names of the sub variables that lead down to the desired subvariable. For example:
<#assign members = [
{"name": {"first": "Joe", "last": "Smith"}, "age": 40},
{"name": {"first": "Fred", "last": "Crooger"}, "age": 35},
{"name": {"first": "Amanda", "last": "Fox"}, "age": 25}]>
Sorted by name.last:
<#list members?sort_by(['name', 'last']) as m>
- ${m.name.last}, ${m.name.first}: ${m.age} years old
</#list>
taken from:
https://freemarker.apache.org/docs/ref_builtins_sequence.html#ref_builtin_sort_by
So far I'm able to do swagger validation if the parameters are from "in": "body" or if the input expected is in a json format.
However, I can't find how to validate a simple string entered as formData.
Below is my swagger script (in json format)
v1swag = {
"cancels_post": {
"tags": ["/api/v1"],
"parameters": [
{
"name": "token",
"in": "formData",
"type": "string",
"required": True,
"description": "Cancels the provided token.",
}
],
"responses": {
"200": {
"description": "Success!",
}
}
}
}
I removed the schema as it seems to only work for "in": "body"
I've been searching the net but can't seem to find the light.
Though I will still be searching... Any hints would be greatly appreciated.
Thank you very much in advance.
A different source media type has to be consumed here. Specify "consumes" member to include media type of application/x-www-form-urlencoded.
v1swag = {
"cancels_post": {
"tags": ["/api/v1"],
"consumes": [
"application/x-www-form-urlencoded"
],
"parameters": [
{
"name": "token",
"in": "formData",
"type": "string",
"required": True,
"description": "Cancels the provided token.",
}
],
"responses": {
"200": {
"description": "Success!",
}
}
}
}
I'm following the tutorial "Extract Data from a Source File." I'm able to upload a file as described, but when I try to convert the file to SVF it fails with "TranslationWorker-InternalFailure." I get the same error with both a 3ds file created with Blender and an f3d created with Autodesk Fusion 360.
Here's my code (python):
r=requests.post(
'https://developer.api.autodesk.com/modelderivative/v2/designdata/job',
headers={
'authorization':'{0} {1}'.format(auth_type,auth_token)
},
json={
"input": {
"urn": urn_cube_base64,
},
"output": {
"formats": [
{
"type": "svf",
"views": [
"2d",
"3d"
]
}
]
}
}
)
print r.json()
while True:
r=requests.get('https://developer.api.autodesk.com/modelderivative/v2/designdata/{0}/manifest'.format(urn_cube_base64),
headers={
'authorization':'{0} {1}'.format(auth_type,auth_token)
},
)
print r.text
rj=r.json()
progress=rj['progress']
status=rj['status']
print rj['status'],progress
if status in ('success','failed','timeout'):
print json.dumps(rj,indent=2)
break
It produces the following output:
{
"hasThumbnail": "false",
"status": "failed",
"derivatives": [
{
"hasThumbnail": "false",
"status": "failed",
"name": "LMV Bubble",
"messages": [
{
"message": "Extractor error code -1073741819",
"code": "TranslationWorker-InternalFailure",
"type": "error"
}
],
"outputType": "svf",
"progress": "complete"
}
],
"region": "US",
"version": "1.0",
"progress": "complete",
"type": "manifest",
"urn": "dXJuOmFkc2sub2JqZWN0czpvcy5vYmplY3Q6dG1wX2J1Y2tldDEvY3ViZS4zZHM"
}