I consume data from a kafka topic (nested json) which has to be filtered based on a field value.
So I used the ExecuteScript processor to filter out records and transform some fields.
For filtering I used findAll function without using QueryRecord and it worked.
My question here is does this approach solve the same purpose as QueryRecord processor for filtering
because I'm NOT sure how to use query record when there are nested json objects from the incoming flowfiles.
I tried queryrecord but it threw an error:
SELECT * FROM FLOWFILE WHERE RPATH(order, '/orderDetail/orderId') = '1126'
Error:
No match found for function signature RPATH(<JavaType(...Record)>,<CHARACTER>)
org.apache.calcite.runtime.CalciteContextException
Sample Data:
{
"retail":{
"retailId":"6133",
"retailName":"Maveric"
},
"order":{
"orderDetail":{
"orderId":"1126",
"orderName":"NNDRFG"
}
}
}
RPATH_STRING is not the correct syntax, it is just RPATH.
See the additional details documentation for QueryRecord:
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.QueryRecord/additionalDetails.html
It has an example of JSON with nested fields.
Related
I am facing problems with nullable fields when trying to use the data returned when executing my query.
Schema
A simplified schema for demonstration purpose:
type Query {
members: [Instrument!]!
}
type Instrument {
series: SeriesType!
...
}
type SeriesType {
dividendYield: SeriesMethods
...
}
type SeriesMethods{
latest: Float!
...
}
There is nullable data that is resolved at dividendYield and a typical response I get when executing the query is shown in the following section.
Typical Response
A picture of the query and response can be seen here.
So what is the problem???
We are using this data to for visuals on a web app. The data is read in as an object such that you are able to access the data by data.series.dividendYield.latest. The problem comes in when dividendYield returns null as there is there no longer exists a latest field anymore.
For the client side it will always be necessary for the field to be present in the data, even if the parent resolver is null. Is it possible for the resolvers to be setup such that all children fields also return null rather than not appearing in the data response? Any other solutions would also be appreciated.
Basically I'm trying to spread a GraphQL field so I don't have as much nesting.
For instance:
query {
user(id: aasdfasdf) {
...address {
street
}
}
}
output
{
user: {
street: '111 Cool Street'
}
}
No. From the spec:
When querying an Object, the resulting mapping of fields are conceptually ordered in the same order in which they were encountered during query execution, excluding fragments for which the type does not apply and fields or fragments that are skipped via #skip or #include directives.
The structure of the response will always match the structure of the request. GraphQL does not inherently support any syntax to manipulate the structure of the returned response. In other words, you get what you request, and you can only request what is in the schema.
Although it's experimental, if you're using a javascript client, you could take a look at graphql-lodash. It plugs into existing clients and enables you to use custom directives to apply arbitrary transformations to your response.
I am dumping data in SOLR database. Earlier I was using Elastic Search and it was allowing me to store nested JSON objects.
Is there any way I can dynamically create nested JSON values when inserting in SOLR?
I am using JAVA as a backend language. My code is:
SolrInputDocument document = new SolrInputDocument();
document.addField("UUID", eventID);
document.addField("eventCategory", eventCategory);
.
.
.
.
document.addField("source", source);
I want something like this:
{
"UUID":"1",
"source":abcd,
"eventCategory": {
"event1":"a",
"event2":"b",
"event3":"c"
}
}
Solr is a collection of flat schema documents. You can add dynamic field to solr but it does not support nested JSON object.
But you can use nested documents specified in following resource.
https://lucene.apache.org/solr/guide/6_6/uploading-data-with-index-handlers.html
But querying on child documents is not as straightforward as ElasticSearch or MongoDB.
you can use addChildDocument to add the nested doc to the parent one. So code would be:
SolrInputDocument document = new SolrInputDocument();
document.addField("UUID", eventID);
document.addField("eventCategory", eventCategory);
...
SolrInputDocument child = new SolrInputDocument();
...
document.addChildDocument(child);
How can I reference two mongodb collections using spring data while the localField is of type ObjectId and foreignField is of type String?
ProjectionOperation convertId=Aggregation.project().and("_id").as("agentId");
LookupOperation activityOperation = LookupOperation.newLookup().
from("activity").
localField("agentId").
foreignField("agent_id").
as("activities");
Aggregation aggregation = Aggregation.newAggregation(convertId,activityOperation);
return mongoTemplate.aggregate(aggregation, "agents", AgentDTO.class).getMappedResults()
However, this doesn't return any records because of the type issue. Is it possible to implement $toString or $convert in ProjectionOperation? or what other options are there?
I was able to solve it by writing native mongodb aggregation operation in java code as described in MongoDB $aggregate $push multiple fields in Java Spring Data
After implementing this solution I was able to add native $addfields as follows:
AggregationOperation addField=new GenericAggregationOperation("$addFields","{ \"agId\": { \"$toString\": \"$_id\" }}");
given this json?
[
{
"CompanyId":20,
"CompanyName":"Walmart",
"CompanyContacts":[
{
"CompanyId":20,
"FirstName":"Bob",
"LastName":"Green",
"Email":"bob#test.com",
"Phone":"1234567",
"IsActive":false
}
]
}
]
The KendoUI datasource schema.Model does not currently support nested json or json with related entities. It needs flat data. Hopefully in the future the schema.Model will support mapping complex json to flat in the model definition. However you can still use complex data in the grid you just can't define it in a schema.Model definition.
The mapping is actually done in the field definitions of the grid.
In addition see schema docs you can parse your data using the schema.parse or schema.data functions to manually transform your nested data into flat data.
Here is a fiddle example with your data
{
field : "CompanyContacts[0].FirstName",
title: "First Name"
}
Also note, if you don't need parent record CompanyName and CompanyID since you have CompanyID in your CompanyContacts in the way your data is currently defined then you can use the data attribute of the schema to indicate the starting point of your records like so
schema : {
model: mySchema,
data: "CompanyContacts"
},