I am dumping data in SOLR database. Earlier I was using Elastic Search and it was allowing me to store nested JSON objects.
Is there any way I can dynamically create nested JSON values when inserting in SOLR?
I am using JAVA as a backend language. My code is:
SolrInputDocument document = new SolrInputDocument();
document.addField("UUID", eventID);
document.addField("eventCategory", eventCategory);
.
.
.
.
document.addField("source", source);
I want something like this:
{
"UUID":"1",
"source":abcd,
"eventCategory": {
"event1":"a",
"event2":"b",
"event3":"c"
}
}
Solr is a collection of flat schema documents. You can add dynamic field to solr but it does not support nested JSON object.
But you can use nested documents specified in following resource.
https://lucene.apache.org/solr/guide/6_6/uploading-data-with-index-handlers.html
But querying on child documents is not as straightforward as ElasticSearch or MongoDB.
you can use addChildDocument to add the nested doc to the parent one. So code would be:
SolrInputDocument document = new SolrInputDocument();
document.addField("UUID", eventID);
document.addField("eventCategory", eventCategory);
...
SolrInputDocument child = new SolrInputDocument();
...
document.addChildDocument(child);
Related
I want to write a mongo query to fetch All Documents in a collection and project just 2 fields and no criteria.
I want to use streams here and iterate over the list of documents to process them.
How can I write this in Java Spring Mongo?
I have something like this:
public CloseableIterator<Employees> streamAllEmployees() {
Query query = new Query();
query.fields().include("_id.name","_id.dept");
query.addCriteria(Criteria.where("_id.dept").is(1));
return mongoOperations.stream(query, Employee.class);
}
but how can I change this to not have any QueryCriteria?
I'm using the Yii2 class yii\elasticsearch\ActiveRecord to recieve data from elastic search. Usually methods in this class for getting data from elastic will return that data as ActiveRecord (AR) object. So it's easy, to create an activeDataProvider from that AR to fill that data into a listview, etc.
But: yii\elasticsearch\ActiveRecord::mget() does not return an AR-object. Instead it return an array of documents.
My questions:
1.) Is there a way to use the mget - feature / elastic multi get feature AND get the result as AR object?
OR
2.) Is there a way to bring that array of documents into an AR object to make ActiveDataProvider including listview working?
I found the following solution:
Runnig Yii2 mget with an list of the ID's I like to get, which returns an array of docs.
Creating an ActiveDataProvider using ArrayDataProvider Class and the mget-output from step before
Example Code:
use app\models\MyModel;
use yii\data\ArrayDataProvider;
$ids = ['123','456','789'];
$myModel = new MyModel(); # Data Model based on yii\mongodb\ActiveRecord;
$result = $myModel->mget($ids); # get documents from elastic where document id is in $ids
$dataProvider = new ArrayDataProvider([
'allModels' => $result
]); # creating ActiveDataProvider, which can be used in Listviews.
I consume data from a kafka topic (nested json) which has to be filtered based on a field value.
So I used the ExecuteScript processor to filter out records and transform some fields.
For filtering I used findAll function without using QueryRecord and it worked.
My question here is does this approach solve the same purpose as QueryRecord processor for filtering
because I'm NOT sure how to use query record when there are nested json objects from the incoming flowfiles.
I tried queryrecord but it threw an error:
SELECT * FROM FLOWFILE WHERE RPATH(order, '/orderDetail/orderId') = '1126'
Error:
No match found for function signature RPATH(<JavaType(...Record)>,<CHARACTER>)
org.apache.calcite.runtime.CalciteContextException
Sample Data:
{
"retail":{
"retailId":"6133",
"retailName":"Maveric"
},
"order":{
"orderDetail":{
"orderId":"1126",
"orderName":"NNDRFG"
}
}
}
RPATH_STRING is not the correct syntax, it is just RPATH.
See the additional details documentation for QueryRecord:
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.QueryRecord/additionalDetails.html
It has an example of JSON with nested fields.
How can I reference two mongodb collections using spring data while the localField is of type ObjectId and foreignField is of type String?
ProjectionOperation convertId=Aggregation.project().and("_id").as("agentId");
LookupOperation activityOperation = LookupOperation.newLookup().
from("activity").
localField("agentId").
foreignField("agent_id").
as("activities");
Aggregation aggregation = Aggregation.newAggregation(convertId,activityOperation);
return mongoTemplate.aggregate(aggregation, "agents", AgentDTO.class).getMappedResults()
However, this doesn't return any records because of the type issue. Is it possible to implement $toString or $convert in ProjectionOperation? or what other options are there?
I was able to solve it by writing native mongodb aggregation operation in java code as described in MongoDB $aggregate $push multiple fields in Java Spring Data
After implementing this solution I was able to add native $addfields as follows:
AggregationOperation addField=new GenericAggregationOperation("$addFields","{ \"agId\": { \"$toString\": \"$_id\" }}");
I want to be able to insert documents and preferably map all inner objects to nested ones automatically. Is this possible?
My specific use case is that I am collecting documents of the same type that may or may not have the same fields of those currently in the store. So I would prefer if it can just automatically do the nested mapping without me having to tell it to do so.
Barring that could I potentially update the index before I insert an object with new fields? And would it be ok if I just set the type of the nested property to nested without specifying the fields of the property?
Code:
client.IndicesPutMapping("captures", "capture", new
{
capture = new
{
properties = new
{
CustomerInformations = new
{
type = "nested",
//...do not specify inner fields ?
}
}
}
});
Is partially mappings allowed when overriding mappings. In other words if I have the mapping above will the other properties of the capture objects still be mapped in a default way?
For those still struggling with the issue:
https://github.com/elastic/elasticsearch/issues/20886
The problem has been resolved in V5