Can someone please help me, how to execute bulk insert with header "Content-Type: application/x-ndjson" in elastic4s ? I have tried this
client.execute {
bulk(
indexInto("cars" / "car").source(getCarsFromJson)
).refresh(RefreshPolicy.WaitFor)
}.await
It works for one element in json, but when i add another element to json, no element are added to elastic.
Are you sure you are using the right syntax? Shouldn't it say
"cars/car"
Instead of
"cars" / "car"
The source method on indexInto will not support multiple json objects, because you're trying to put multiple documents inside a single document insert.
Instead, you will need to take your json, parse it into objects, and iterate over them adding an insert document for each one.
Something like the following:
def getCarsFromJson: Seq[String] = /// must return a sequence of json strings
val inserts = getCarsFromJson.map { car => indexInto("cars" /"car").source(car) }
client.execute {
bulk(inserts:_*).refresh(RefreshPolicy.WaitFor)
}
Related
I am looking to get all of the fields in a record in filebeat using the Script processor and perform an action on them. Using the event.Get() from the script processor, it says, "Get a value from the event (either a scalar or an object). If the key does not exist null is returned. If no key is provided then an object containing all fields is returned."
https://www.elastic.co/guide/en/beats/filebeat/current/processor-script.html
Therefore, my question is, what would I do to ensure that no key is provided to get an object that contains all of the fields are returned?
The event.Get() field will provide the top level fields. To look through these top level fields, use a for loop like:
- script:
lang: javascript
id: get_fields
source: >
function process(event) {
var a = event.Get();
for (var key in a) {
if(event.Get(key) == ""){
event.Delete(key);
}
}
}
I am unsure how to do this for nested fields in this way nor have I tried to extend it to nested fields, but this is how it works for now.
Can anyone help with a MongoTemplate question?
I have got a record structure which has nested arrays and I want to update a specific entry in a 2nd level array. I can find the appropriate entry easier enough by the Set path needs the indexes of both array entries & the '$' only refers to the leaf item. For example if I had an array of teams which contained an array of player I need to generate an update path like :
val query = Query(Criteria.where( "teams.players.playerId").`is`(playerId))
val update = Update()
with(update) {
set("teams.$.players.$.name", player.name)
This fails as the '$' can only be used once to refer to the index in the players array, I need a way to generate the equivalent '$' for the index in the teams array.
I am thinking that I need to use a separate Aggregate query using the something like this but I can't get it to work.
project().and(ArrayOperators.arrayOf( "markets").indexOf("")).`as`("index")
Any ideas for this Mongo newbie?
For others who is facing similar issue, One option is to use arrayFilters in UpdateOptions. But looks like mongotemplate in spring does not yet support the use of UpdateOptions directly. Hence, what can be done is:
Sample for document which contain object with arrays of arrayObj (which contain another arrays of arrayObj).
Bson filter = eq("arrayObj.arrayObj.id", "12345");
UpdateResult result = mongoTemplate.getDb().getCollection(collectionName)
.updateOne(filter,
new Document("$set", new Document("arrayObj.$[].arrayObj.$[x].someField"), "someValueToUpdate"),
new UpdateOptions().arrayFilters(
Arrays.asList(Filters.eq("x.id, "12345))
));
I have a table where each row has a JSON structure as follows that I'm trying to index in a postgresql database and was wondering what the best way to do it is:
{
"name" : "Mr. Jones",
"wish_list": [
{"present_name": "Counting Crows",
"present_link": "www.amazon.com"},
{ "present_name": "Justin Bieber",
"present_link": "www.amazon.com"},
]
}
I'd like to put an index on each present_name within the wish_list array. The goal here is that I'd like to be able to find each row where the person wants a particular gift through an index.
I've been reading on how to create an index on a JSON which makes sense. The problem I'm having is creating an index on each element of an array within a JSON object.
The best guess I have is using something like the json_array_elements function and creating an index on each item returned through that.
Thanks for a push in the right direction!
Please check JSONB Indexing section in Postgres documentation.
For your case index config may be the following:
CREATE INDEX idx_gin_wishlist ON your_table USING gin ((jsonb_column -> 'wish_list'));
It will store copies of every key and value inside wish_list, but you should be careful with a query which hits the index. You should use #> operator:
SELECT jsonb_column->'wish_list'
FROM your_table WHERE jsonb_column->'wish_list' #> '[{"present_link": "www.amazon.com", "present_name": "Counting Crows"}]';
Strongly suggested to check existing nswers:
How to query for array elements inside JSON type
Index for finding an element in a JSON array
Using the following query:
r.db('somedb').table('sometable')('users')
I get the following data from the result:
[
[
{
"fn": "dpw",
"u": "usertwo"
},
{
"fn": "dwd",
"u": "userone"
}
]
]
I would like to take the field "u" and specify lets say "usertwo" and get the value of "fn" for that "u". I want to have the result filtered using ReQL so that I am not just parsing the json result in nodejs as the result will be enormous eventually. What would be the best and most efficient approach. I am new to RethinkDB and would appreciate if you could explain the answer as best you can.
I'm not sure of what you exactly want, but from my understanding, this is what you are looking for:
r.db('somedb').table('sometable')('users').filter(function(user) {
return user("u").eq("usertwo")
})("fn")
You seem to have an array of array of users. if that was not a typo, the query should probably be
r.db('somedb').table('sometable')('users').nth(0).filter(function(user) {
return user("u").eq("usertwo")
})("fn")
I have some documents in couchdb that have fields that are arrays of id's for different associated documents:
{
associatedAssets: ["4c67f6241f4a0efb7dc2abc24a004dfe", "270fd4508a1222a1e2a27cbe7f002d9z"]
}
I would like to write a view that will let me pass in a key that is itself an array of ids, and then return documents whose associatedAssets fields contain one or more of the ids passed in via the key array e.g.
$.ajax({
url: "/db/_design/design_doc/_view/summaryByAssociatedAssets",
type: "post",
data: JSON.stringify({keys: ["4c67f6241f4a0efb7dc2abc24a004dfe", "6c67f6241f4a0efb7dc2abc24a004dfd"]}),
dataType: "json",
contentType: "application/json",
})
.done(function(resp){
console.log(resp[0]);
});
would return documents whose associatedAssets array contains one or more of the keys "4c67f6241f4a0efb7dc2abc24a004dfe", "6c67f6241f4a0efb7dc2abc24a004dfd".
I can't access the keys in my view, so I'm not sure if I can do this? Is there a better way to accomplish this?
Thanks!
Your view just needs to generate an output row per associatedAssets element, something like this:
function(doc) {
if( doc.associatedAssets ) {
for( var i=0, l=doc.associatedAssets.length; i<l; i++) {
emit( doc.associatedAssets[i], doc );
}
}
}
Then you'd need to adjust your call so it ends up passing that keys array as a query string parameter, which will return only the rows from the view that match keys in that array.
A total aside - assuming a recent CouchDB version, the best practices would be to replace doc in your emit with { _id: doc._id } and then use include_docs=true in your query so your view index doesn't get filled up (unnecessarily) with full documents.
In the view code you can access anything in the document itself, but there is no way to access any parameters that you pass in.
You could use temporary views that are generated for the specific query you are doing.
You could also use ElasticSearch instead. ElasticSearch has a much richer query DSL because that's the whole point of ElasticSearch. The CouchDB River allows you to automatically index all documents from CouchDB.
Using ElasticSearch, you could just search for any document whose associatedAssets contains any element from a list that you pass in.