Could any one, please tell me how to do -
Indexing and Searching an "array" type within an embedded document in MongoDB using Java.
For example: The outer document id UserDetails and the array is given below
"languages_known" :
[
"English",
"Kannada",
"Hindi",
"German"
]
I referred this : http://docs.mongodb.org/manual/core/index-multikey/#index-type-multikey.
But still I could not do.
Please tell to do Indexing and Searching for the above in Java.
You build an index on array inside a document using below in mongo shell:-
db.collection_name.ensureIndex({languages_known: 1}) // In your case
In java driver you can use:-
collection.createIndex(DBObject keys);
Can you please clarify what did you try and errors that you may have encountered?
Related
We want to add version info in a metadata like information to all of our Documents in Couchbase.
Why?
Since our Entity objects are constantly evolving, we want to achieve the following without breaking our data services:
GET all Books of version v1
GET all Books of version v2
Update all Books of v1 to v2 (via a script)
This way we can support multiple document versions for our various consumers.
What is the most ideal way to achieve this? How can I store Documents in a way that I can query Couchbase based on versions?
I'm not a Java dev, so I don't understand all the limitations you're working with in Spring, but have you considered adding a 'version' property to documents?
book::0001
{
"title" : "Moby Dick",
"version" : 1
}
and then
book::0002
{
"title" : "Gone Whalin'",
"isbn10" : "1492966851",
"version" : 2
}
For the purpose of versions in couchbase document is Optimistic Locking.
Plesae have a look here. And one more important thing is we can't modify this version it's handle by couchbase it self.
You can easily handle with spring data couchbase. If you need example for more detail let me know.
I'm trying to write logs to an Elasticsearch index from a Kubernetes cluster. Fluent-bit is being used to read stdout and it enriches the logs with metadata including pod labels. A simplified example log object is
{
"log": "This is a log message.",
"kubernetes": {
"labels": {
"app": "application-1"
}
}
}
The problem is that a few other applications deployed to the cluster have labels of the following format:
{
"log": "This is another log message.",
"kubernetes": {
"labels": {
"app.kubernetes.io/name": "application-2"
}
}
}
These applications are installed via Helm charts and the newer ones are following the label and selector conventions as laid out here. The naming convention for labels and selectors was updated in Dec 2018, seen here, and not all charts have been updated to reflect this.
The end result of this is that depending on which type of label format makes it into an Elastic index first, trying to send the other type in will throw a mapping exception. If I create a new empty index and send in the namespaced label first, attempting to log the simple app label will throw this exception:
object mapping for [kubernetes.labels.app] tried to parse field [kubernetes.labels.app] as object, but found a concrete value
The opposite situation, posting the namespaced label second, results in this exception:
Could not dynamically add mapping for field [kubernetes.labels.app.kubernetes.io/name]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text].
What I suspect is happening is that Elasticsearch sees the periods in the field name as JSON dot notation and is trying to flesh it out as an object. I was able to find this PR from 2015 which explicitly disallows periods in field names however it seems to have been reversed in 2016 with this PR. There is also this multi-year thread from 2015-2017 discussing this issue but I was unable to find anything recent involving the latest versions.
My current thoughts on moving forward is to standardize the Helm charts we are using to have all of the labels use the same convention. This seems like a band-aid on the underlying issue though which is that I feel like I'm missing something obvious in the configuration of Elasticsearch and dynamic field mappings.
Any help here would be appreciated.
I opted to use the Logstash mutate filter with the rename option as described here:
https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-rename
The end result looked something like this:
filter {
mutate {
'[kubernetes][labels][app]' => '[kubernetes][labels][app.kubernetes.io/name]'
'[kubernetes][labels][chart]' => '[kubernetes][labels][helm.sh/chart]'
}
}
Although personally I've never encountered the exact same issue, I had similar problems when I indexed some test data and afterwards changed the structure of the document that should have been indexed (especially when "unflattening" data structures).
Your interpretation of the error message is correct. When you first index the document
{
"log": "This is another log message.",
"kubernetes": {
"labels": {
"app.kubernetes.io/name": "application-2"
}
}
}
Elasticsearch will recognize the app as an object/structure due to dynamic mapping.
When you then try to index the document
{
"log": "This is a log message.",
"kubernetes": {
"labels": {
"app": "application-1"
}
}
}
the previously, dynamically created mapping defined the field app as an object with sub-fields but elasticsearch encounters a concrete value, namely "application-1".
I suggest that you setup an index template to define the correct mappings. For the 'outdated' logging-versions I suggest to pre-process the particular documents either through an elasticsearch ingest-pipeline or with e.g. Logstash to get the documents in the correct format.
Hope that helps.
My question is more in general assuming i have a simple query like this to elastic search
Page<MyEntity> findAll(Pageable pageable);
I want to be able to set a timeout for this query for instance so it doesn't hang forever, although I read the documentation I didn't see anything clear about how to do it.
Is the any way to do it? a way to set a timeout for Spring-data-elasticsearch queries that I can make sure that nothing will get for too long?
One way of achieving a 'timeout' in a search request query is using the 'timeout' parameter in the query itself. here
Lets assume we want to perform a full-text 'match query', we can add 'timeout' before the query itself:
{
"timeout": "1ms",
"query": {
"match" : {
"description" : "This is a fullText test"
}
}
}
You will have to use Elasticsearch time units as mentioned here and ship them as String values.
In your case - I don't see any way to achieve this using spring-data-es repository, but - you can add custom functionality to your repository and use the ElasticsearchIndexTemplate with the matchAllQuery() (java elastic api),
Something like that (haven't tested it):
nodeEsTemplate.getClient().prepareSearch("test-index")
.setQuery(QueryBuilders.matchAllQuery())
.setTimeout(TimeValue.timeValueMillis(1))
.execute().actionGet();
As nodeEsTemplate is of type ElasticsearchIndexTemplate and assuming you have created a custom findAllWithTimeOut method in your repository class.
I recently started working in a company that uses Elasticsearch. While most of its concepts are somewhat similar to relational databases and I am able to understand them, I still don't quite get the concept of aliases.
I did not find any such question here and the information provided on the Elasticsearch website did not help much either.
Can someone explain what aliases are for and ideally include an example of a situation where they are needed?
aliases are like soft links or shortcuts to actual indexes
the advantage is to be able to have an alias pointing to index1a while building or re-indexing on index2b and the moment of swapping them is atomic thanks to the alias, to which all code should point
Renaming an alias is a simple remove then add operation within the same API. This operation is atomic, no need to worry about a short period of time where the alias does not point to an index:
[EDIT] as pointed out #wholevinski aliases have other functionalities like:
Multiple indices can be specified for an action ...
all the info is in the page you have linked
[EDIT2] more on why the need/benefit of the atomicity
the key being "zero downtime" https://en.wikipedia.org/wiki/Zero_unscheduled_downtime or https://en.wikipedia.org/wiki/High_availability
https://www.elastic.co/guide/en/elasticsearch/guide/current/index-aliases.html
We will talk more about the other uses for aliases later in the book. For now we will explain how to use them to switch from an old index to a new index with zero downtime.
#arhak covered the topic pretty well.
One use case that (at least) made me understand the value of indices was the need to remove out-of-date documents and more specifically when using time-based-indices.
For example, you need to keep the logs of an application for at least one year. You decide to use time-based-indices, meaning you save into indices with the following format: 2018-02-logs, 2018-03-logs etc.. In order to be able to search in every index you create the following alias:
POST /_aliases
{
"actions": [{
"add": {
"alias": "current-logs", "indices": [ "2018-02-logs","2018-03-logs" ]
}
}]
}
And query like:
GET /current-logs/_search
Another advantage is that you can delete the out-of-date values very easily:
POST /_aliases
{
"actions": [
{ "remove": { "alias": "current-logs", "index": "logs_2018-01" }}
]
}
and DELETE /logs_2018-01
Aliases are basically created to group a set of indices and make them accessible regarless the name they have. Is a pointer to a set of indices. You can also apply a query/condition to all of these indices. It is very useful when performing queries or creating dashboards over the same group of indices all the time. In addition, if in the future you change the name of the indices that are part of an alias, the end users will not notice that change since it is for transparent for them and you will only update the pointer.
Using NEST, I need to be able to order a terms aggregation with multiple criteria (requires ElasticSearch 1.5 or later). For example:
"order": [{"avg_rank": "desc"}, {"avg_score": "desc"}]
This is working great using the raw JSON that I created to verify I was getting the expected behavior. Now, in trying to translate that over to code using the NEST library, I'm not seeing how that would be accomplished.
The OrderDescending() method has only one implementation that takes a string for the key. I need a C# "params" type method that can take a list of OrderDescending() and\or OrderAscending() elements.
Is there a way to do this in NEST that I'm overlooking?
Is there a way in NEST to work around this where I can inject a little raw JSON where I need it?
FWIW, I've been using the "fluent" style to create my queries.
EDIT:
I see that, using "object initializer" syntax, I could manually create the dictionary and add my criteria elements. Problem is, I have large amounts of code written in "fluent" syntax. So,
Is there a way to use an "object initializer" object and convert it to a "fluent" descriptor? In this case, a TermsAggregator to a TermsAggregationDescriptor?
EDIT 2:
I should have mentioned originally that I tried .OrderDescending("avg_rank").OrderDescending("avg_score") already. That simply took that last one in the chain. In looking at the code, I can see why. Each call to OrderDescending blindly news up the dictionary instead of checking to see if one was already newed up and adding a new key to the dictionary if it already exists.
Based on this, I believe this is a bug for which I have entered a report here:
OrderDescending and OrderAscending cannot be chained for multi-criteria ordering
EDIT 3:
I appreciate all the answers (some of which are getting deleted) because they're helping drive this along and are responsible for these edits. I should also have mentioned originally that I discovered that:
"order": { "avg_rank": "desc", "avg_score": "desc" }
does not work. I don't know why for sure but ES will only use the last one in that case. It has be a list of dictionaries as shown in my example above at the top. I've verified that correctly sub-orders the aggregation on the second element. So, the underlying object cannot be typed as a simple dictionary. I've also added this information to the bug report I created (as noted in EDIT 2).
If you're using the fluent syntax you can just chain the sorts together.
Sample:
var esClient = ninjectKernel.Get<IElasticClient>();
var query = esClient.Search<RedemptionES>(s=> s
.SortAscending(a=>a.Date)
.SortDescending(d=>d.Input.User.Name)
);
Response:
{
"sort": [
{
"#timestamp": {
"order": "asc"
}
},
{
"input.user.name": {
"order": "desc"
}
}
]
}
Martijn Laarman of the NEST team was very responsive and kind enough to provide a work around for the bug I reported in EDIT 2 of the description above. The fix can be found in the comments of that same bug report: Work around for NEST library multi-criteria aggregation ordering.
Note that he provided a work around for both object initializer and fluent syntaxes (the one I needed).