I have an index, whose mapping has a company_tag keyword field, and a data.company_name keyword field. The company_tag is not a sub-field. The data is an object field. However, when I try the following
from elasticsearch_dsl import Search,A
s = Search()
a = A('terms', field='data__company_name')
s.aggs.bucket("company_name",a)
response = s.execute()
response.success()
The success is True, but response.aggs.to_dict() returns:
{'company_name': {'doc_count_error_upper_bound': 0,
'sum_other_doc_count': 0,
'buckets': []}}
If I do
s = Search()
a = A('terms', field='company_tag')
s.aggs.bucket("company_name",a)
response = s.execute()
response.success()
I do get a success and non-empty buckets.
Why is that?
Related
I want to check whether the field exists in the index of OpenSearch or not. I expect the response should be in Boolean 'True' Or 'False'.
In Elasticsearch to check index exists or not, we use the following code
if es.indices.exists(index="index"):
Is there any function/logic to check for field names present in the AWS OpenSearch index? If the field does not exist, then I have to return the response as "Field not found in the index"
My Sample Code:
from opensearchpy import OpenSearch, RequestsHttpConnection, AWSV4SignerAuth
import boto3
host = 'my-test-domain.us-east-1.es.amazonaws.com'
region = 'us-east-1'
credentials = boto3.Session().get_credentials()
auth = AWSV4SignerAuth(credentials, region)
index_name = 'movies'
client = OpenSearch(
hosts = [{'host': host, 'port': 443}],
http_auth = auth,
use_ssl = True,
verify_certs = True,
connection_class = RequestsHttpConnection
)
q = 'miller'
query = {
'size': 5,
'query': {
'multi_match': {
'query': q,
'fields': ['title^2', 'director']
}
}
}
response = client.search(
body = query,
index = index_name
)
print('\nSearch results:')
print(response)
In my application I have a response like this:
{
"items": [
{
"id": 10,
"field": "foo"
},
{
"id": 20,
"field": "bar"
}
]
}
I need to test the content of items and validate each id.
I've tried many solutions but no one works, for example (this is just a kind of pseudo-code):
assertJson(fn (AssertableJson $json) =>
$json->where('items.*.id', [10, 20])
)
Is there a way to use a wildcard to pick every ID and validate using an array?
You can use array_filter:
$idArray = [10, 20];
$myObj = json_decode($json); // Turn JSON to obj
$items = $myObj["items"]; // Get items from object
// Filter the items for items that aren't in the ID list
$invalidItems = array_filter($items, function ($el) {
// If the item has an id which isn't in the array, return true
return !in_array($el["id"], $idArray);
});
// This returns true if we found 0 items with IDs not in the ID list
return $invalidItems == [];
You can similarly use array_map to simplify your array, then compare it to your ID array:
$myObj = json_decode($json); // Turn JSON to obj
$items = $myObj["items"]; // Get items from object
$outIdArray = array_map(function($el) {
return $el["id"];
}, $items);
// Compare $outIdArray to [10, 20]
Not tested yet but below should work.
We attach an each on each child element under items and add a callback to where on that id key of each child.
<?php
assertJson(fn (AssertableJson $json) =>
$json->each('items', fn (AssertableJson $childJson) =>
$childJson->where('id', fn($idVal) =>
in_array($idVal, [10,20])
)
)
)
I would like to perform a query in a table where the column is of json type and it contains array of objects, but only if a certain condition is met
This is my current code
$initial_results = DB::table('toys')->select('id','name')->where(['name' => 'sammy', 'email' => 'whateveremail']);
if($sk ==='yes') {
$results = $initial_results->>whereRaw('JSON_CONTAINS(`info`,\'{"sku":"B07V3SSLN11"}\')')
>whereRaw('JSON_CONTAINS(`info`,\'{"asin":"DTI-LALF3-EA18"}\')')
->get();
} else {
$results = $initial_results->get();
}
But I always get 0 result if the condition is met. In database, the info I want to query indeed exist. What is the proper way to query a json column which contains array of objects? See my example data
[
{
"sku": "DTI-LALF3-EA18",
"adId": 244077676726655,
"asin": "B07V3SSLN11",
"cost": 0,
},
{
"sku": "DTI-LALF3-EA18",
"adId": 242968940906362,
"asin": "B07V3SSLN11",
"cost": 10,
.........
................
I even tried
$initial_results = DB::table('toys')->select('id','name')->where(['name' => 'sammy', 'email' => 'whateveremail'])->->whereIn(DB::raw("JSON_EXTRACT(info, '$[*].asin')"),['B07V3SSLN11']);
Thanks in advance
You can query JSON columns by using the -> operator in your clause:
->where('info->asin', 'DTI-LALF3-EA18')
JSON Where Clauses Docs
I´ve been trying to get multiple averages grouping by an identifier using the Spring boot Aggregation. The fields whose average I want are defined by an Array given by parameter, but I can´t retrieve more than one average.
The data is stored like :
{
"_id": {
"$oid": "XX"
},
"space_id": "AA",
"temperature": 50,
"temperatureUoMCode": 0,
"pm25": 50,
"pm25UoMCode": 0,
"co2": 0.0005,
"co2UoMCode": 0,
"co": 1,
"coUoMCode": 0,
"gas": 50,
"gasUoMCode": 0,
"humidity": 50,
"humidityUoMCode": 0,
"occupants": 6,
"maxCapacity": 15,
"_class": "Record"
}
The code that I am using:
List<AggregationOperation> aggregationOperations = new ArrayList<>();
aggregationOperations.add(Aggregation.match(
Criteria.where("space_id").is(space_id)
));
for(Variable var : concerns){
System.out.println(var.getField());
aggregationOperations.add(Aggregation.group(
space_id).
avg(var.getField()).as(var.getAvg()));
}
Aggregation aggregation = Aggregation.newAggregation(
aggregationOperations
);
AggregationResults<HashMap> results = mongoTemplate.aggregate(
aggregation, "records",HashMap.class);
Also, Variable class is an enum with two fields, the name of the field in the db (field) and the average alias (avg).
When I run it I get the following output:
co
pm25
temperature
humidity
gas
2021-09-22 11:34:17.046 ERROR 42971 --- [a1-0b2af6e69097] TConfig$$EnhancerBySpringCGLIB$$673c396b : Invalid reference 'pm25'!
java.lang.IllegalArgumentException: Invalid reference 'pm25'!
at org.springframework.data.mongodb.core.aggregation.ExposedFieldsAggregationOperationContext.getReference(ExposedFieldsAggregationOperationContext.java:114)
at org.springframework.data.mongodb.core.aggregation.ExposedFieldsAggregationOperationContext.getReference(ExposedFieldsAggregationOperationContext.java:86)
at org.springframework.data.mongodb.core.aggregation.GroupOperation$Operation.getValue(GroupOperation.java:529)
at org.springframework.data.mongodb.core.aggregation.GroupOperation$Operation.toDocument(GroupOperation.java:507)
at org.springframework.data.mongodb.core.aggregation.GroupOperation.toDocument(GroupOperation.java:441)
at org.springframework.data.mongodb.core.aggregation.AggregationOperation.toPipelineStages(AggregationOperation.java:55)
at org.springframework.data.mongodb.core.aggregation.AggregationOperationRenderer.toDocument(AggregationOperationRenderer.java:56)
at org.springframework.data.mongodb.core.aggregation.AggregationPipeline.toDocuments(AggregationPipeline.java:81)
at org.springframework.data.mongodb.core.aggregation.Aggregation.toPipeline(Aggregation.java:716)
at org.springframework.data.mongodb.core.AggregationUtil.createPipeline(AggregationUtil.java:112)
at org.springframework.data.mongodb.core.MongoTemplate.doAggregate(MongoTemplate.java:2144)
at org.springframework.data.mongodb.core.MongoTemplate.doAggregate(MongoTemplate.java:2119)
at org.springframework.data.mongodb.core.MongoTemplate.aggregate(MongoTemplate.java:2113)
at org.springframework.data.mongodb.core.MongoTemplate.aggregate(MongoTemplate.java:2014)
at co.edu.javeriana.smartuj.EnvironmentalAlerts.services.AlertService.makeQuery(AlertService.java:131)
If someone is having a similar problem I managed to solve it by following this post's answer. I realized that instead of creating various GroupOperations, I could just create a single one and use it to store all the operations by saving the return of the "as" function in the same object:
MatchOperation matchOperation = Aggregation.match(Criteria.where("space_id").is(space_id));
GroupOperation groupOperation = null;
for(Variable var : concerns){
if (groupOperation == null){
groupOperation = Aggregation.group("space_id")
.avg(var.getField())
.as(var.getAvg());
}else
groupOperation = groupOperation
.avg(var.getField())
.as(var.getAvg());
}
Aggregation aggregation = Aggregation.newAggregation(
matchOperation,
groupOperation
);
AggregationResults<HashMap> results = mongoTemplate.aggregate(
aggregation, "records",HashMap.class);
return results.getUniqueMappedResult();
I am trying to write a query to sort out documents based on descending dates ...{sort: {paymentDate: -1 }} order. The first time the query runs, the query section {sort: {paymentDate: -1 }} seems get ignored!
However when I refresh the page in the browser, the query section {sort: {paymentDate: -1 }} is applied, and the query displays in the correct sort order.
I need to know how to correct this issue!
Find below the contents of my document after I run the recipientsDetails.find().fetch(); query in the browser console:
0:
payersUserId: "hbieZBFNE53GpE8LP"
paymentDate: "2019-02-11 02:37:05"
payersNumber: "+25478887633"
paymentStatus: "Failed"
recipientNumber: "+25478887633"
_id: "eFShDRzp9JM9ejG5S"
1:
payersUserId: "hbieZBFNE53GpE8LP"
paymentDate: "2019-02-08 16:02:25"
payersNumber: "+2547078887633"
paymentStatus: "Failed"
recipientNumber: "+25478887633"
_id: "SnpNwsx49mZfPNSg7"
2:
payersUserId: "hbieZBFNE53GpE8LP"
paymentDate: "2019-02-08 15:00:02"
payersNumber: "+254707888633"
paymentStatus: "Failed"
recipientNumber: "+25478087703"
_id: "ZHWSiRBYk2xoZvDzb"
The above results is also the desired sorted order.
Perhaps the below helper code might shade some light.
../client/main.js
Template.paymentB2C.helpers({
'enableButton': function () {
var enableButtonStatusArray = [];
var userIdCode = Meteor.userId();
var phoneNumber = Meteor.users.findOne({_id: userIdCode }, { fields: { "profile.telephoneNumber": 1 } } );
var usersPhoneNumber = phoneNumber.profile.telephoneNumber;
var selectedRecipientDetails = recipientsDetails.find( { $or: [ { payersNumber: usersPhoneNumber }, { recipientNumber: usersPhoneNumber } ] },
{ fields: {
"payersUserId": 1,
"paymentDate": 1,
"paymentStatus": 1,
"_id": 1
} }).fetch();
selectedRecipientDetails.forEach((user) => {
payersUserId = user.payersUserId;
paymentDate = user.paymentDate;
paymentStatus = user.paymentStatus;
_id = user._id;
if(paymentStatus === "Failed"){
enableButtonStatusArray.push({
paymentStatus: paymentStatus,
paymentDate: paymentDate,
_id: _id
});
}
else if(paymentStatus === "Passed"){
enableButtonStatusArray.push({
paymentStatus: paymentStatus,
paymentDate: paymentDate,
_id: _id});
}
Session.set('enableButtonStatusArray2', enableButtonStatusArray );
});
var enableButtonStatusArrayForPrint = Session.get('enableButtonStatusArray2');
return enableButtonStatusArrayForPrint;
}
});
Note that the query here lacks a ...{sort: {paymentDate: -1 }} function.
Find below my Router code:
../client/main.js
Router.route('/paymentB2C', {
name: 'paymentB2C',
template: 'paymentB2C',
waitOn: function(){
return Meteor.subscribe('pendingPayments')
}
});
This leads to my Meteor.subscribe('pendingPayments') publish function:
../server/main.js
Meteor.publish('pendingPayments', function pendingPayments(){
return recipientsDetails.find({}, {sort: {paymentDate: -1 }});
});
Note that here is where I have the sort function.
Can someone explain why when codes first runs, the sort is ignored and the the document is randomly sorted, however is sorted out as designed (correctly) after hitting refresh in the browser?
Looking forward to your help.
Ideally, you should sort the data on the client-side query after you subscribe, instead of sorting it in the publish method.
The reason is that if the client subscribes to more than one publish function which will publish data from the same collections, your find query in the client-side will have access to the data from both publish as well as sort won't be effective. Moreover, publish is something which will grant data access to the subscriber and if the mini-mongo on the client side already has the data, it won't sync the data unless new data arrives.
Hence, you should always do sort and filter in your find queries on the client side as well.
Also, I notice that the format of the paymentDate field is not a 'Date'. It should ideally be of the Date format and should look something like ISODate("2019-02-11T02:37:05.000Z") instead of String format "2019-02-11 02:37:05". So if the sorting on the client side is also not working, try saving the paymentDate in the database as Date instead as a String.