I have a JArray with this configuration
[
{
"name" : "text",
"age" : 32
},
{
"name" : "text2",
"age" : 33
},
]
and I want to use LINQ query to select a JArray containing JObjects with just the key value of a specified given key.
For example:
GetCollectionOfPropertiesByKey("name");
this would result in:
[
{
"name" : "text"
},
{
"name" : "text2"
},
]
using the same logic,
GetCollectionOfPropertiesByKey("age");
would result in:
[
{
"age" : 32
},
{
"age" : 33
},
]
You didn't specify whether you needed case-invariant filtering for properties by name, so here is an extension method with an optional StringComparison argument:
public static class JsonExtensions
{
public static JArray GetCollectionOfPropertiesByKey(this JArray array, string key, StringComparison comparison = StringComparison.Ordinal)
{
if (array == null)
return null;
var query = array
// Filter out array items that are not objects
.OfType<JObject>()
// Select the value JToken with the specified key using the specified StringComparison
.Select(o => o.GetValue(key, comparison))
// Filter for null (property was not found)
.Where(v => v != null)
// And select a new JObject containing just the JPropery parent of the selected value
.Select(v => new JObject(v.Parent));
return new JArray(query);
}
}
If a case-insensitive match is required, pass StringComparison.OrdinalIgnoreCase.
Notes:
The query filters for all items in the input array that are of type JObject, then uses JObject.GetValue(String, StringComparison) to get the corresponding value by key name.
Then, if a value is found, it constructs a new JObject using the parent of the selected value, which should be a JProperty.
If case insensitivity is never required, you could simplify the query a bit by using JObject.Property(String) to get the JProperty by key name, and then constructing a JObject containing it, like so:
var query = array
// Filter out array items that are not objects
.OfType<JObject>()
// Select the JProperty with the specified key
.Select(o => o.Property(key))
// Filter for null (property was not found)
.Where(p => p != null)
// And select a new JObject with just this property
.Select(p => new JObject(p));
Sample working .Net fiddle.
Related
I am trying to build a query with Criteria Builder(LIKE), to look for a string in JSONARRAY field like this:
[
{
"family_class": "Electric",
"family_name": "lightBulb"
},
{
"family_class": "Others",
"family_name": "Oil"
}
]
one option would be to look for the family_name attribute, or maybe check if it contains the string there.
if (residues != null && residues.length > 0) {
List<Predicate> predicates = new ArrayList<Predicate>();
for (String residue : residues) {
predicates.add(cb.like(root.get("jsonColumn"), residue.toLowerCase()));
}
cr.select(root).where(predicates.toArray(new Predicate[] {}));
Query<SyncCollectionPoints> q = sess.createQuery(cr);
List<SyncCollectionPoints> result= q.getResultList();
This is the error i get:
Unrecognized token 'oil': was expecting ('true', 'false' or 'null')
All i want is to return the lines that have that string in the jsonColumn field.
I got it to work like this:
#Formula(value = "lower(jsonColumn::text)")
private String residuesToSearch;
just a simple cast did the trick
I have three main questions I need help answering.
How do you correctly map and store a nested map?
How do you search a nested part of a document?
How do you bulk index?
I'm using Nest version 2 and have been looking over the new documentation which can be found Here. The documentation has been useful in creating certain parts of the code but unfortunately doesn't explain how they fit together.
Here is the class I'm trying to map.
[ElasticsearchType(Name = "elasticsearchproduct", IdProperty = "ID")]
public class esProduct
{
public int ID { get; set; }
[Nested]
public List<PriceList> PriceList { get; set; }
}
[ElasticsearchType(Name = "PriceList")]
public class PriceList
{
public int ID { get; set; }
public decimal Price { get; set; }
}
and my mapping code
var node = new Uri(HOST);
var settings = new ConnectionSettings(node).DefaultIndex("my-application");
var client = new ElasticClient(settings);
var map = new CreateIndexDescriptor("my-application")
.Mappings(ms => ms
.Map<esProduct>(m => m
.AutoMap()
.Properties(ps => ps
.Nested<PriceList>(n => n
.Name(c => c.PriceList)
.AutoMap()
)
)
)
);
var response = client.Index(map);
This is the response I get:
Valid NEST response built from a succesful low level call on POST: /my-application/createindexdescriptor
So that seems to work. next index.
foreach (DataRow dr in ProductTest.Tables[0].Rows)
{
int id = Convert.ToInt32(dr["ID"].ToString());
List<PriceList> PriceList = new List<PriceList>();
DataRow[] resultPrice = ProductPriceTest.Tables[0].Select("ID = " + id);
foreach (DataRow drPrice in resultPrice)
{
PriceList.Add(new PriceList
{
ID = Convert.ToInt32(drPrice["ID"].ToString()),
Price = Convert.ToDecimal(drPrice["Price"].ToString())
}
esProduct product = new esProduct
{
ProductDetailID = id,
PriceList = PriceList
};
var updateResponse = client.Update<esProduct>(DocumentPath<esProduct>.Id(id), descriptor => descriptor
.Doc(product)
.RetryOnConflict(3)
.Refresh()
);
var index = client.Index(product);
}
}
Again this seems to work but when I come to search it does seem to work as expected.
var searchResults = client.Search<esProduct>(s => s
.From(0)
.Size(10)
.Query(q => q
.Nested(n => n
.Path(p => p.PriceList)
.Query(qq => qq
.Term(t => t.PriceList.First().Price, 100)
)
)
));
It does return results but I was expecting
.Term(t => t.PriceList.First().Price, 100)
to look move like
.Term(t => t.Price, 100)
and know that is was searching the nested PriceList class, is this not the case?
In the new version 2 documentation I can't find the bulk index section. I tried using this code
var descriptor = new BulkDescriptor();
***Inside foreach loop***
descriptor.Index<esProduct>(op => op
.Document(product)
.Id(id)
);
***Outside foreach loop***
var result = client.Bulk(descriptor);
which does return a success response but when I search I get no results.
Any help would be appreciated.
UPDATE
After a bit more investigation on #Russ advise I think the error must be with my bulk indexing of a class with a nested object.
When I use
var index = client.Index(product);
to index each product I can use
var searchResults = client.Search<esProduct>(s => s
.From(0)
.Size(10)
.Query(q => q
.Nested(n => n
.Path(p => p.PriceList)
.Query(qq => qq
.Term(t => t.PriceList.First().Price, 100)
)
)
)
);
to search and return results, but when I bulk index this no long works but
var searchResults = client.Search<esProduct>(s => s
.From(0)
.Size(10)
.Query(q => q
.Term(t => t.PriceList.First().Price, 100)
)
);
will work, code b doesn't work on the individual index method. Does anyone know why this has happened?
UPDATE 2
From #Russ suggested I have taken a look at the mapping.
the code I'm using to index is
var map = new CreateIndexDescriptor(defaultIndex)
.Mappings(ms => ms
.Map<esProduct>(m => m
.AutoMap()
.Properties(ps => ps
.Nested<PriceList>(n => n
.Name(c => c.PriceList)
.AutoMap()
)
)
)
);
var response = client.Index(map);
Which is posting
http://HOST/fresh-application2/createindexdescriptor {"mappings":{"elasticsearchproduct":{"properties":{"ID":{"type":"integer"},"priceList":{"type":"nested","properties":{"ID":{"type":"integer"},"Price":{"type":"double"}}}}}}}
and on the call to http://HOST/fresh-application2/_all/_mapping?pretty I'm getting
{
"fresh-application2" : {
"mappings" : {
"createindexdescriptor" : {
"properties" : {
"mappings" : {
"properties" : {
"elasticsearchproduct" : {
"properties" : {
"properties" : {
"properties" : {
"priceList" : {
"properties" : {
"properties" : {
"properties" : {
"ID" : {
"properties" : {
"type" : {
"type" : "string"
}
}
},
"Price" : {
"properties" : {
"type" : {
"type" : "string"
}
}
}
}
},
"type" : {
"type" : "string"
}
}
},
"ID" : {
"properties" : {
"type" : {
"type" : "string"
}
}
}
}
}
}
}
}
}
}
}
}
}
}
fresh-application2 returned mapping doesn't mention nested type at all, which I'm guessing is the issue.
The mapping my working nested query looks more like this
{
"my-application2" : {
"mappings" : {
"elasticsearchproduct" : {
"properties" : {
"priceList" : {
"type" : "nested",
"properties" : {
"ID" : {
"type" : "integer"
},
"Price" : {
"type" : "double"
}
}
},
"ID" : {
"type" : "integer"
},
}
}
}
}
}
This has the nested type returned. I think the one which isn't returning nested as a type is when I started using .AutoMap() , am I using it correctly?
UPDATE
I have fixed my mapping problem. I have changed my mapping code to
var responseMap = client.Map<esProduct>(ms => ms
.AutoMap()
.Properties(ps => ps
.Nested<PriceList>(n => n
.Name(c => c.PriceList)
.AutoMap()
)
)
);
Whilst you're developing, I would recommend logging out requests and responses to Elasticsearch so you can see what is being sent when using NEST; this'll make it easier to relate to the main Elasticsearch documentation and also ensure that the body of the requests and responses match your expectations (for example, useful for mappings, queries, etc).
The mappings that you have look fine, although you can forgo the attributes since you are using fluent mapping; there's no harm in having them there but they are largely superfluous (the type name for the esProduct is the only part that will apply) in this case because .Properties() will override inferred or attribute based mapping that is applied from calling .AutoMap().
In your indexing part, you update the esProduct and then immediately after that, index the same document again; I'm not sure what the intention is here but the update call looks superfluous to me; the index call will overwrite the document with the given id in the index straight after the update (and will be visible in search results after the refresh interval). The .RetryOnConflict(3) on the update will use optimistic concurrency control to perform the update (which is effectively a get then index operation on the document inside of the cluster, that will try 3 times if the version of the document changes in between the get and index). If you're replacing the whole document with an update i.e. not a partial update then the retry on conflict is not really necessary (and as per previous note, the update call in your example looks unnecssary altogether since the index call is going to overwrite the document with the given id in the index).
The nested query looks correct; You specify the path to the nested type and then the query to a field on the nested type will also include the path.I'll update the NEST nested query usage documentation to better demonstrate.
The bulk call looks fine; you may want to send documents in batches e.g. bulk index 500 documents at a time, if you need to index a lot of documents. How many to send in one bulk call is going to depend on a number of factors including the document size, how it is analyzed, performance of the cluster, so will need to experiment to get a good bulk size call for your circumstances.
I'd check to make sure that you are hitting the right index, that the index contains the count of documents that you expect and find a document that you know has a PriceList.Price of 100 and see what is indexed for it. It might be quicker to do this using Sense while you're getting up an running.
I need to select users that have a "3" in their json array.
{
"People":[
{
"id" : "123",
"firstName" : "Bill",
"lastName" : "Gates",
"roleIds" : {
"int" : ["3", "9", "1"]
}
},
{
"id" : "456",
"firstName" : "Steve",
"lastName" : "Jobs",
"roleIds" : {
"int" : ["3", "1"]
}
},
{
"id" : "789",
"firstName" : "Elon",
"lastName" : "Musk",
"roleIds" : {
"int" : ["3", "7"]
}
},
{
"id" : "012",
"firstName" : "Agatha",
"lastName" : "Christie",
"roleIds" : {
"int" : "2"
}
}
]}
In the end, my results should be Elon Musk & Steve Jobs. This is the code that I used (& other variations):
var roleIds = pplFeed["People"]["roleIds"].Children()["int"].Values<string>();
var resAnAssocInfo = pplFeed["People"]
.Where(p => p["roleIds"].Children()["int"].Values<string>().Contains("3"))
.Select(p => new
{
id = p["id"],
FName = p["firstName"],
LName = p["lastName"]
}).ToList();
I'm getting the following error:
"Accessed JArray values with invalid key value: "roleIds". Int32 array index expected"
I changed .Values<string>() to .Values<int>() and still no luck.
What am I doing wrong?
You are pretty close. Change your Where clause from this:
.Where(p => p["roleIds"].Children()["int"].Values<string>().Contains("3"))
to this:
.Where(p => p["roleIds"]["int"].Children().Contains("3"))
and you will get you the result you want (although there are actually three users in your sample data with a role id of "3", not two).
However, there's another issue that you might hit for which this code still won't work. You'll notice that for Agatha Christie, the value of int is not an array like the others, it is a simple string. If the value will sometimes be an array and sometimes not, then you need a where clause that can handle both. Something like this should work:
.Where(p => p["roleIds"]["int"].Children().Contains(roleId) ||
p["roleIds"]["int"].ToString() == roleId)
...where roleId is a string containing the id you are looking for.
Fiddle: https://dotnetfiddle.net/Zr1b6R
The problem is that not all objects follow the same interface. The last item in that list has a single string value in the roleIds.int property while all others has an array. You need to normalize that property and then do the check. It'll be easiest if they were all arrays.
You should be able to do this:
var roleId = "3";
var query =
from p in pplFeed["People"]
let roleIds = p.SelectToken("roleIds.int")
let normalized = roleIds.Type == JTokenType.Array ? roleIds : new JArray(roleIds)
where normalized.Values().Contains(roleId)
select new
{
id = p["id"],
FName = p["firstName"],
LName = p["lastName"],
};
Following http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-suggesters-completion.html
How can I index/insert (I can do mapping) an object using Nest client library to be able to provide following options:
"input": ...,
"output": ...,
"payload" : ...,
"weight" : ...
I would like to be able to provide multiple values in 'input' option.
Can't find anyway of doing this using NEST.
Thank you
NEST provides the SuggestField type in order to assist in indexing completion suggestions. You don't necessarily need to use this type, you can provide your own that contains the expected completion fields (input, output, etc...), but the purpose of SuggestField is to make the whole process easier by already providing a baked in type.
Usage:
Add a suggest field to the document/type you are indexing:
public class MyType
{
public SuggestField Suggest { get; set; }
}
Your mapping should look something like:
client.Map<MyType>(m => m
.Properties(ps => ps
.Completion(c => c.Name(x => x.Suggest).Payloads(true))
)
);
Indexing example:
var myType = new MyType
{
Suggest = new SuggestField
{
Input = new [] { "Nevermind", "Nirvana" },
Output = "Nirvana - Nevermind",
Payload = new { id = 1234 },
Weight = 34
}
};
client.Index<MyType>(myType);
Hope that helps.
I store an object in mongodb which contains a Guid. Mongodb converts the Guid to a binary value
{
"_id" : ObjectId("52cf4a467b302a4797db23e8"),
"name" : "test",
"guid" : new BinData(3, "qZ8PQdmDv0+K500wnj6skA=="),
}
I get an empty result set when I use the Guid in a Linq expression and the count is always 0.
var queryable = _database.GetCollection<MyObject>("myname").AsQueryable();
var guid = new Guid("410f9fa9-83d9-4fbf-8ae7-4d309e3eac90");
var count = queryable.Where(x => x.Guid == guid).Count();
The reason is that Linq generates the following request
count: "member_variables", query: { panelid: "410f9fa9-83d9-4fbf-8ae7-4d309e3eac90" }
but the request should be
count: "member_variables", query: { panelid: new BinData(3, "qZ8PQdmDv0+K500wnj6skA==") }
I tried various Linq expressions, but none worked.
queryable.Where(x => x.Guid.ToString() == guid.ToString())
throws the exception: "Unable to determine the serialization information for the expression: x.Guid.ToString()."
queryable.Where(x => x.Guid == new BsonBinaryData(guid));
throws the exception: "Unsupported where clause: ((BsonBinaryData)x.Guid == UuidLegacy:0x24f7ceb84f06d143b6426e8f01cb7825)."
How can I request the documents?
Note: I need to use IQuerable and can't use Mongodb queries.