Elasticsearch - NEST - Upsert - elasticsearch

I have the following two classes
Entry
public class Entry
{
public Guid Id { get; set; }
public IEnumerable<Data> Data { get; set; }
}
EntryData
public class EntryData
{
public string Type { get; set; }
public object Data { get; set; }
}
I have a bunch of different applications that produces messages to a queue that I then consume in a separate application to store that data in elasticsearch.
Im using a CorrelationId for all the messages and I want to use this ID as the ID in elasticsearch.
So given the following data:
var id = Guid.Parse("1befd5b62b944b4aa600c85632159e11");
var entries = new List<Entry>
{
new Entry
{
Id = id,
Data = new List<EntryData>
{
new EntryData
{
Data = DateTime.UtcNow,
Type = "APPLICATION1_Received"
},
new EntryData
{
Data = DateTime.UtcNow,
Type = "APPLICATION1_Validated"
},
new EntryData
{
Data = DateTime.UtcNow,
Type = "APPLICATION1_Published"
},
}
},
new Entry
{
Id = id,
Data = new List<EntryData>
{
new EntryData
{
Data = DateTime.UtcNow,
Type = "APPLICATION2_Received"
},
new EntryData
{
Data = DateTime.UtcNow,
Type = "APPLICATION2_Validated"
},
new EntryData
{
Data = DateTime.UtcNow,
Type = "APPLICATION2_Published"
},
}
},
new Entry
{
Id = id,
Data = new List<EntryData>
{
new EntryData
{
Data = DateTime.UtcNow,
Type = "APPLICATION3_Received"
},
new EntryData
{
Data = DateTime.UtcNow,
Type = "APPLICATION3_Validated"
},
new EntryData
{
Data = DateTime.UtcNow,
Type = "APPLICATION3_Published"
},
}
},
};
I want this to be saved as one entry in elasticsearch where ID == 1befd5b6-2b94-4b4a-a600-c85632159e11 and a data array that contains 9 elements.
Im struggling a bit with getting this to work, when trying the following:
var result = await _elasticClient.BulkAsync(x => x.Index("journal").UpdateMany(entries, (descriptor, entry) => {
descriptor.Doc(entry);
return descriptor.Upsert(entry);
}), cancellationToken);
But this is just overwriting whatever already exists in the data array and the count is 3 instead of 9 (only the Application3 entries are saved).
So, is it possible to do what I want to do? I never worked with elasticsearch before so it feels like maybe Im missing something simple here... :)

Managed to solve it like this:
var result = await _elasticClient.BulkAsync(x => x.Index("journal").UpdateMany(entries, (descriptor, entry) => {
var script = new InlineScript("ctx._source.data.addAll(params.data)")
{
Params = new Dictionary<string, object> {{"data", entry.Data}}
};
descriptor.Script(b => script);
return descriptor.Upsert(entry);
}), cancellationToken);

Related

NEST (ElasticSearch) search response drops aggregates

Here is a query that works in ElasticSearch.
"query":{
"match_all":{
}
},
"size":20,
"aggs":{
"CompanyName.raw":{
"terms":{
"field":"CompanyName.raw",
"size":20,
"order":{
"_count":"desc"
}
}
}
}
}
The response from ElasticSearch has a property aggregations['CompanyName.raw']['buckets'] which is an array.
I use this code to exeute the same query via NEST
string responseJson = null;
ISearchResponse<ProductPurchasing> r = Client.Search<ProductPurchasing>(rq);
using (MemoryStream ms = new MemoryStream())
{
Client.RequestResponseSerializer.Serialize<ISearchResponse<ProductPurchasing>>(r, ms);
ms.Position = 0;
using (StreamReader sr = new StreamReader(ms))
{
responseJson = sr.ReadToEnd();
}
}
However, in the resulting responseJson this array is always empty.
Where hs it gone?
How can I get it back?
Or is it that NEST doesn't support aggregates?
NEST does support aggregation, you can have a look into docs on how to handle aggregation response with NEST help.
Here you can find a short example of writing and retrieving data from simple terms aggregation:
class Program
{
public class Document
{
public int Id { get; set; }
public string Name { get; set; }
public string Brand { get; set; }
public string Category { get; set; }
public override string ToString() => $"Id: {Id} Name: {Name} Brand: {Brand} Category: {Category}";
}
static async Task Main(string[] args)
{
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var connectionSettings = new ConnectionSettings(pool);
connectionSettings.DefaultIndex("documents");
var client = new ElasticClient(connectionSettings);
var deleteIndexResponse = await client.Indices.DeleteAsync("documents");
var createIndexResponse = await client.Indices.CreateAsync("documents", d => d
.Map(m => m.AutoMap<Document>()));
var indexDocument = await client
.IndexDocumentAsync(new Document {Id = 1, Brand = "Tommy", Category = "men"});
var indexDocument2 = await client
.IndexDocumentAsync(new Document {Id = 2, Brand = "Diesel", Category = "men"});
var indexDocument3 = await client
.IndexDocumentAsync(new Document {Id = 3, Brand = "Boss", Category = "men"});
var refreshAsync = client.Indices.RefreshAsync();
var searchResponse = await client.SearchAsync<Document>(s => s
.Query(q => q.MatchAll())
.Aggregations(a => a
.Terms("brand", t => t
.Field(f => f.Brand.Suffix("keyword")))));
var brands = searchResponse.Aggregations.Terms("brand");
foreach (var bucket in brands.Buckets)
{
Console.WriteLine(bucket.Key);
}
}
}
Prints:
Boss
Diesel
Tommy
Hope that helps.

MongoDB search nested objects in an array

I am using MongoDB to store all the events in my Eventbrite clone. So I have a collection called events then the objects in this collection consists of their name and and array of users that have rsvp to the event. I can query for any events that the current user has created but unable to figure out how to query both events the user has created and rsvp to.
Here is the compiled query that I am using to try to get all the users events.
events.find({"$and":[{"user_id":"5d335704802df000076bad97"},{"user_id":{"$ne":null}}],"$or":[{"checkins.user_id":"5d335704802df000076bad97"}]},{"typeMap":{"root":"array","document":"array"}})
I am using the Laravel MongoDB plugin to query my data in php it looks like this
$user->events()->orWhere(function ($query) use ($user){
return $query->where('checkins.user_id',new ObjectID($user->id));
})->get()
The event object looks something like this:
{
"name": "test",
"user_id": "1"
"rsvp": [
{"user_id": "12"}
]
}
An user can rsvp to other event that are not their own.
you need an $or filter and $elemMatch to get events that belong to a given user or events they've rsvp'd to.
db.events.find({
"$or": [
{
"user_id": "5d33e732e1ea9d0d6834ef3d"
},
{
"rsvp": {
"$elemMatch": {
"user_id": "5d33e732e1ea9d0d6834ef3d"
}
}
}
]
})
unfortunately i can't help you with laravel version of the query. in case it helps, below is the c# code that generated the above mongo query.
using MongoDB.Entities;
using System.Linq;
namespace StackOverflow
{
public class Program
{
public class user : Entity
{
public string name { get; set; }
}
public class Event : Entity
{
public string name { get; set; }
public string user_id { get; set; }
public rsvp[] rsvp { get; set; }
}
public class rsvp
{
public string user_id { get; set; }
}
private static void Main(string[] args)
{
new DB("test");
var mike = new user { name = "mike" };
var dave = new user { name = "dave" };
mike.Save();
dave.Save();
(new[] {
new Event
{
name = "mike's event",
user_id = mike.ID,
rsvp = new[]
{
new rsvp { user_id = dave.ID }
}
},
new Event
{
name = "dave's event",
user_id = dave.ID,
rsvp = new[]
{
new rsvp { user_id = mike.ID }
}
}
}).Save();
var result = DB.Find<Event>()
.Many(e =>
e.user_id == mike.ID ||
e.rsvp.Any(r => r.user_id == mike.ID));
}
}
}

How to working with data from client side using Observables

I am using the Grid of Kendo (Angular 2) for Add/Edit/Delete a Row in the grid:
http://www.telerik.com/kendo-angular-ui/components/grid/editing/
In the original Code, the data is obtained from a rest service like this:
private fetch(action: string = "", data?: Product): Observable<Product[]> {
return this.jsonp
.get(`http://demos.telerik.com/kendo-ui/service/Products/${action}? callback=JSONP_CALLBACK${this.serializeModels(data)}`)
.map(response => response.json());
}
But, I want to work with a array for add/edit/delete rows in memory. Next, I want to do click in the button submit and send the data (with all my changes) to the server.
My solution for this is like this:
https://gist.github.com/joedayz/9e318a47d06a7a8c2170017eb133a87e
Overview:
I declare an array:
private view: Array = [{ProductID: 1, ProductName: "pelotas", Discontinued: undefined, UnitsInStock: 80}];
and override the fetch method like this:
private fetch(action: string = "", data?: Product): Observable<Product[]> {
/*return this.jsonp
.get(`http://demos.telerik.com/kendo-ui/service/Products/${action}?callback=JSONP_CALLBACK${this.serializeModels(data)}`)
.map(response => response.json());*/
debugger;
if(action=="create"){
var product : Product = new Product(-1, data.ProductName, data.Discontinued, data.UnitsInStock);
this.view.push(product);
}else if(action=="update"){
var indice = this.view.indexOf(data);
if(indice>=0)
this.view[indice] = (JSON.parse(JSON.stringify(data)));
}else if(action=="destroy"){
var indice = this.view.indexOf(data);
if(indice>=0)
this.view.splice(indice, 1);
}
return Observable.of(this.view);
}
My Question is: Exists some way of communicate the create/update/delete of items in my array of a simple or reactive form to my grid?
As you are using in-memory array you do not need to use Observables. The Grid component is already bound to the array, thus it is just necessary to manipulate the data. For example:
export class AppComponent {
public dataItem: Product;
#ViewChild(GridEditFormComponent) protected editFormComponent: GridEditFormComponent;
private view: Array<Product> = [{ ProductID: 1, ProductName: "pelotas", Discontinued: undefined, UnitsInStock: 80 }];
public onEdit(dataItem: any): void {
this.dataItem = Object.assign({}, dataItem);
}
public onCancel(): void {
this.dataItem = undefined;
}
public addProduct(): void {
this.editFormComponent.addProduct();
}
public onSave(product: Product): void {
if (product.ProductID === undefined) {
this.createProduct(product);
} else {
this.saveProducts(product);
}
}
public onDelete(e: Product): void {
this.deleteProduct(e);
}
public saveProducts(data: Product): void {
var index = this.view.findIndex(x => x.ProductID === data.ProductID);
if (index !== -1) {
this.view = [
...this.view.slice(0, index),
data,
...this.view.slice(index + 1)
];
}
}
public createProduct(data: Product): void {
this.view = [...this.view, data];
}
public deleteProduct(data: Product): void {
this.view = this.view.filter(x => x.ProductID !== data.ProductID);
}
}

Not able to get TermVector results properly in SolrNet

I'm not able to get TermVector results properly thru SolrNet. I tried with the following code.
QueryOptions options = new QueryOptions()
{
OrderBy = new[] { new SortOrder("markupId", Order.ASC) },
TermVector = new TermVectorParameters
{
Fields = new[] { "text" },
Options = TermVectorParameterOptions.All
}
};
var results = SolrMarkupCore.Query(query, options);
foreach (var docVectorResult in results.TermVectorResults)
{
foreach (var vectorResult in docVectorResult.TermVector)
System.Diagnostics.Debug.Print(vectorResult.ToString());
}
In the above code, results.TermVectorResults in the outer foreach gives the proper count whereas docVectorResult.TermVector in the inner foreach is empty.
I've copied the generated solr query of the above code and issued against solr admin and I'm properly getting the termVectors values. The actual query I issued is below
http://localhost:8983/solr/select/?sort=markupId+asc&tv.tf=true&start=0&q=markupId:%2823%29&tv.offsets=true&tv=true&tv.positions=true&tv.fl=text&version=2.2&rows=50
First you should check HTTP query to sure termvector feature is set property.
If it's not OK, change your indexing based on:
The Term Vector Component
If it is OK,You can use "ExtraParams" by changing the handler to termvector handler. Try this:
public SolrQueryExecuter<Product> instance { get; private set; }
public ICollection<TermVectorDocumentResult> resultDoc(string q)
{
string SERVER="http://localhost:7080/solr/core";//change this
var container = ServiceLocator.Current as SolrNet.Utils.Container;
instance = new SolrQueryExecuter<Product>(
container.GetInstance<ISolrAbstractResponseParser<Product>>(),
new SolrConnection(SERVER),
container.GetInstance<ISolrQuerySerializer>(),
container.GetInstance<ISolrFacetQuerySerializer>(),
container.GetInstance<ISolrMoreLikeThisHandlerQueryResultsParser<Product>>());
instance.DefaultHandler = "/tvrh";
SolrQueryResults<Product> results =
instance.Execute(new SolrQuery(q),
new QueryOptions
{
Fields = new[] { "*" },
Start = 0,
Rows = 10,
ExtraParams = new Dictionary<string, string> {
{ "tv.tf", "false" },
{ "tv.df", "false" },
{ "tv.positions", "true" },
{ "tv", "true" },
{ "tv.offsets", "false" },
{ "tv.payloads", "true" },
{ "tv.fl", "message" },// change the field name here
}
}
);
return results.TermVectorResults;
}

How to project simple data set into to anonymous type of a different shape

I have a simple set of data that I am having trouble figuring out how to create the projection I want using LINQ.
public class Score {
public string Name { get; set; }
public int Value { get; set; }
}
var scores = new List<Score> {
new Score { Name = "jesse", Value = 10 },
new Score { Name = "jesse", Value = 12 },
new Score { Name = "jesse", Value = 15 },
new Score { Name = "billy", Value = 5 },
new Score { Name = "billy", Value = 7 },
new Score { Name = "billy", Value = 20 },
new Score { Name = "colin", Value = 25 },
new Score { Name = "colin", Value = 13 },
new Score { Name = "colin", Value = 8 }
};
I need to project 'scores' into an anonymous type with the following structure.
{
series : [
{ name : "jesse", data : [10, 12, 15 ] },
{ name : "billy", data : [ 5, 7, 20 ] },
{ name : "colin", data : [25, 13, 8 ] }
]
}
Any help is appreciated.
var result = new {
series = from score in scores
group score.Value by score.Name into nameValues
select new
{
name = nameValues.Key,
data = nameValues
}
};
Does this match the structure you want?
var query = scores.GroupBy(s => s.Name);
var result = query.Select(q => new {
Name = q.Key,
Data = q.ToArray().Select(k => k.Value)
});
var anotherAnon = new {series = result.ToArray()};

Resources