Parallel call to Elasticsearch in c# - elasticsearch

I am trying to make a parallel call to Elasticsearch index for multiple queries and aggregate the result, using Parallel.ForEach.
My code:
private static List<SearchResponse<dynamic>> SearchQueryInParallel(string indexName, string lang, string[] eQueries)
{
var result = new List<SearchResponse<dynamic>>();
var exceptions = new ConcurrentQueue<Exception>();
object mutex = new object();
try
{
Parallel.ForEach(eQueries,
() => new SearchResponse<dynamic>()
, (q, loopState, subList) =>
{
var x = LowlevelClient.Search<SearchResponse<dynamic>>(indexName, $"article_{lang}", q);
subList = x;
return subList;
}, subList =>
{
lock (result)
result.Add(subList);
}
);
}
catch (AggregateException ae)
{
foreach (var e in ae.InnerExceptions)
{
exceptions.Enqueue(e);
}
}
if (exceptions.ToList().Any())
{
//there are exceptions, do something with them
//do something?
}
return result;
}
The problem I am facing is that sublist in the above case is long.
It gives me the following error:
Can not convert from SearchResponse to long.
The same thing is working when I used without multithreading, the code is:
var items = new List<dynamic>();
var searchResponse = lowLevelClient.Search<SearchResponse<dynamic>>(elasticIndexName, $"article_{languageCode.ToLowerInvariant()}", query);
foreach (var document in searchResponse.Body.Documents)
{
items.Add(document);
}
Any help, please? If somebody has any other way to achieve the parallel call and aggregating data from returned values would be greatly appreciated.

Related

Mock IDocumentQuery with ability to use query expressions

I need to be able to mock IDocumentQuery, to be able to test piece of code, that queries document collection and might use predicate to filter them:
IQueryable<T> documentQuery = client
.CreateDocumentQuery<T>(collectionUri, options);
if (predicate != null)
{
documentQuery = documentQuery.Where(predicate);
}
var list = documentQuery.AsDocumentQuery();
var documents = new List<T>();
while (list.HasMoreResults)
{
documents.AddRange(await list.ExecuteNextAsync<T>());
}
I've used answer from https://stackoverflow.com/a/49911733/212121 to write following method:
public static IDocumentClient Create<T>(params T[] collectionDocuments)
{
var query = Substitute.For<IFakeDocumentQuery<T>>();
var provider = Substitute.For<IQueryProvider>();
provider
.CreateQuery<T>(Arg.Any<Expression>())
.Returns(x => query);
query.Provider.Returns(provider);
query.ElementType.Returns(collectionDocuments.AsQueryable().ElementType);
query.Expression.Returns(collectionDocuments.AsQueryable().Expression);
query.GetEnumerator().Returns(collectionDocuments.AsQueryable().GetEnumerator());
query.ExecuteNextAsync<T>().Returns(x => new FeedResponse<T>(collectionDocuments));
query.HasMoreResults.Returns(true, false);
var client = Substitute.For<IDocumentClient>();
client
.CreateDocumentQuery<T>(Arg.Any<Uri>(), Arg.Any<FeedOptions>())
.Returns(query);
return client;
}
Which works fine as long as there's no filtering using IQueryable.Where.
My question:
Is there any way to capture predicate, that was used to create documentQuery and apply that predicate on collectionDocuments parameter?
Access the expression from the query provider so that it will be passed on to the backing collection to apply the desired filter.
Review the following
public static IDocumentClient Create<T>(params T[] collectionDocuments) {
var query = Substitute.For<IFakeDocumentQuery<T>>();
var queryable = collectionDocuments.AsQueryable();
var provider = Substitute.For<IQueryProvider>();
provider.CreateQuery<T>(Arg.Any<Expression>())
.Returns(x => {
var expression = x.Arg<Expression>();
if (expression != null) {
queryable = queryable.Provider.CreateQuery<T>(expression);
}
return query;
});
query.Provider.Returns(_ => provider);
query.ElementType.Returns(_ => queryable.ElementType);
query.Expression.Returns(_ => queryable.Expression);
query.GetEnumerator().Returns(_ => queryable.GetEnumerator());
query.ExecuteNextAsync<T>().Returns(x => new FeedResponse<T>(query));
query.HasMoreResults.Returns(true, true, false);
var client = Substitute.For<IDocumentClient>();
client
.CreateDocumentQuery<T>(Arg.Any<Uri>(), Arg.Any<FeedOptions>())
.Returns(query);
return client;
}
The important part is where the expression passed to the query is used to create another query on the backing data source (the array).
Using the following example subject under test for demonstration purposes.
public class SubjectUnderTest {
private readonly IDocumentClient client;
public SubjectUnderTest(IDocumentClient client) {
this.client = client;
}
public async Task<List<T>> Query<T>(Expression<Func<T, bool>> predicate = null) {
FeedOptions options = null; //for dummy purposes only
Uri collectionUri = null; //for dummy purposes only
IQueryable<T> documentQuery = client.CreateDocumentQuery<T>(collectionUri, options);
if (predicate != null) {
documentQuery = documentQuery.Where(predicate);
}
var list = documentQuery.AsDocumentQuery();
var documents = new List<T>();
while (list.HasMoreResults) {
documents.AddRange(await list.ExecuteNextAsync<T>());
}
return documents;
}
}
The following sample tests when an expression is passed to the query
[TestMethod]
public async Task Should_Filter_DocumentQuery() {
//Arrange
var dataSource = Enumerable.Range(0, 3)
.Select(_ => new Document() { Key = _ }).ToArray();
var client = Create(dataSource);
var subject = new SubjectUnderTest(client);
Expression<Func<Document, bool>> predicate = _ => _.Key == 1;
var expected = dataSource.Where(predicate.Compile());
//Act
var actual = await subject.Query<Document>(predicate);
//Assert
actual.Should().BeEquivalentTo(expected);
}
public class Document {
public int Key { get; set; }
}

Catching exceptions from list of tasks

I would like to know the proper way to handle exceptions in scenario like
---Updated code with solution---
var wt = new List<PendingSettlement>();
var ot = new List<PendingSettlement>();
var otTask = Task.Run(async () => ot = await LoadOtDataAsync().ConfigureAwait(false));
var wtTask = Task.Run(async () => wt = await LoadWtDataAsync().ConfigureAwait(false));
var tasks = new List<Task>
{
otTask, wtTask
};
try
{
await Task.WhenAll(tasks);
}
catch
{
run.WarningMessage = GetInnermostExceptionMessage(wtTask.Exception) + Environment.NewLine + GetInnermostExceptionMessage(otTask.Exception);
string GetInnermostExceptionMessage(Exception e)
{
if (e == null) return string.Empty;
_logger.LogError(e, e.Message);
while (e.InnerException != null) e = e.InnerException;
return e.Message;
}
}
Is there way to figure which inner exception corresponds to which task?
Store the tasks (e.g. var t1 = Task.Run(...);). Then, you can query the exception using t1.Exception. You can simply ignore the exception you get from done.Wait(); then. I'd do that by writing done.ContinueWith(_ => { }).Wait(); to avoid throwing and catching.

Lucene.net Negation clause is not working

I am very much new to Lucene.net and though I am not able to achieve basic functionality i.e. Not in.
My requirement is to search "road?construction" without "Works" word.
e.g.
Main Road Construction Works -- Invalid
Road Construction And Maintenance Services -- Valid (Doesn't contains word Works)
Please refer my code below.
string searchQuery = "\"road?construction\"*";
BooleanQuery query2 = new BooleanQuery();
Query query;
try
{
query = parser.Parse(searchQuery.Trim());
}
catch (ParseException)
{
query = parser.Parse(QueryParser.Escape(searchQuery.Trim()));
}
query2.Add(query,Occur.SHOULD);
query2.Add(new BooleanClause(new TermQuery (new Term("Name", "Works")), Occur.MUST_NOT));
This still gets both above mentioned record in to search result. I wish to cut invalid record(first).
Here is the result query generated in backend.
Please suggest workaround.
Thanks in advanced.
Not sure why your putting wildcard characters into the phrase. If you're looking for "road construction" then that's all you need. If you are looking to allow some variations then maybe a "slop phrase" is what you need ie. "road construction"~2. The number part allows for n "operations" like n additional words inbetween.
Here's a set of tests that show your examples (TestExpr2, TestExpr3) and some working variations (TestExpr1 and TestQuery).
Hope this helps
[TestClass]
public class UnitTest7
{
[TestMethod]
public void TestExpr1()
{
TestExpr("\"road construction\" -works");
}
[TestMethod]
public void TestExpr2()
{
TestExpr("\"road?construction\"* -works");
}
[TestMethod]
public void TestExpr3()
{
TestExpr(QueryParser.Escape("\"road?construction\"* -works"));
}
private void TestExpr(string expr)
{
var writer = CreateIndex();
Add(writer, "Main Road Construction Works");
Add(writer, "Road Construction And Maintenance Services");
writer.Flush(true, true, true);
var searcher = new IndexSearcher(writer.GetReader());
var result = Search(searcher, expr);
Assert.AreEqual(1, result.Count);
Assert.IsTrue(result.Contains("Road Construction And Maintenance Services"));
writer.Dispose();
}
[TestMethod]
public void TestQuery()
{
var writer = CreateIndex();
Add(writer, "Main Road Construction Works");
Add(writer, "Road Construction And Maintenance Services");
writer.Flush(true, true, true);
var searcher = new IndexSearcher(writer.GetReader());
var query = new BooleanQuery();
var p = new PhraseQuery();
p.Add(new Term("name", "road"));
p.Add(new Term("name", "construction"));
query.Add(p, Occur.MUST);
query.Add(new TermQuery(new Term("name", "works")), Occur.MUST_NOT);
var result = Search(searcher, query);
Assert.AreEqual(1, result.Count);
Assert.IsTrue(result.Contains("Road Construction And Maintenance Services"));
writer.Dispose();
}
private List<string> Search(IndexSearcher searcher, string expr)
{
var analyzer = new StandardAnalyzer(Lucene.Net.Util.Version.LUCENE_30);
var queryParser = new QueryParser(Lucene.Net.Util.Version.LUCENE_30, "name", analyzer);
var query = queryParser.Parse(expr);
return Search(searcher, query);
}
private List<string> Search(IndexSearcher searcher, Query query)
{
var collector = TopScoreDocCollector.Create(10, true);
searcher.Search(query, collector);
var result = new List<string>();
var matches = collector.TopDocs().ScoreDocs;
foreach (var item in matches)
{
var id = item.Doc;
var doc = searcher.Doc(id);
result.Add(doc.GetField("name").StringValue);
}
return result;
}
IndexWriter CreateIndex()
{
var directory = new RAMDirectory();
var analyzer = new StandardAnalyzer(Lucene.Net.Util.Version.LUCENE_30);
var writer = new IndexWriter(directory, analyzer, new IndexWriter.MaxFieldLength(1000));
return writer;
}
void Add(IndexWriter writer, string text)
{
var document = new Document();
document.Add(new Field("name", text, Field.Store.YES, Field.Index.ANALYZED));
writer.AddDocument(document);
}
}

LINQ - Operator to be used to return all records

What operator needs to be used when I want all the records to be returned irrespective I have one or multiple record.
Below is the example:
public static async Task<SampleDataItem> GetFavouriteItemAsync(string uniqueId)
{
List<string> lst = new List<string>() { "11","12"};
await _sampleDataSource.GetSampleDataAsync();
// Simple linear search is acceptable for small data sets
var matches = _sampleDataSource.Groups.SelectMany(group => group.Items).Where(item=>lst.Contains(item.UniqueId));
if (matches.Count() !=0) return (SampleDataItem)matches.AsEnumerable();
return null;
}
matches.First() return only first record; however I want all the records present in matches.
public static async Task<List<SampleDataItem>> GetFavouriteItemAsync(string uniqueId)
{
List<string> lst = new List<string>() { "11","12"};
await _sampleDataSource.GetSampleDataAsync();
// Simple linear search is acceptable for small data sets
var matches = _sampleDataSource.Groups.SelectMany(group => group.Items).Where(item=>lst.Contains(item.UniqueId));
if (matches.Any())
{
var ReturnTask = Task.Factory.StartNew(() => matches .ToList());
return await ReturnTask;
}
return null;
}

Getting data from a Linq DbContext to a stored procedure

I am calling a stored procedure using Linq. I want to put that data in list. Can anyone help me how to do this? You can see my code here:
public getidrange(int startrange, int endrange)
{
List<string> idRange = new List<string>();
string Conn = System.Configuration.ConfigurationManager.ConnectionStrings["mydbconnection"].ToString();
GetIDRangeDataContext GetIdRange = new GetIDRangeDataContext(Conn);
try
{
GetIdRange.sp_GetIdRange(startRange, endRange);
}
catch(Exception ex)
{
}
return idRange;
}
I just did the following to and it worked .
try
{
var res = GetIdRange.sp_GetIdRange(startRange, endRange);
foreach (var ids in res)
{
idRange.Add(ids.NewIDAnalisi.ToString());
objGetIdRangeResults.success = true;
}

Resources