Elasticsearch NEST client ReindexOnServer type problem - elasticsearch

Basically I want to rewrite this code:
var values = "{\"source\":{\"index\": \"" + oldIndex.Index + "\",\"type\": \"" + MyType+ "\"},"
+ "\",\"size\": \"" + size + "\"},"
+ "\"dest\": {\"index\": \"" + tempIndex + "\",\"type\": \"_doc\"}}";
var result = this.httpClient.PostAsync(ReIndexUrl, new StringContent(values, System.Text.Encoding.UTF8, "application/json")).Result;
var response = result.Content.ReadAsStringAsync().Result;
to NEST client:
var response = elasticClient.ReindexOnServer(
r => r
.Source(
s => s
.Index(sourceIndex)
)
.Destination(
d => d
.Index(targetIndex)
).Refresh(false).Size(maximumDocuments)
.WaitForCompletion()
);
While reindexing docs from V6 to V7 using NEST version, I get errors because of duplicate types on the target:
java.lang.IllegalArgumentException: Rejecting mapping update to [audit-trail-2020-04-temp] as the final mapping would have more than 1 type: [_doc, audittrailentry]
I cannot find a way to workaround this issue using NEST since there is no way to exclude type?

Unfortunately, I dit not find a solution for this using NEST Client.
At the end I wrote my own extension:
internal static long ReIndex(this IElasticClient elasticClient, ReindexModel reindexModel)
{
elasticClient.DisableRefreshing(reindexModel.SourceIndex);
var result = HttpClient.PostAsync(ReIndexUrl, new StringContent(JsonConvert.SerializeObject(reindexModel.Value), Encoding.UTF8, "application/json")).Result;
var response = result.Content.ReadAsStringAsync().Result;
var responseObj = JsonConvert.DeserializeObject<ReindexResponse>(response);
return responseObj.Total;
}
internal class ReindexModel
{
public ReindexModel(string sourceIndex, string targetIndex, string docType, int maxBulkSize)
{
this.SourceIndex = sourceIndex;
this.TargetIndex = targetIndex;
this.DocType = docType;
this.Value = new { source = new { index = this.SourceIndex, type = this.DocType, size = maxBulkSize }, dest = new { index = this.TargetIndex, type = "_doc" } };
}
public object Value { get; }
public string SourceIndex { get; }
public string TargetIndex { get; }
private string DocType { get; }
}
internal class ReindexResponse
{
[JsonProperty]
public long Total { get; set; }
}

Related

Elasticsearch - Trying to index MS Word attachment & making a full text search within

As the title is already indicating, I am trying to index MS Word documents and making a full text search within.
I have seen several examples, but I am not able to figure out what I am doing incorrectly.
Relevant Code:
[ElasticsearchType(Name = "AttachmentDocuments")]
public class Attachment
{
[String(Name = "_content")]
public string Content { get; set; }
[String(Name = "_content_type")]
public string ContentType { get; set; }
[String(Name = "_name")]
public string Name { get; set; }
public Attachment(Task<File> file)
{
Content = file.Result.FileContent;
ContentType = file.Result.FileType;
Name = file.Result.FileName;
}
}
The "Content" property above is set to "file.Result.FileContent" in the constructor. The "Content" property is a base64 string.
public class Document
{
[Number(Name = "Id")]
public int Id { get; set; }
[Attachment]
public Attachment File { get; set; }
public String Title { get; set; }
}
Below is the method for indexing documents to elasticsearch database.
public void IndexDocument(Attachment attachmentDocument)
{
// Create the index if it does not already exist
var indexExists = _client.IndexExists(new IndexExistsRequest(ElasticsearchIndexName));
if (!indexExists.Exists)
{
var indexDescriptor =
new CreateIndexDescriptor(new IndexName {Name = ElasticsearchIndexName}).Mappings(
ms => ms.Map<Document>(m => m.AutoMap()));
_client.CreateIndex(indexDescriptor);
}
var doc = new Document()
{
Id = 1,
Title = "Test",
File = attachmentDocument
};
_client.Index(doc);
}
Based on the code above, the document get indexed into the correct index(Screenshot from Elasticsearch host - Searchly):
Searchly Screenshot
The content in the file is : "VCXCVXCVXCVXCVXVXCVXCV" and with the following query I get zero hits in return:
QueryContainer queryContainer = null;
queryContainer |= new MatchQuery()
{
Field = "file",
Query = "VCXCVXCVXCVXCVXVXCVXCV"
};
var searchResult =
await _client.LowLevel.SearchAsync<string>(ApplicationsIndexName, "document", new SearchRequest()
{
From = 0,
Size = 10,
Query = queryContainer,
Aggregations = GetAggregations()
});
I would appericiate if someone could hint me what I am doing incorrectly or should look into?
Providing screenshot of mapping in my Elasticsearch database:
Elasticsearch - Mapping
Because you refer to wrong field. Field should be file.content
queryContainer |= new MatchQuery()
{
Field = "file.content",
Query = "VCXCVXCVXCVXCVXVXCVXCV"
};

Linq to Sql Using a function to set a value

I wanted to get from a database a IEnumerable<T> and within the 'Select' method using a function to return a string value. But I always get back the
'method cannot be translated into a store expression'
error.
I already took a look all the post on Stack Overflow about the error 'LINQ to Entities does not recognize the method .... and this method cannot be translated into a store expression"
The only way that I found to get around this error is apply the function after the query has run.
void Main()
{
int eventId = 17;
IEnumerable<OccurrenceDropDownList> model = Occurrences.Select (s => new OccurrenceDropDownList
{
OccurrenceId = s.OccurrenceId,
Name = s.Name
})
.AsEnumerable()
.Select(m => new OccurrenceDropDownList
{
OccurrenceId = m.OccurrenceId,
Name = m.Name,
Selected = setSelected(m.OccurrenceId, eventId)
}).AsEnumerable();
foreach(var item in model)
{
Console.WriteLine(item.Name + " - id : " + item.OccurrenceId + " " + item.Selected);
}
}
public class OccurrenceDropDownList
{
public int OccurrenceId { get; set; }
public string Name { get; set; }
public string Selected { get; set; }
}
static string setSelected(int occurrence, int selectedid){
if(occurrence == selectedid){
return "selected";
}
return "";
}
Is there any way to apply the function as result of the first query?
It should be simplier:
int eventId = 17;
IEnumerable<OccurrenceDropDownList> model = Occurrences
.Select(s => new OccurrenceDropDownList
{
OccurrenceId = s.OccurrenceId,
Name = s.Name,
//magic ternary if
Selected = (eventId == s.OccurrenceId) ? "selected" : String.Empty
});
That's all. I used ternary if operator that should be translated to SQL.

How to apply exact instead match by Nest api?

How can I get AEntity where BEntityProp equals "Bprop 3"?
internal class Program
{
public class AEntity
{
public Guid Id { get; set; }
public BEntity BEntityProp { get; set; }
}
public class BEntity
{
public Guid Id { get; set; }
public string Bprop { get; set; }
}
private static void Main(string[] args)
{
var node = new Uri("http://localhost:9200");
var settings = new ConnectionSettings(node, defaultIndex: "default_index");
var client = new ElasticClient(settings);
var entities = new List<AEntity>(100);
for (var i = 0; i < 100; i++)
client.Index(new AEntity
{
Id = Guid.NewGuid(),
BEntityProp =
new BEntity
{
Id = Guid.NewGuid(),
Bprop = "Bprop " + i.ToString(CultureInfo.InvariantCulture)
}
});
Thread.Sleep(1000);
var searchDescriptor = new SearchDescriptor<AEntity>();
//how to apply exact instead match by Nest api?
List<AEntity> expected = client.Search<AEntity>(
searchDescriptor.Query(qd => qd.Match(
mqd => mqd.OnField(x => x.BEntityProp.Bprop).Query("bprop 3")))).Documents.ToList();
try
{
Assert.IsTrue(expected.Count == 1);
}
finally
{
client.DeleteIndex(di => di.Indices("default_index", "default_index" + "*"));
}
}
}
As a result I want to have 1 AEntity with Bprop equals "Bprop 3" but I have all matches "Bprop".
The request looks like:
expected :
{
"query": {
"match": { <-- how to apply exact instead match by Nest api?
"bEntityProp.bprop": {
"query": "bprop 3"
}
}
}
}
I found the ansvwer :
List<AEntity> expected = client.Search<AEntity>(
searchDescriptor.Query(qd => qd.MatchPhrase(
mqd => mqd.OnField(x => x.BEntityProp.Bprop).Query("bprop 3")))).Documents.ToList();
If you want the exact match then use a Term query:
List<AEntity> expected = client.Search<AEntity>(
searchDescriptor.Query(qd => qd.Term(
mqd => mqd.OnField(x => x.BEntityProp.Bprop).Value("bprop 3"))))
.Documents.ToList();
More on Term vs Match can be found in this discussion.

ProtoBuf-Linq error message “ Invalid field in source data: 0”

I've encountered the following issue while using protobuf-linq:
using (var stream = new MemoryStream())
{
SerializeMultiple(PrepareData(), stream);
}
private static void SerializeMultiple(IEnumerable<Person> persons, Stream stream)
{
foreach (var person in persons)
{
Serializer.Serialize(stream, person);
}
stream.Position = 0;
var q = RuntimeTypeModel.Default.AsQueryable<Person>(stream,null);
var results = from e in q
where e.Id % 2 == 0
select new { e.Id, e.Name };
Console.WriteLine("first : " + results.First().Id);
Console.ReadLine();
}
static IEnumerable<Person> PrepareData()
{
for (int i = 0; i < (int) 1e+04; i++)
{
yield return new Person {Id = i, Name= "John" + i, Address = "Address" + i*i};
}
}
[ProtoContract]
class Person
{
[ProtoMember(1)]
public int Id { get; set; }
[ProtoMember(2)]
public string Name { get; set; }
[ProtoMember(3)]
public string Address { get; set; }
}
The AsQueryable line throws the aforementioned exception:
Invalid field in source data: 0
Any thoughts on this matter?
It's not protobuf-linq error. When serializing items into a stream, you should use SerializeWithLengthPrefix to prefix every message with its length, to allow separate them. By default, protobuf-linq uses PrefixStyle.Base128. Below you can find a snippet making it right:
Serializer.SerializeWithLengthPrefix(stream, person, PrefixStyle.Base128);

PaginatedList for pagination for MVC 3 application? Error: has some invalid arguments

I have the following code, I can figure why its invalid argument:
AuditDAL ad = new AuditDAL();
var agencies = ad.SearchAgencies("Ak001", "");
string col = param.sColumns.Split(',')[param.iSortCol_0];
string orderby = col + " " + param.sSortDir_0;
// The best overloaded method match for 'AMS.Helper.PaginatedList.PaginatedList(System.Linq.IQueryable, int, int)' has some invalid arguments C:\NexGen\AMS\DEV\Source\AMS\Controllers\AuditController.cs
var qry = new PaginatedList<AuditAgency>(agencies, param.iDisplayStart, param.iDisplayLength);
PaginatedList Code:
namespace AMS.Helper
{
public class PaginatedList<T> : List<T> {
public int PageIndex { get; private set; }
public int PageSize { get; private set; }
public int TotalCount { get; private set; }
public int TotalPages { get; private set; }
public PaginatedList(IQueryable<T> source, int pageIndex, int pageSize) {
PageIndex = pageIndex;
PageSize = pageSize;
TotalCount = source.Count();
TotalPages = (int) Math.Ceiling(TotalCount / (double)PageSize);
this.AddRange(source.Skip(PageIndex * PageSize).Take(PageSize));
}
public bool HasPreviousPage {
get {
return (PageIndex > 0);
}
}
public bool HasNextPage {
get {
return (PageIndex+1 < TotalPages);
}
}
}
}
Search Agencies Code:
public IEnumerable<AuditAgency> SearchAgencies(string ori, string name)
{
List<AuditAgency> agencies = new List<AuditAgency>();
using (var conn = new SqlConnection(_connectionString))
{
var com = new SqlCommand();
com.Connection = conn;
com.CommandType = CommandType.StoredProcedure;
string term = "Ori";
if (!String.IsNullOrEmpty(ori))
{
term = "Ori";
com.Parameters.Add(new SqlParameter
{
ParameterName = "#ORI",
Value = ori
});
}
if (!String.IsNullOrEmpty(name))
{
term = "legal_name";
com.Parameters.Add(new SqlParameter
{
ParameterName = "#Name",
Value = name
});
}
com.CommandText = "Audit_Get_Agency_List";
var adapt = new SqlDataAdapter();
adapt.SelectCommand = com;
var dataset = new DataSet();
adapt.Fill(dataset);
agencies = (from c in dataset.Tables[0].AsEnumerable()
select new AuditAgency()
{
Agency_ID = Convert.ToInt32(c["Agency_Id"]),
Agency_Name = c["Agency_Name"].ToString(),
Agency_Ori = c["ORI"].ToString(),
COPSAuditNumber = c["COPSAuditNumber"].ToString(),
OIGAuditNumber = c["OIGAuditNumber"].ToString()
}).ToList<AuditAgency>();
return agencies;
}
}
The error should tell you where to start.
If you fire up the debugger, I think you'll find agencies is an IEnumberable, but not an IQueryable
correct it by changing the return type of SearchAgencies from IQueryable to IEnumerable
or alternatively, you can change the type of the PaginatedList to accept IEnumberables instead of IQueryables. this may be safer as IQueryable inherits from IEnumerable
(see
http://msdn.microsoft.com/en-us/library/system.linq.iqueryable.aspx or
Differences between IQueryable, List, IEnumerator?
for the difference between the two)

Resources