Here is my question. Due to project needs, we have to keep our dates within elasticsearch index in the same format. What we've tried is the next way -
var connectionPool = new SniffingConnectionPool(nodeList);
var connectionSettings = new ConnectionSettings(connectionPool)
.SetJsonSerializerSettingsModifier(
m => m.DateFormatString = "yyyy-MM-ddTHH:mm:ss.fffffffK")
// other configuration goes here
But it didn't work out. Searching through ES index, I saw dates with dropped trailing zeros ( like 2015-05-05T18:55:27Z insted of expected 2015-05-05T18:55:27.0000000Z). Neither did next option help:
var connectionPool = new SniffingConnectionPool(nodeList);
var connectionSettings = new ConnectionSettings(connectionPool)
.SetJsonSerializerSettingsModifier(m =>
{
m.Converters.Add(new IsoDateTimeConverter { DateTimeFormat = "yyyy'-'MM'-'dd'T'HH':'mm':'ss.fffffffK"});
})
// other configuration goes here
With digging into ElasticClient at run-time, I've found that eventually there is a contract resolver which seems like overrides all those settings:
public class ElasticContractResolver : DefaultContractResolver
{
protected override JsonContract CreateContract(Type objectType)
{
JsonContract contract = base.CreateContract(objectType);
...
if (objectType == typeof(DateTime) || objectType == typeof(DateTime?))
contract.Converter = new IsoDateTimeConverter();
...
if (this.ConnectionSettings.ContractConverters.HasAny())
{
foreach (var c in this.ConnectionSettings.ContractConverters)
{
var converter = c(objectType);
if (converter == null)
continue;
contract.Converter = converter;
break;
}
}
return contract;
}
}
So if I have it right, without specifying a converter explicitly(via Connection Settings.AddContractJsonConverters()), my json settings will be gone since IsoDateTimeConverter is instantiated with the default settings rather than ones I've passed through SetJsonSerializerSettingsModifier.
Has anyone run into this issue? Or I'm just missing something? Thanks in advance!
This is how I handled custom date format for my needs:
public class Document
{
[ElasticProperty(DateFormat = "yyyy-MM-dd", Type = FieldType.Date)]
public string CreatedDate { get; set; }
}
client.Index(new Document {CreatedDate = DateTime.Now.ToString("yyyy-MM-dd")});
My document in ES
{
"_index": "indexname",
"_type": "document",
"_id": "AU04kd4jnBKFIw7rP3gX",
"_score": 1,
"_source": {
"createdDate": "2015-05-09"
}
}
Hope it will help you.
Related
This question already has an answer here:
ElasticSearch 5.x Context Suggester NEST .Net
(1 answer)
Closed 4 years ago.
I"m trying to setup context for my completion suggestion attribute on my POJO class however, most of the documentation online are based on configuration setting. Any ideas how I can achieve this?
[Completion]
public CompletionField Suggest { get; set; }
For indexing:
[Completion(Name = "FieldNameSuggest")]
public CompletionField Suggest
{
get
{
List<string> data = new List<string>();
data.AddRange(new List<string>() {"word1",word2"word3",... });//for suggest
return new CompletionField()
{
Input = data,
Weight = 1
};
}
}
For get suggest:
Query Making:
var sugContainer = new SuggestContainer
{
{ "completion-suggest", new SuggestBucket
{
Prefix ="word",//text for search
Completion = new CompletionSuggester
{
//Fuzzy = new FuzzySuggester
//{
// Fuzziness = Fuzziness.Auto,
// MinLength = 1,
// PrefixLength = 2,
// Transpositions = true,
// UnicodeAware = false
//},
Analyzer = "simple",
Field =new Field("FieldNameSuggest"),
Size =10, //SuggestionCount
SkipDuplicates=true,
}
}
}
};
Finaly for search:
var Result = Client.Search<T>(new SearchRequest<T>
{
Suggest = sugContainer ,
...
});
I would like to consume my organizations dynamics oData endpoint but with early bound classes. However, there are a lot of early bound tools out there and I wanted to know which one provides the best developer experience/least resistance?
For example, there is this one:
https://github.com/daryllabar/DLaB.Xrm.XrmToolBoxTools
https://github.com/yagasoft/DynamicsCrm-CodeGenerator
and so on. Is there a developer preference/method out there?
Early bound classes are for use with the Organization Service which is a SOAP service. The normal way to generate those classes is using CrmSvcUtil.
OData can be used in Organization Data Service or Web API, but those don't have Early Bound classes.
Further reading: Introducing the Microsoft Dynamics 365 web services
It's not impossible to use with standard SOAP Early bound class. We just have to be creative. If we work just with basic attributes (fields, not relationships, ecc) it seems possible. For example. for create and update, OData will not accept the entire early bounded class, just pass the attibutes:
class Program
{
static void Main(string[] args)
{
string token = System.Threading.Tasks.Task.Run(() => GetToken()).Result;
CRMWebAPI dynamicsWebAPI = new CRMWebAPI("https:/ORG.api.crm4.dynamics.com/api/data/v9.1/",
token);
CRMGetListOptions listOptions = new CRMGetListOptions
{
Select = new string[] { "EntitySetName" },
Filter = "LogicalName eq 'contact'"
};
dynamic entityDefinitions = dynamicsWebAPI.GetList<ExpandoObject>("EntityDefinitions", listOptions).Result;
Contact contact = new Contact
{
FirstName = "Felipe",
LastName = "Test",
MobilePhone = "38421254"
};
dynamic ret = System.Threading.Tasks.Task.Run(async () => await dynamicsWebAPI.Create(entityDefinitions.List[0].EntitySetName, KeyPairValueToObject(contact.Attributes))).Result;
}
public static async Task<string> GetToken()
{
string api = "https://ORG.api.crm4.dynamics.com/";
ClientCredential credential = new ClientCredential("CLIENT_ID", "CLIENT_SECRET");
AuthenticationContext authenticationContext = new AuthenticationContext("https://login.microsoftonline.com/commom/oauth2/authorize");
return authenticationContext.AcquireTokenAsync(api, credential).Result.AccessToken;
}
public static object KeyPairValueToObject(AttributeCollection keyValuePairs)
{
dynamic expando = new ExpandoObject();
var obj = expando as IDictionary<string, object>;
foreach (var keyValuePair in keyValuePairs)
obj.Add(keyValuePair.Key, keyValuePair.Value);
return obj;
}
}
It's a simple approach and I didn't went further.
Maybe we have to serealize other objects as OptionSets, DateTime (pass just the string) and EntityReferences but this simple test worked fine to me. I'm using Xrm.Tools.WebAPI and Microsoft.IdentityModel.Clients.ActiveDirectory. Maybe it's a way.
[Edit]
And so I decided to go and created a not well tested method to cast the attributes. Problems: We have to follow OData statments to use the API. To update/create an entity reference we can use this to reference https://www.inogic.com/blog/2016/02/set-values-of-all-data-types-using-web-api-in-dynamics-crm/
So
//To EntityReference
entityToUpdateOrCreate["FIELD_SCHEMA_NAME#odata.bind"] = "/ENTITY_SET_NAME(GUID)";
So, it's the Schema name, not field name. If you use CamelCase when set you fields name you'll have a problem where. We can resolve that with a (to that cute) code
public static object EntityToObject<T>(T entity) where T : Entity
{
dynamic expando = new ExpandoObject();
var obj = expando as IDictionary<string, object>;
foreach (var keyValuePair in entity.Attributes)
{
obj.Add(GetFieldName(entity, keyValuePair), CastEntityAttibutesValueOnDynamicObject(keyValuePair.Value));
}
return obj;
}
public static object CastEntityAttibutesValueOnDynamicObject(object attributeValue)
{
if (attributeValue.GetType().Name == "EntityReference")
{
CRMGetListOptions listOptions = new CRMGetListOptions
{
Select = new string[] { "EntitySetName" },
Filter = $"LogicalName eq '{((EntityReference)attributeValue).LogicalName}'"
};
dynamic entitySetName = dynamicsWebAPI.GetList<ExpandoObject>("EntityDefinitions", listOptions).Result.List[0];
return $"/{entitySetName.EntitySetName}({((EntityReference)attributeValue).Id})";
}
else if (attributeValue.GetType().Name == "OptionSetValue")
{
return ((OptionSetValue)attributeValue).Value;
}
else if (attributeValue.GetType().Name == "DateTime")
{
return ((DateTime)attributeValue).ToString("yyyy-MM-dd");
}
else if (attributeValue.GetType().Name == "Money")
{
return ((Money)attributeValue).Value;
}
else if (attributeValue.GetType().Name == "AliasedValue")
{
return CastEntityAttibutesValueOnDynamicObject(((AliasedValue)attributeValue).Value);
}
else
{
return attributeValue;
}
}
public static string GetFieldName<T>(T entity, KeyValuePair<string, object> keyValuePair) where T : Entity
{
switch (keyValuePair.Value.GetType().Name)
{
case "EntityReference":
var entityNameList = System.Threading.Tasks.Task.Run(async () => await dynamicsWebAPI.GetEntityDisplayNameList()).Result;
var firstEntity = entityNameList.Where(x => x.LogicalName == entity.LogicalName).FirstOrDefault();
var attrNameList = System.Threading.Tasks.Task.Run(async () => await dynamicsWebAPI.GetAttributeDisplayNameList(firstEntity.MetadataId)).Result;
return attrNameList.Where(x => x.LogicalName == keyValuePair.Key).Single().SchemaName + "#odata.bind";
case "ActivityParty":
throw new NotImplementedException(); //TODO
default:
return keyValuePair.Key;
}
}
Please, note that this approach do not seems fast or good in anyway. It's better if you have all this values as static so we can save some fetches
[Edit 2]
I just found on XRMToolBox a plugin called "Early bound generator for Web API" and it seems to be the best option. Maybe you should give it a try if you're still curious about that. I guess its the best approach.
The final code is this:
static void Main(string[] args)
{
string token = Task.Run(() => GetToken()).Result;
dynamicsWebAPI = new CRMWebAPI("https://ORG.api.crm4.dynamics.com/api/data/v9.1/",
token);
Contact contact = new Contact
{
FirstName = "Felipe",
LastName = "Test",
MobilePhone = "38421254",
new_Salutation = new EntityReference(new_salutation.EntitySetName, new Guid("{BFA27540-7BB9-E611-80EE-FC15B4281C8C}")),
BirthDate = new DateTime(1993, 04, 14),
};
dynamic ret = Task.Run(async () => await dynamicsWebAPI.Create(Contact.EntitySetName, contact.ToExpandoObject())).Result;
Contact createdContact = dynamicsWebAPI.Get<Contact>(Contact.EntitySetName, ret, new CRMGetListOptions
{
Select = new string[] { "*" }
}).Result;
}
and you have to change the ToExpandoObject on Entity.cs class (generated by the plugin)
public ExpandoObject ToExpandoObject()
{
dynamic expando = new ExpandoObject();
var expandoObject = expando as IDictionary<string, object>;
foreach (var attributes in Attributes)
{
if (attributes.Key == GetIdAttribute())
{
continue;
}
var value = attributes.Value;
var key = attributes.Key;
if (value is EntityReference entityReference)
{
value = $"/{entityReference.EntitySetName}({entityReference.EntityId})";
}
else
{
key = key.ToLower();
if (value is DateTime dateTimeValue)
{
var propertyForAttribute = GetPublicInstanceProperties().FirstOrDefault(x =>
x.Name.Equals(key, StringComparison.InvariantCultureIgnoreCase));
if (propertyForAttribute != null)
{
var onlyDateAttr = propertyForAttribute.GetCustomAttribute<OnlyDateAttribute>();
if (onlyDateAttr != null)
{
value = dateTimeValue.ToString(OnlyDateAttribute.Format);
}
}
}
}
expandoObject.Add(key, value);
}
return (ExpandoObject)expandoObject;
}
Links:
https://github.com/davidyack/Xrm.Tools.CRMWebAPI
https://www.xrmtoolbox.com/plugins/crm.webApi.earlyBoundGenerator/
We currently use XrmToolkit which has it's own version of early binding called ProxyClasses but will allow you to generate early binding using the CRM Service Utility (CrmSvcUtil). It does a lot more than just early binding which is why we use it on all of our projects but the early binding features alone would have me sold on it. in order to regenerate an entity definition all you do is right click the cs file in visual studio and select regenerate and it is done in a few seconds.
For my first 3 years of CRM development I used the XrmToolbox "Early Bound Generator" plugin which is really helpful as well.
How can I specify default analyzer in NEST? Or alternative in Elasticsearch? I want change standard analyzer to language analyzer!
If you are using automap in nest you can use an attribute like so
public class A
{
[Text(Analyzer = "NameOfTheAnalyzer")]
public string Prop1 { get; set; }
}
If you want the default mapping you can set it like so
var request = new CreateIndexRequest(indexName)
{
Mappings = new Mappings()
{
["_default_"] = new TypeMapping()
{
Properties = new Properties
{
["id"] = new KeywordProperty { Index = false },
["title"] = new TextProperty { Analyzer = "NameOfTheAnalyzer" }
}
}
}
};
var create = client.CreateIndex(request);
I'm trying to port an Android app with a Realm to Xamarin so it'll be also available for iOS devices. In Android, I have several JSON files with some necessary initial data, e.g. cities.json, and I import it at the beginning with realm.createOrUpdateAllFromJson(Class<E> clazz, InputStream in) method, like this:
private void loadInitialCities(Realm realm) {
InputStream stream = context.getAssets().open("data/cities.json");
realm.createOrUpdateAllFromJson(City.class, stream);
}
I also find this method very useful when retrieving data from a web service in form of JSON.
Now with Xamarin I don't see any equivalent to such method. Is there any method to achieve this? Or at least a workaround/tool to create a RealmObject from a JSON in C#?
I wrote my own extension methods for doing this (yes, I miss the built-in helper methods also).
https://github.com/sushihangover/Realm.Json.Extensions
Here is a basic example of how I do it:
JSON Model:
[
{
"name": "Alabama",
"abbreviation": "AL"
},
{
"name": "Alaska",
"abbreviation": "AK"
},
~~~~
]
Realm Model:
public class State : RealmObject
{
public string name { get; set; }
public string abbreviation { get; set; }
}
Xamarin.Android asset and Newtonsoft Streaming reader:
var config = RealmConfiguration.DefaultConfiguration;
config.SchemaVersion = 1;
using (var theRealm = Realm.GetInstance("StackOverflow.realm"))
using (var assetStream = Assets.Open("States.json"))
using (var streamReader = new StreamReader(assetStream))
using (var jsonTextReader = new JsonTextReader(streamReader))
{
var serializer = new JsonSerializer();
if (!jsonTextReader.Read() || jsonTextReader.TokenType != JsonToken.StartArray)
throw new Exception("Bad Json, start of array missing");
while (jsonTextReader.Read())
{
if (jsonTextReader.TokenType == JsonToken.EndArray)
break;
var state = serializer.Deserialize<State>(jsonTextReader);
theRealm.Write(() =>
{
var realmState = theRealm.CreateObject<State>();
realmState.abbreviation = state.abbreviation;
realmState.name = state.name;
});
}
}
Update: One of my extensions methods:
Extension Method Usage:
using (var theRealm = Realm.GetInstance("StackOverflow.realm"))
using (var assetStream = Assets.Open("States.json"))
{
theRealm.JsonArrayToRealm<State>(assetStream);
}
Extension Method:
Note: This uses AutoMapper to copy RealmObject and avoid reflection, also using Newtonsoft.Json.
public static class RealmDoesJson
{
public static void JsonArrayToRealm<T>(this Realm realm, Stream stream) where T : RealmObject
{
Mapper.Initialize(cfg =>
{
cfg.CreateMap<T, T>();
});
using (var streamReader = new StreamReader(stream))
using (var jsonTextReader = new JsonTextReader(streamReader))
{
var serializer = new JsonSerializer();
if (!jsonTextReader.Read() || jsonTextReader.TokenType != JsonToken.StartArray)
throw new Exception("MALFORMED JSON, Start of Array missing");
while (jsonTextReader.Read())
{
if (jsonTextReader.TokenType == JsonToken.EndArray)
break;
var jsonObject = serializer.Deserialize<T>(jsonTextReader);
realm.Write(() => // inside while loop / single object transaction for memory manangement reasons...
{
var realmObject = realm.CreateObject(typeof(T).Name);
Mapper.Map<T, T>(jsonObject, realmObject);
});
}
}
}
}
I want to use the IsoDateTimeConverter from Newtonsoft to format the json version of my DateTime properties.
However, I cant figure out how this is done in Nest 2.x.
Here is my code:
var connectionPool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var settings = new ConnectionSettings(connectionPool, s => new MyJsonNetSerializer(s));
var client = new ElasticClient(settings);
public class MyJsonNetSerializer : JsonNetSerializer
{
public MyJsonNetSerializer(IConnectionSettingsValues settings) : base(settings) { }
protected override void ModifyJsonSerializerSettings(JsonSerializerSettings settings)
{
settings.NullValueHandling = NullValueHandling.Ignore;
}
protected override IList<Func<Type, JsonConverter>> ContractConverters => new List<Func<Type, JsonConverter>>()
{
type => new Newtonsoft.Json.Converters.IsoDateTimeConverter()
};
}
I'm getting this exception:
message: "An error has occurred.",
exceptionMessage: "Unexpected value when converting date. Expected DateTime or DateTimeOffset, got Nest.SearchDescriptor`1[TestProject.DemoProduct].",
exceptionType: "Elasticsearch.Net.UnexpectedElasticsearchClientException"
Any help is appreciated
with the Func<Type, JsonConverter>, you need to check that the type is the right one for the converter that you want to register; if it is, return the converter instance, otherwise return null
public class MyJsonNetSerializer : JsonNetSerializer
{
public MyJsonNetSerializer(IConnectionSettingsValues settings) : base(settings) { }
protected override void ModifyJsonSerializerSettings(JsonSerializerSettings settings)
{
settings.NullValueHandling = NullValueHandling.Ignore;
}
protected override IList<Func<Type, JsonConverter>> ContractConverters => new List<Func<Type, JsonConverter>>()
{
type =>
{
return type == typeof(DateTime) ||
type == typeof(DateTimeOffset) ||
type == typeof(DateTime?) ||
type == typeof(DateTimeOffset?)
? new Newtonsoft.Json.Converters.IsoDateTimeConverter()
: null;
}
};
}
NEST uses the IsoDateTimeConverter for those types by default, so you won't need to register a converter for them unless you would like to change other settings on the converter.