Below is the code for serilog elastic search.It's working perfectly when elastic service is running but fails when the server is not available
.WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200/")) // humio elastic bulk endpoint
{
AutoRegisterTemplate = true,
OverwriteTemplate = true,
DetectElasticsearchVersion = true,
AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv7,
NumberOfReplicas = 1,
NumberOfShards = 2,
RegisterTemplateFailure = RegisterTemplateRecovery.FailSink,
FailureCallback = e => Console.WriteLine("Unable to submit event " + e.MessageTemplate),
EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog |
EmitEventFailureHandling.WriteToFailureSink |
EmitEventFailureHandling.RaiseCallback,
//FailureSink = new FileSink("./fail-{Date}.txt", new JsonFormatter(), null, null)
//ModifyConnectionSettings = x => x.BasicAuthentication(username: "", password: "88WS28JEuB0G2WC3cYufdgTINAxOGQvliBrqH5Vqutjb") // password is ingest token from humio
})
exception is occuring when elastic server is missing
Elasticsearch.Net.ElasticsearchClientException: 'No connection could be made because the target machine actively refused it.. Call: Status code unknown from: GET /_cat/nodes?h=v'
HttpRequestException: No connection could be made because the target machine actively refused it.
SocketException: No connection could be made because the target machine actively refused it.
This exception was originally thrown at this call stack:
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()
System.Net.Http.ConnectHelper.ConnectAsync(string, int, System.Threading.CancellationToken)
Related
I am trying to log into Elasticsearch in one of my ASP.NET Web API project using Serilog, but unfortunately, I can't find the logs in Kibana.
public class Logger
{
private readonly ILogger _localLogger;
public Logger()
{
ElasticsearchSinkOptions options = new ElasticsearchSinkOptions(new Uri("xxx"))
{
IndexFormat = "log-myservice-dev",
AutoRegisterTemplate = true,
ModifyConnectionSettings = (c) => c.BasicAuthentication("yyy", "zzz"),
NumberOfShards = 2,
NumberOfReplicas = 0
};
_localLogger = new LoggerConfiguration()
.MinimumLevel.Information()
.WriteTo.File(HttpContext.Current.Server.MapPath("~/logs/log-.txt"), rollingInterval: RollingInterval.Day)
.WriteTo.Elasticsearch(options)
.CreateLogger();
}
public void LogError(string error)
{
_localLogger.Error(error);
}
public void LogInformation(string information)
{
_localLogger.Information(information);
}
}
I can see the logs in the file specified above, just not in Elasticsearch. So, I am wondering is there is any way I can debug why it failed to log into Elasticsearch? I am also open to using other logging framework to log into Elasticsearch.
*The credentials and url for Elasticsearch are valid as I have implemented this in my other AWS Lambda project (.net core).
To see exactly what went wrong, the easiest way is to write into console, and in case of ASP.NET project, it will be Debug.WriteLine. So the code to see what went wrong would be
Serilog.Debugging.SelfLog.Enable(msg => Debug.WriteLine(msg));
ElasticsearchSinkOptions options = new ElasticsearchSinkOptions(new Uri("xxx"))
{
IndexFormat = "log-myservice-dev",
AutoRegisterTemplate = true,
ModifyConnectionSettings = (c) => c.BasicAuthentication("yyy", "zzz"),
NumberOfShards = 2,
NumberOfReplicas = 1,
EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog,
MinimumLogEventLevel = Serilog.Events.LogEventLevel.Information
};
The following error message was retrieved from the output console.
Failed to create the template.
Elasticsearch.Net.ElasticsearchClientException: The request was
aborted: Could not create SSL/TLS secure channel.. Call: Status code
unknown from: HEAD /_template/serilog-events-template --->
System.Net.WebException: The request was aborted: Could not create
SSL/TLS secure channel.
The issue is quite clear cut. Added the following in my logger class constructor helped with the issue.
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
Hope it helps others that encounter issue trying to use Serilog to log into Elasticsearch for .Net Framework.
I was able to successfully integrate the elastic search sink in my .net app, now I am trying to setup the FailureCallback option but every time there is an error the exception in LogEvent is null, I can see the actual error in the console but I want to be able to capture this exception in my failure callback function, here is my current configuration:
myLogger = new LoggerConfiguration()
.WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri(elasticSearchUrl))
{
AutoRegisterTemplate = true,
AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv6,
IndexDecider = (#event, dateTimeOffset) =>
{
//some logic here
return $"custom-index";
},
EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog | EmitEventFailureHandling.RaiseCallback,
FailureCallback = HandleElasticError
})
.Enrich.WithProperty("Environment", Config.Environment)
.Destructure.ByTransforming<ExpandoObject>(JsonConvert.SerializeObject)
.CreateLogger();
Here is my HandleElasticError function:
private void HandleElasticError(LogEvent e)
{
FileLogger.Error(e.Exception, e.MessageTemplate.Text, e.Properties);
}
When I attach the debugger and inspect the LogEvent, exception is null, however in the output window in Visual Studio I can see the actual error:
2020-11-30T20:55:07.7820674Z Caught exception while preforming bulk operation to Elasticsearch: Elasticsearch.Net.ElasticsearchClientException: The underlying connection was closed: An unexpected error occurred on a send.. Call: Status code unknown from: POST /_bulk ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: The handshake failed due to an unexpected packet format.
at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)
at System.Net.PooledStream.EndWrite(IAsyncResult asyncResult)
at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
I am not concerned about the error, I know what the issue is, I just want to be able to capture this exception in my failure callback function.
Any ideas?
LogEvent.Exception is the Exception shipped with your call to (for example) logger.Error(myException) - not the exception thrown by failure to write to Elastic Search.
The exception you're trying to catch doesn't look to be exposed by the sink as it just passes the LogEvent to FailureCallback and not exception that caused the failure.
Here's what it could look like were the exception exposed.
you can use:
StreamWriter writer = File.CreateText(...);
Serilog.Debugging.SelfLog.Enable(writer);
I am trying to connect to aws DocumentDB with async mongoClient.
I created a DocumentDB cluster in aws and success connect via ssh command line.
I went over here and created MongoClient and success connected and insert events.
But when I tried create com.mongodb.async.client.MongoClient, connection failed with folowing error:
No server chosen by WritableServerSelector from cluster description
ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE,
serverDescriptions=[ServerDescription{address=aws-cluster:27017,
type=UNKNOWN, state=CONNECTING,
exception={com.mongodb.MongoSocketReadTimeoutException: Timeout while
receiving message}, caused by
{io.netty.handler.timeout.ReadTimeoutException}}]}. Waiting for 30000
ms before timing out.
ClusterSettings clusterSettings = ClusterSettings.builder()
.applyConnectionString(new ConnectionString(connectionString)).build();
List<MongoCredential> credentials = new ArrayList<>();
credentials.add(
MongoCredential.createCredential(
mongoUserName,
mongoDBName,
mongoPassword));
MongoClientSettings settings = MongoClientSettings.builder()
.credentialList(credentials)
.clusterSettings(clusterSettings)
.streamFactoryFactory(new NettyStreamFactoryFactory())
.writeConcern(WriteConcern.ACKNOWLEDGED)
.build();
com.mongodb.async.client.MongoClient mongoClient = MongoClients.create(settings);
MongoDatabase testDB = mongoClient.getDatabase("myDB");
MongoCollection<Document> collection = testDB.getCollection("test");
Document doc = new Document("name", "MongoDB").append("type", "database");
//**trying insert document => here I got an error**
collection.insertOne(doc, new SingleResultCallback<Void>() {
#Override
public void onResult(final Void result, final Throwable t) {
System.out.println("Inserted!");
}
});
Do you have any ideas, why does it happen?
I solved it by using uri:
String uri = "mongodb://<username>:<Password>#<hostname>:27017/?ssl=true&ssl_ca_certs=cert";
MongoClientSettings settings = MongoClientSettings.builder()
.streamFactoryFactory(new NettyStreamFactoryFactory())
.applyConnectionString(new ConnectionString(uri))
.build();
com.mongodb.async.client.MongoClient mongoClient = MongoClients.create(settings);
I encountered a similar error , for me it was related to the TLS configs.
I disabled the TLS in documentDB https://docs.aws.amazon.com/documentdb/latest/developerguide/security.encryption.ssl.html
In my case I had to restart the cluster after disabling the TLS. (TLS was not needed for the use case). After the restart the connection was established successfully.
I am connecting to Elasticsearch cluster using Nest client and I'm getting this issue while adding Index records.
An i missing something?
Nest v 7.2.1
Elastic version 7.2.1
Exception
Failed to create DB Index for Alarm Event.Invalid NEST response built from a unsuccessful (401) low level call on HEAD: /
Audit trail of this API call
[1] BadResponse: Node: http://192.168.0.4:9200/ Exception: PipelineException Took: 00:00:00.0588591
var addresses = new[] {
new Uri("http://192.168.0.4:9200/"),
new Uri("http://192.168.0.5:9200/"),
new Uri("http://192.168.0.6:9200/"),
new Uri("http://192.168.0.7:9200/"),
};
connectionPool = new StaticConnectionPool(addresses);
connectionSettings = new ConnectionSettings(connectionPool).BasicAuthentication("testuser", "Something_123").DisablePing();
ElasticSearchClient = new ElasticClient(connectionSettings);
var indexes = GetAsxIndexes();
var response = ElasticSearchClient.Ping().DebugInformation;
I am using Alamofire 4.5, and XCode 9.0.1. I cannot connect to my local Nginx configured with a self-signed cert.
As per this answer, I am using the following code:
private static var Manager : Alamofire.SessionManager = {
// Create the server trust policies
let serverTrustPolicies: [String: ServerTrustPolicy] = [
"localhost:8443": .disableEvaluation
]
// Create custom manager
let configuration = URLSessionConfiguration.default
configuration.httpAdditionalHeaders = Alamofire.SessionManager.defaultHTTPHeaders
let man = Alamofire.SessionManager(
configuration: URLSessionConfiguration.default,
serverTrustPolicyManager: ServerTrustPolicyManager(policies: serverTrustPolicies)
)
return man
}()
And making the call like this:
DataService.Manager.request("https://localhost:8443/foo", parameters: parameters).validate().responseJSON
I tried various configurations in my project, but to no avail:
Here is the error I am seeing:
2017-10-27 08:59:17.118751-0600 hptest[53607:8423753] TIC SSL Trust
Error [1:0x60400016e580]: 3:0
2017-10-27 08:59:17.119020-0600 hptest[53607:8423753]
NSURLSession/NSURLConnection HTTP load failed (kCFStreamErrorDomainSSL,
-9813)
2017-10-27 08:59:17.119111-0600 hptest[53607:8423753] Task <A8A3F1A4-
D6A5-4AD6-9C3F-23697D8B63AD>.<1> HTTP load failed (error code: -1202
[3:-9813])
2017-10-27 08:59:17.119281-0600 hptest[53607:8423742] Task <A8A3F1A4-D6A5-4AD6-9C3F-23697D8B63AD>.<1> finished with error - code: -1202
Error Domain=NSURLErrorDomain Code=-1202 "The certificate for this server is invalid. You might be connecting to a server that is pretending to be “localhost” which could put your confidential information at risk." UserInfo={NSURLErrorFailingURLPeerTrustErrorKey=<SecTrustRef: 0x6000001149a0>, NSLocalizedRecoverySuggestion=Would you like to connect to the server anyway?, _kCFStreamErrorDomainKey=3, _kCFStreamErrorCodeKey=-9813, NSErrorPeerCertificateChainKey=(
"<cert(0x7fc41b870000) s: localhost i: localhost>"
), NSUnderlyingError=0x60800005f980 {Error Domain=kCFErrorDomainCFNetwork Code=-1202 "(null)" UserInfo={_kCFStreamPropertySSLClientCertificateState=0, kCFStreamPropertySSLPeerTrust=<SecTrustRef: 0x6000001149a0>, _kCFNetworkCFStreamSSLErrorOriginalValue=-9813, _kCFStreamErrorDomainKey=3, _kCFStreamErrorCodeKey=-9813, kCFStreamPropertySSLPeerCertificates=(
"<cert(0x7fc41b870000) s: localhost i: localhost>"
)}}, NSLocalizedDescription=The certificate for this server is invalid. You might be connecting to a server that is pretending to be “localhost” which could put your confidential information at risk., NSErrorFailingURLKey=https://localhost:8443/foo, NSErrorFailingURLStringKey=https://localhost:8443/foo, NSErrorClientCertificateStateKey=0}
I managed to solve it with the following code for the session manager -- should have RTFM instead of SO first:
private static var Manager : Alamofire.SessionManager = {
// Create the server trust policies
let serverTrustPolicies: [String: ServerTrustPolicy] = [
"localhost:8443": .disableEvaluation
]
// Create custom manager
let configuration = URLSessionConfiguration.default
configuration.httpAdditionalHeaders = Alamofire.SessionManager.defaultHTTPHeaders
let man = Alamofire.SessionManager(
configuration: URLSessionConfiguration.default,
serverTrustPolicyManager: ServerTrustPolicyManager(policies: serverTrustPolicies)
)
man.delegate.sessionDidReceiveChallenge = { session, challenge in
var disposition: URLSession.AuthChallengeDisposition = .performDefaultHandling
var credential: URLCredential?
if challenge.protectionSpace.authenticationMethod == NSURLAuthenticationMethodServerTrust {
disposition = URLSession.AuthChallengeDisposition.useCredential
credential = URLCredential(trust: challenge.protectionSpace.serverTrust!)
} else {
if challenge.previousFailureCount > 0 {
disposition = .cancelAuthenticationChallenge
} else {
credential = man.session.configuration.urlCredentialStorage?.defaultCredential(for: challenge.protectionSpace)
if credential != nil {
disposition = .useCredential
}
}
}
return (disposition, credential)
}
return man
}()