Logging into ElasticSearch with Serilog in ASP.NET Web API - elasticsearch

I am trying to log into Elasticsearch in one of my ASP.NET Web API project using Serilog, but unfortunately, I can't find the logs in Kibana.
public class Logger
{
private readonly ILogger _localLogger;
public Logger()
{
ElasticsearchSinkOptions options = new ElasticsearchSinkOptions(new Uri("xxx"))
{
IndexFormat = "log-myservice-dev",
AutoRegisterTemplate = true,
ModifyConnectionSettings = (c) => c.BasicAuthentication("yyy", "zzz"),
NumberOfShards = 2,
NumberOfReplicas = 0
};
_localLogger = new LoggerConfiguration()
.MinimumLevel.Information()
.WriteTo.File(HttpContext.Current.Server.MapPath("~/logs/log-.txt"), rollingInterval: RollingInterval.Day)
.WriteTo.Elasticsearch(options)
.CreateLogger();
}
public void LogError(string error)
{
_localLogger.Error(error);
}
public void LogInformation(string information)
{
_localLogger.Information(information);
}
}
I can see the logs in the file specified above, just not in Elasticsearch. So, I am wondering is there is any way I can debug why it failed to log into Elasticsearch? I am also open to using other logging framework to log into Elasticsearch.
*The credentials and url for Elasticsearch are valid as I have implemented this in my other AWS Lambda project (.net core).

To see exactly what went wrong, the easiest way is to write into console, and in case of ASP.NET project, it will be Debug.WriteLine. So the code to see what went wrong would be
Serilog.Debugging.SelfLog.Enable(msg => Debug.WriteLine(msg));
ElasticsearchSinkOptions options = new ElasticsearchSinkOptions(new Uri("xxx"))
{
IndexFormat = "log-myservice-dev",
AutoRegisterTemplate = true,
ModifyConnectionSettings = (c) => c.BasicAuthentication("yyy", "zzz"),
NumberOfShards = 2,
NumberOfReplicas = 1,
EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog,
MinimumLogEventLevel = Serilog.Events.LogEventLevel.Information
};
The following error message was retrieved from the output console.
Failed to create the template.
Elasticsearch.Net.ElasticsearchClientException: The request was
aborted: Could not create SSL/TLS secure channel.. Call: Status code
unknown from: HEAD /_template/serilog-events-template --->
System.Net.WebException: The request was aborted: Could not create
SSL/TLS secure channel.
The issue is quite clear cut. Added the following in my logger class constructor helped with the issue.
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
Hope it helps others that encounter issue trying to use Serilog to log into Elasticsearch for .Net Framework.

Related

How to start GraphQL server when running .net5 integration tests?

I believe I am missing/misunderstanding something fundamental about the way .net5 works. In setting up an integration test environment for my GraphQL API, I am missing the step on how to start the GraphQL server from said test environment.
When I run the main project, the server is started properly and I can navigate to localhost in the browser and successfully execute GraphQL queries/mutations. My goal here is to set up some automated integration tests.
I'm using NUnit as my test runner and am using WebApplicationFactory<Startup> to "start the server" as I understand it.
In my test project, I'm under the impression that WebApplicationFactory<Startup> is supposed to basically use the Startup.cs class from my main project in my test project so that I don't have to duplicate all the settings, configurations, and injected services. Please correct me if that assumption is not correct.
I've pasted the code I think is relevant.
ApiWebApplicationFactory<Startup>
public class ApiWebApplicationFactory<TStartup> : WebApplicationFactory<Startup> where TStartup : class
{
public static IConfigurationRoot Configuration { get; set; }
public ApiWebApplicationFactory()
{
var configBuilder = new ConfigurationBuilder()
.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true);
Configuration = configBuilder.Build();
}
protected override void ConfigureClient(HttpClient client)
{
base.ConfigureClient(client);
client.BaseAddress = new Uri("https://localhost");
client.Timeout = new TimeSpan(0, 10, 0);
}
// Based on my assumption this class reuses everything in the Startup.cs class
// I don't actually think this is necessary, but thought it was worth trying
// the test with and without this code.
protected override void ConfigureWebHost(IWebHostBuilder builder)
{
builder.ConfigureServices(services =>
{
services
.AddGraphQLServer()
.AddQueryType<Query>()
.AddMutationType<Mutation>()
.AddType<GraphQLContentItem>()
.AddType<GraphQLFolder>();
});
}
}
OneTimesetUp
[OneTimeSetUp]
public void OneTimeSetUp()
{
_factory = new ApiWebApplicationFactory<Startup>();
_client = _factory.WithWebHostBuilder(builder =>
{
builder.ConfigureServices(services =>
{
services.AddScoped<ICacheRepository, MockCache>();
});
}).CreateClient();
var connString = ApiWebApplicationFactory<Startup>.Configuration.GetConnectionString("DefaultConnection");
var options = new DbContextOptionsBuilder<CmsContext>()
.UseMySql(connString, ServerVersion.AutoDetect(connString))
.Options;
_dbContext = new CmsContext(options);
_dbContext.Database.EnsureCreated();
}
Test
[Test]
public async Task Test()
{
// If I set a breakpoint here, I can't navigate to the URL like I'm expecting to
var graphQLHttpClient =
new GraphQLHttpClient(
new GraphQLHttpClientOptions { EndPoint = new Uri("https://localhost/graphql") },
new NewtonsoftJsonSerializer(),
_client);
var request = new GraphQLRequest
{
Query = #"
query GetCurrentSession() {
getCurrentSession() {
id
name
}
}",
OperationName = "GetCurrentSession"
};
// Error is thrown here with "Bad Request"
var response = await graphQLHttpClient.SendQueryAsync<Session>(request);
// Further code is omitted
}
Please let me know if you see what I am missing. Thanks in advance~

MassTransit endpoint name is ignored in ConsumerDefinition

The EndpointName property in a ConsumerDefinition file seems to be ignored by MassTransit. I know the ConsumerDefinition is being used because the retry logic works. How do I get different commands to go to a different queue? It seems that I can get them all to go through one central queue but I don't think this is best practice for commands.
Here is my app configuration that executes on startup when creating the MassTransit bus.
Bus.Factory.CreateUsingAzureServiceBus(cfg =>
{
cfg.Host(_config.ServiceBusUri, host => {
host.SharedAccessSignature(s =>
{
s.KeyName = _config.KeyName;
s.SharedAccessKey = _config.SharedAccessKey;
s.TokenTimeToLive = TimeSpan.FromDays(1);
s.TokenScope = TokenScope.Namespace;
});
});
cfg.ReceiveEndpoint("publish", ec =>
{
// this is done to register all consumers in the assembly and to use their definition files
ec.ConfigureConsumers(provider);
});
And my handler definition in the consumer (an azure worker service)
public class CreateAccessPointCommandHandlerDef : ConsumerDefinition<CreateAccessPointCommandHandler>
{
public CreateAccessPointCommandHandlerDef()
{
EndpointName = "specific";
ConcurrentMessageLimit = 4;
}
protected override void ConfigureConsumer(
IReceiveEndpointConfigurator endpointConfigurator,
IConsumerConfigurator<CreateAccessPointCommandHandler> consumerConfigurator
)
{
endpointConfigurator.UseMessageRetry(r =>
{
r.Immediate(2);
});
}
}
In my app that is sending the message I have to configure it to send to the "publish" queue, not "specific".
EndpointConvention.Map<CreateAccessPointsCommand>(new Uri($"queue:specific")); // does not work
EndpointConvention.Map<CreateAccessPointsCommand>(new Uri($"queue:publish")); // this does work
Because you are configuring the receive endpoint yourself, and giving it the name publish, that's the receive endpoint.
To configure the endpoints using the definitions, use:
cfg.ConfigureEndpoints(provider);
This will use the definitions that were registered in the container to configure the receive endpoints, using the consumer endpoint name defined.
This is also explained in the documentation.

Adjust ElasticsearchSinkOptions.NumberOfShards in SeriLog not working in .Net core

I have a problem when setting NumberOfShards for ElasticSearch while writing log by SeriLog.
I do config for Serilog like this in .Net Core
.WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri(config.ElasticConnectionUrl))
{
AutoRegisterTemplate = true,
IndexFormat = config.ElasticIndex + "-{0:yyyy.MM.dd}",
NumberOfShards = 2,
NumberOfReplicas = 0
}));
But when I query the setting of the created Index in Kibana, the numberOfShards still 5 (default value). Even for NumberOfReplicas won't affect.
I am using ELK stack to trace logs.
Is anyone khow why?
You can do Serilog configuration in code, or in appSettings.json configuration. If you are doing this:
var loggerConfiguration = new LoggerConfiguration()
.ReadFrom.Configuration(configuration) // <= this reads from config
.WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri(config.ElasticConnectionUrl))
{
AutoRegisterTemplate = true,
IndexFormat = config.ElasticIndex + "-{0:yyyy.MM.dd}",
NumberOfShards = 2,
NumberOfReplicas = 0
})); // this gets partially ignored
And you have the following configuration file:
"Serilog": {
"MinimumLevel": "Debug",
"WriteTo": [
{
"Name": "Elasticsearch",
"Args": {
"nodeUris": "http://localhost:9100"
}
}
]
}
If you have left the index format empty in the JSON file, then you'll probably have an index named like "log-stash-{0:YYYY-MM-DD}", but you'll also have the one you've added in code. Same goes for the URIs. You might have two if you've added one in in the JSON sink WriteTo and in the ElasticsearchSinkOptions in code.
Config as JSON or as code. Choose one (sadly). See this for further information: https://github.com/serilog/serilog-sinks-elasticsearch/issues/180
I was looking to have a way to have a default configuration in code, that could potentially be overwritten using the configuration file, since I'm trying to create a generic opinionated IHostBuilder for our company various APIs, but Serilog is playing nicely. My solution is to move the Sinks config out of the Serilog config section, and define it separately, and load it myself, (in the same format - string Name, Args Dictionary<string, string>), and then create the config manually in code.

App running in Android Emulator fails to perform an HTTP Post to localhost

I'm unable to perform an HTTP Post with an app running in an Android Emulator.
{StatusCode: 400, ReasonPhrase: 'Bad Request', Version: 1.1, Content:
System.Net.Http.HttpConnection+HttpConnectionResponseContent, Headers:
{ Server: Microsoft-HTTPAPI/2.0 Date: Wed, 23 Oct 2019 00:58:01
GMT Connection: close Forwarded: host=XXX.XXX.X.XX:XXXXX;
proto=https Content-Type: text/html; charset=us-ascii
Content-Length: 374 }}
Setup:
I'm using an IP address generated by Conveyor by Keyoti
I installed a security certificate on the emulator required by Conveyor by Keyoti
I swapped out Microsoft.AspNetCore.Mvc.HttpPost attribute with System.Web.Http.HttpPost
Emulator:
Successful: HTTP Get
Failed: HTTP Post
Integration Test:
Successful: HTTP Post (using same endpoint)
Code:
I wrote an automated test that calls the same HTTP Post implementation.
Because I executed the same code successfully on my laptop via an automated test, I don't think the actual code is the issue:
open Microsoft.AspNetCore.Mvc
open Newtonsoft.Json
[<ApiController>]
[<Route("api/[controller]")>]
type RegisterController () =
inherit ControllerBase()
[<System.Web.Http.HttpPost>]
member x.Post([<FromBody>] json:string) =
...
Summary:
In conclusion, I have isolated the environment to the Android Emulator and not my laptop. Hence, the emulator can successfully trigger an HTTP Get. However, it fails to perform a HTTP Post even though my laptop device can do both.
UPDATE:
I applied guidance from this Xamarin Android ASP.Net Core WebAPI document.
Specifically, I installed another security certificate on the Android emulator.
I was then able to observe an HTTP Get on the Android Emulator.
However, I continue to get an error for HTTP Post.
OperationCanceledException
Physical Device:
If I run the app from a physical android device I observe the following:
{StatusCode: 500, ReasonPhrase: 'Internal Server Error', Version: 1.1, Content: System.Net.Http.HttpConnection+HttpConnectionResponseContent, Headers:
{
Date: Wed, 23 Oct 2019 13:33:20 GMT
Server: Kestrel
Transfer-Encoding: chunked
Forwarded: host=xxx.xxx.x.xx:xxxxx; proto=https
Content-Type: text/plain
}}
New Update:
I disabled debugging on just my code on the server implementation and discovered the following exception:
Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: 'Bad chunk size data.'
Any suggestions?
this might not be a direct answer to your question, but i would like to suggest
localtunnel. a very easy way to temporarily expose your local api so that you can test it either on emulator or even physical device. Have used this alot my self, as it is very convenient to just type a single line in terminal to start it.
The following reference solved my issue.
Infrastructure:
type GlobalHttpClient private () =
static let mutable (httpClient:System.Net.Http.HttpClient) = null
static member val Instance = httpClient with get,set
Xamarin.Android project:
using Android.Http;
using Android.Net;
using Javax.Net.Ssl;
using System.Net.Http;
using Xamarin.Android.Net;
using Xamarin.Forms;
using WebGatewaySupport;
[assembly: Dependency(typeof(HTTPClientHandlerCreationService_Android))]
namespace Android.Http
{
public class HTTPClientHandlerCreationService_Android : IHTTPClientHandlerCreationService
{
public HttpClientHandler GetInsecureHandler()
{
return new IgnoreSSLClientHandler();
}
}
internal class IgnoreSSLClientHandler : AndroidClientHandler
{
protected override SSLSocketFactory ConfigureCustomSSLSocketFactory(HttpsURLConnection connection)
{
return SSLCertificateSocketFactory.GetInsecure(1000, null);
}
protected override IHostnameVerifier GetSSLHostnameVerifier(HttpsURLConnection connection)
{
return new IgnoreSSLHostnameVerifier();
}
}
internal class IgnoreSSLHostnameVerifier : Java.Lang.Object, IHostnameVerifier
{
public bool Verify(string hostname, ISSLSession session)
{
return true;
}
}
}
Xamarin.Forms App:
switch (Device.RuntimePlatform)
{
case Device.Android:
GlobalHttpClient.Instance = new HttpClient(DependencyService.Get<IHTTPClientHandlerCreationService>().GetInsecureHandler());
break;
default:
ServicePointManager.ServerCertificateValidationCallback += (sender, cert, chain, sslPolicyErrors) => true;
GlobalHttpClient.Instance = new HttpClient(new HttpClientHandler());
break;
}
Client Gateway:
let postTo (baseAddress:string) (resource:string) (payload:Object) =
GlobalHttpClient.Instance.BaseAddress <- Uri(baseAddress)
let encoded = Uri.EscapeUriString(resource)
let result = GlobalHttpClient.Instance.PostAsJsonAsync(encoded, payload) |> toResult
result
Looks like you have a .NET Core Api. .NET Core does not have System.Web in Asp.NET. The HttpPost attribute and HttpGet attributes should come from Microsoft.AspNetCore.Mvc namespace which you have open.
Also since you are using the ApiController attribute model binding will just work as long as you bind to a model and not just a json string.
Create a model that you want the json to bind to and use that type for your parameter on Post and remove the FromBody attribute. Also if you do that you probably don't need newtonsoft.json.
open Microsoft.AspNetCore.Mvc
[<ApiController>]
[<Route("api/[controller]")>]
type RegisterController () =
inherit ControllerBase()
[<HttpPost>]
member x.Post(thing:TypeOfThing) =

Web API add openid scope to auth url for swagger/swashbuckle UI

We have a asp.net web api application which uses swagger/swashbuckle for it's api documentation. The api is secured by azure AD using oauth/openid-connect. The configuration for swagger is done in code:
var oauthParams = new Dictionary<string, string>
{
{ "resource", "https://blahblahblah/someId" }
};
GlobalConfiguration.Configuration
.EnableSwagger(c =>
{
c.SingleApiVersion(Version, Name);
c.UseFullTypeNameInSchemaIds();
c.OAuth2("oauth2")
.Description("OAuth2 Implicit Grant")
.Flow("implicit")
.AuthorizationUrl(
"https://login.microsoftonline.com/te/ourtenant/ourcustompolicy/oauth2/authorize")
.TokenUrl(
"https://login.microsoftonline.com/te/ourtenant/ourcustompolicy/oauth2/token");
c.OperationFilter<AssignOAuth2SecurityRequirements>();
})
.EnableSwaggerUi(c =>
{
c.EnableOAuth2Support(_applicationId, null, "http://localhost:49919/swagger/ui/o2c-html", "Swagger", " ", oauthParams);
c.BooleanValues(new[] { "0", "1" });
c.DisableValidator();
c.DocExpansion(DocExpansion.List);
});
When swashbuckle constructs the auth url for login, it automatically adds:
&scope=
However I need this to be:
&scope=openid
I have tried adding this:
var oauthParams = new Dictionary<string, string>
{
{ "resource", "https://blahblahblah/someId" },
{ "scope", "openid" }
};
But this then adds:
&scope=&someotherparam=someothervalue&scope=openid
Any ideas how to add
&scope=openid
To the auth url that swashbuckle constructs?
Many thanks
So, found out what the issue was, the offending code can be found here:
https://github.com/swagger-api/swagger-ui/blob/2.x/dist/lib/swagger-oauth.js
These js files are from a git submodule that references the old version of the UI.
I can see on lines 154-158 we have this code:
url += '&redirect_uri=' + encodeURIComponent(redirectUrl);
url += '&realm=' + encodeURIComponent(realm);
url += '&client_id=' + encodeURIComponent(clientId);
url += '&scope=' + encodeURIComponent(scopes.join(scopeSeparator));
url += '&state=' + encodeURIComponent(state);
It basically adds scopes regardless of whether there are scopes or not. This means you cannot add scopes in the additionalQueryParams dictionary that gets sent into EnableOAuth2Support as you will get a url that contains 2 scope query params i.e.
&scope=&otherparam=otherparamvalue&scope=openid
A simple length check around the scopes would fix it.
I ended up removing swashbuckle from the web api project and added a different nuget package called swagger-net, found here:
https://www.nuget.org/packages/Swagger-Net/
This is actively maintained and it resolved the issue and uses a newer version of the swagger ui. The configuration remained exactly the same, the only thing you need to change is your reply url which is now:
http://your-url/swagger/ui/oauth2-redirect-html

Resources