tl;dr:
my web api service is running slow, added logging
logging is giving me errors about not being able to find timeouts
didn't see the ravendb database for the webapi project on the server, but dev machines have it
raven is running and running on 8080
tried reinstalling nservicebus via platform installer(downloaded from nservicebus yesterday)
still have the same issue
i think i either messed up the deployment or I need to do a clean install somehow
What do I need to do in order for the webapi project to read the timeouts?
I'm trying to get our application working on a production machine in preparation for an alpha.
Our application has multiple web api sites and many service hosts.
The errors I'm seeing are currently in the web api host on production only:
2014-08-06 01:55:24,789 [52] DEBUG NServiceBus.Timeout.Hosting.Windows.TimeoutPersisterReceiver [(null)] - Polling for timeouts at 08/06/2014 01:55:24.
2014-08-06 01:55:24,793 [46] WARN NServiceBus.Timeout.Hosting.Windows.TimeoutPersisterReceiver [(null)] - Failed to fetch timeouts from the timeout storage
This gets repeated awhile. Eventually (about 2 minutes) the endpoint gives up and just dies
2014-08-06 01:57:03,645 [17] WARN NServiceBus.CircuitBreakers.RepeatedFailuresOverTimeCircuitBreaker [(null)] - The circuit breaker for TimeoutStorageConnectivity will now be triggered
2014-08-06 01:57:03,645 [17] FATAL NServiceBus [(null)] - Repeated failures when fetching timeouts from storage, endpoint will be terminated.
System.InvalidOperationException: There is no index named: dynamic/TimeoutData
I'm getting these logs by enabling log4net file appender for nservicebus to figure out whats going on with the web api services.
The host services work fine on our dev machines, but for some reason they're really slow on the production machines. We don't see the errors on our dev machines. We have the licenses installed on the production machines so its not like we're being throttled by an expired license.
The weird thing is that on our dev machines we see the database for our web api projects, but on the production machines, only the installed service hosts have database entries.
We've been in dev for this particular application for about 18 months. It could be that when we installed nservicebus, the timeouts were not set to be installed. However, I have tried running the platform installer (downloaded yesterday) and the timeouts are still not present. I'm assuming that when I install the service host, the timeouts are stupposed to be installed then, but they're not showing up for some reason.
At this point I'm contemplating uninstaling nservicebus and ravendb, but i'm not sure how to go about getting a clean reinstall.
Here's some relevant lines from my global.asax
IContainer container = IoC.Initialize();
DependencyResolver.SetResolver(new StructureMapDependencyResolver(container));
GlobalConfiguration.Configuration.DependencyResolver = new StructureMapDependencyResolver(container);
Configure.Transactions.Enable();
Configure.Serialization.Json();
Configure.With()
.DefiningEventsAs(t => t.Namespace != null && t.Namespace.Contains(".Events"))
.DefiningCommandsAs(t => t.Namespace != null && t.Namespace.Contains(".Commands"))
.StructureMapBuilder(container)
.Log4Net()
.UseTransport<Msmq>()
.UnicastBus()
.CreateBus()
.Start(() => Configure.Instance.ForInstallationOn<NServiceBus.Installation.Environments.Windows>().Install());
To fix this, perform the following steps
Open the the RavenDB control panel, usually located at http://RAVEN_DB_SERVER:8080
Select the database of your endpoint
Go to Indexes tab
Click at Delete All Indexes
The indexes will be recreated as soon as the service endpoint is started.
It looks like Raven is not installed in your production environment.
For install instructions please take a look here
HTH
Related
I published a simple .NET 6.0 web app with user authentication on my IIS 10.0 web server hosted on Azure.
The application read and write data from a SQL Server 2017 database hosted on another machine on the same virtual network of the web server.
Everything works fine but after a while, even though nobody is using the app, it stop working and send Http Error 503 Service Unavailable and I have to restart it every time from the application pool.
I enabled tracing on the event log, but I don't receive any message explaining this.
These are the application settings:
IIS application settings
I tried to follow these advices but none of them worked:
IIS HTTP Error 503. The service is unavailable
503 “Service Unavailable” error for all apps after installing .NET Core 2.2 Runtime & Hosting Bundle for Windows
.Net core Service Unavailable after Publish to IIS
On the same web server I have other applications but this problem appears only on this one. None of the other apps is using .NET 6, most use .NET Framework 4.7/4.8 and another couple .NET Core 3.1.
I also tried to publish the application on Azure WebApp service and it has the same problem.
1. If you're deploying on IIS, we can install DebugDiag Tools and then use the DebugDiag collection to crawl the dump logs.
2. If you are using the azure app service, it is recommended to get dump in the azure portal.
My Suggestion
Enable Failed Request Tracing, and find which request will cause the Http 503.
Maybe this 503 is not a problem with the code, and it is possible to find some useful information in the log.
Check the EventViewer, try to find the useful info about the application.
Download the DebugDiag Tools, use debugdiag collection to collect the dump file when the issue occured, and use the debugdiag analysis to analyze the dump file and check the stacktrace.
My suggestion can't tell you the root cause directly, but it will help you to find it. You are also welcome to update the useful stacktrace information in the future, and there will be more people to help you.
I'm currently having the issue but using MySQL on a remote server.
Azure WebApp, Azure VM (IIS 10) and AWS EC2.
Http Error 503 Service Unavailable when the database connection timesout or fails to connect and the AppPool failed to restart.
No crashes when debugging in VS via console app.
I changed the hosting model to out-of-process and this solved the issue.
<aspNetCore processPath="%LAUNCHER_PATH%" arguments="%LAUNCHER_ARGS%" stdoutLogEnabled="true" stdoutLogFile=".\logs\stdout" hostingModel="outofprocess">
I am trying to move my backend API app (node.js express server) from Heroku to AWS Elastic Beanstalk. But I did not realize the amount of features that Heroku was providing automatically and which I now have to set up manually in AWS.
So here is the list of features which I discovered were missing in AWS and the solutions I have implemented.
Could you please let me know if I am missing something in order to run smoothly my APIs in AWS and get the equivalent of what I had in Heroku?
auto-restart server when crashed : I am using PM2 to automatically restart my server in case of critical error
SSL certificate : I am using AWS ACM certificate,
logging : have inserted the datadog agent in order to receive logs in datadog
logging response time : I have added the "morgan-body" package to get each requests' duration and response code (had to manually filter the AWS healthchecks and search engine bots, because AWS gave me an IP adress which was visited constatntly by Baidu bots)
server timeout : I have implemented a 1200000ms timeout on the whole app (any better option ?)
auto deploy from Github : I have implemented a github automation to deploy code automatically (better options?)
Am I missing something? This app is already live so I do not want to put my customers at risk when I will move from Heroku to AWS...
Thanks for your help!
I believe you are covered:
Heroku Dynos restart after crashing or raising an error (Heroku Restarting Policy)
SSL certificates are provided for free
logging: Heroku supports various plugins, including Datadog
response time (in millisec) is logged automatically
HTTP timeout is 30 sec (it cannot be changed)
deploy from Github is possible (connecting the accounts), Docker deployment is also supported. Better options? Using Github Actions to deploy a new version after code push or tagging.
If you are migrating a production environment I strongly suggest first to setup a Heroku (Free) Dyno to test and verify all your needs are satisfied.
Web API has GlobalConfiguration.Configuration.IncludeErrorDetailPolicy
= IncludeErrorDetailPolicy.Always; to turn on remote errors. (Allowing them to see them in a browser even if you are not browsing on the local machine.
But, near as I can tell, Service Fabric, running Web Api, does not support GlobalConfiguration.
Is there a way to configure things so I don't have to log into one of my Service Fabric server machines each time I want to see what a services error message is?
I recommend you don't show error details to everyone.
It's a security risk.
Consider moving your error logs out of your cluster. For instance, by using OMS, ELK or Application Insights.
I have three mvc 3 web applications in a single web role on Azure Cloud Services. All of a sudden this morning both the single instance in production and in staging give "HTTP Error 503. The service is unavailable." when navigated to in a browser.
I rebooted the production instance but nothing changed. Then I deployed a remote desktop enabled version to the staging instance and logged in. However, strangely when I navigate to the web applications in a browser inside the remoted staging instance everything works.
Looking at IIS server logs it seems there are some issues starting the roles (warning level about the appPoolId being incorrect) but obviously the roles are starting as they are accessible from inside the staging server.
The World Wide Web Publishing service is also running and even after restarting this service the web application is not accessible externally.
Does anyone have an explaination for why the sites are accessible locally but not remotely that would help me debug this issue?
I found out where the issue came from, the bindings that were configured in IIS7 did not include bindings for the actual [abc].cloudapp.net host headers.
I assume that for some reason since some time last night or this weekend the requests seem to come with those headers instead of the original headers for the website. This is really strange but adding these bindings fixed both the staging and production instances and they were available again after this change.
hej,
since march we got some troubles with our deployments on our windows azure server.
It seems that the deployments doesn't work and because sometimes it works sometimes not i don't get it.
Some kind of error i saw on my windows azure dashboard -> deployments:
"Response status code does not indicate success: 500 (Internal Server Error)."
or
"Host key verification failed. fatal: The remote end hung up unexpectedly"
i tried to restart the server or make new commits to github or simply redeploy or disconnect and reconnect to github. At the moment nothing works. There is a development server and a live server. Both contains the exact similiar version of the website (written in node.js). i've checked it a lot of times. Deployment with the development version works fine. live dont.
Do you have any suggestions? I have no idea where to start :-(
If it's a web role you can check the iisnode/ folder, should contain the log files. They can also be accessed with the emulator running, try
http://127.0.0.1:81/iisnode/