Randomly raising Internal Server Error 500. How to debug? - ajax

Recently I moved my zend app from simple hosting to azure, configured to process requests via nginx->haproxy->apache chain. nginx, haproxy and apache are docker containers. While loading the application makes 20-30 AJAX requests to different controllers. Here's the common init method I use in controllers:
public function init()
{
$this->sa = System_Auth::getInstance();
$this->data = $this->getRequest()->getParams();
$this->current = $this->sa->getCurrent();
$this->data['customer_id'] = $this->current->customer ? $this->current->customer->id : $this->current->id;
$this->_helper->contextSwitch()
->addActionContext('test', 'json')
->setAutoJsonSerialization(true)
->initContext();
$this->_helper->layout()->disableLayout();
$this->_helper->viewRenderer->setNoRender(true);
}
So there is nothing interesting in it. But every time I load application about 3 or 4 AJAX requests return 500 error. Each time different requests failed with 500 error. Apache and nginx logs are empty. And I can't dump data passed to controller because failed controllers aren't even called.
Anyone has idea?

OK quys, finally it was fixed.
Where happened:
HaProxy and round-robined mariaDB Cluster Master-Master-Master
What happened:
Some queries randomly failed each time
Why:
sql_mode=only_full_group_by was found on one of three mysql servers
Earlier I cleared sql_mode variable on all servers, but it seems later it was restored when one of servers was restarted. And variables aren't replicable, you should manually change them on each server*
*It may be possible but I don't know right now

Related

Oracle ORDS: Get request returns old data, then after period of time the changed data

I am having a problem with the Oracle Rest Data Services (short ORDS) and I can't find a solution.
The Problem is as follows:
We are using ORDS via a TomCat Webserver and I have 2 Endpoints defined, one to Update a dataset and one to get all datasets from this table.
If I update the value via my Endpoint the change is written in the Table, but if I try to get the table with this change ORDS only response with the old not changed table. After a certain period of Time while constantly trying to get the change it repondes with the expected values. (happens after max 1 minute, can be earlier).
Because of this behaviour I accused some type of caching, but I cant find no configuration in the oracle database or on the TomCat.
Another Point for this theory was that I logged what happens in my GET procedure and found that only the one request with the correct values gets logged, like the others didnt even happen ..
The Request giving me the old value are coming back in the 4-8 ms range while the request with the correct data is in the 100-200 ms.
Ty for your help :)
I tried logging what happens, but I got that only the request with the fresh values was logged.
I tried to restart the TomCat Webserver to make sure that the cache is cleared, but this didnt fix the Problem
I searched for a configuration in ORDS or oracle where a cache would be defined, but it was never set.
I tried to set the value via a SQL update and not an endpoint, but even here I get the change only delayed
Do you have a full overview of the communication path? Maybe there is a proxy between?
When the TomCat has no caching configuration and you restartet the webserver during your tests and still have the same issue, then there is maybe more...
Kind regards
M-Achilles

File does not exist at path /storage/debugbar/

We had enabled successfully debugbar in laravel, but when we inspect in the browser there showed 500 on get call. Here is exact the error
"exception: "Illuminate\Contracts\Filesystem\FileNotFoundException"
file: "/var/www/public_html/shared/vendor/laravel/framework/src/Illuminate/Filesystem/Filesystem.php"
line: 41
message: "File does not exist at path /var/www/public_html/releases/20200730095734/storage/debugbar/Xc528860d1984ecd3c22a9cac793a19d0.json".
There is AWS ALB running behind the webserver server in that case it file doesn't exist one server and the other one have. Can someone reply on it, what can be the reason?
You seem to have found the problem in both you question and in your comment.
When you have DebugBar enabled, it creates a JSON file on the server in app/storage/debugbar/{HASH}.json. Since this file is created on the instance, it is only available to the instance on which it was created.
Amazon's Application Load Balancer (ALB) will route requests to any of your instances based on the ALB's own algorithm. This routing can result in the first request being sent to instance A on which the JSON file is generated and saved. Then a subsequent request may be routed to another instance, instance B, which does not contain the file.
The DebugBar requests a specific file based on the hash, but that file does not exist on instance B. This causes an Illuminate\Filesystem\FileNotFoundException to be triggered which bubbles up and results in an HTTP 500 response.
A solution, maybe the best solution for environments running DebugBar, would be to enable "sticky sessions". Using sticky sessions you can ensure that, once you have created a session with one instance, you then stay connected to that instance.
To enabled sticky sessions, you should read the AWS documentation linked above, but as a simple example you can do:
aws elbv2 modify-target-group-attributes --target-group-arn ARN --attributes Key=stickiness.enabled,Value=true Key=stickiness.lb_cookie.duration_seconds,Value=86400
This would create a sticky session for 86400 seconds (1 day).

Springboot #Async in multiserver environment

We have a requirement to write to the Database in #async mode. We have two servers and one DB. Its working fine on local environment but on the server, the request is going to both the servers and for single request, its inserting same records in DB two times.
Please let me know how to fix this issue.
two servers got the one request? you do not use nginx for loadbalance?

CodeDeploy allowTraffic Fails but my code is still deployed on instances

I am using codeDeploy and when I run it gets stuck on in progress mode. By further researching the problem I found that it fails because of the AllowTraffic script. it just says script failed. I have looked into the logs but there are no errors. Also aws documentation suggested that it may be health check problem but both my instances are healthy in my target group.
Weird thing is that the code gets deployed despite failed status.
Can someone help?
Thanks a bunch
did you enable the elastic load balancer? If so then check your healthcheck settings on your ELB. If it fails on the AllowTraffic it means that it's not getting a successful return from your load balancer. For example, you are doing redirects on your ELB. The status code will be 301. You should add this on your ELB healthcheck.
If health check is fine you can also try to change Application Deployment Settings:
to CodeDeployDefault.OneAtATime
for me CodeDeployDefault.AllAtOnce was failing with same error.
If AllowTraffic stage isn't passing successfully, then usually there are 2 issues-
Either the target group in your application load balancer is unhealthy.
The Target group setting in your deployment configuration is not pointing to the target group mentioned in your load balancer under the rules.

VS2010 Load Test Failing - Cannot Open Database - NOT The Load Test Results database

Hi I have been battling with this issue all day. I have a vs2010 load test which consists of three scenarios which are composed of three different web performance tests.
Each of the web performance tests select urls from a database which is configured correctly and runs locally. However when the load test is run remotely it fails with the error:
Could not run load test 'Load Test' on agent 'AGENTSERVER'. Could not open the database 'URLSDB' requested by the login. login failed for useraccount
In an attempt to get this working the agents and controller are set to run under a domain admin account, I can login to the database through Management Studio. I've checked the connection string and can run the test locally but not remotely. Does anyone have any ideas? My next step is to set the connection string to the UrlsDB to use SQL Authentication
Finally managed to resolve it at 01:20AM. When checking the datasources of three individual tests which made up the mixes in the scenario, I found that the UI was showing that once one had been updated all three updated the connection string so that is why I was baffled as to why I was getting these errors, plus the error doesn't indicate which connection was having the issue.
So to eliminate the tests as being the issue I removed the datasource from each test and created individually named brand new datasources all till effectively pointing to the same sql server and the same database. Then I ran the tests and all performed correctly, finally!!
So the core issue was the connection strings in the underlying tests were incorrect. Will be testing the UI further to check if I was just my own error or there may actually be a bug in the UI, if I find a bug I'll report it.
Thanks to those who took the time to try to help me solve it, gutted that the issue was so minor when it had me baffled for nearly 20 hours :/
The domain admin account you are running the test from cannot connect to the database server from the agent machine.
Log into the agent and debug the database connection from there.
Please be aware that a thread blocking call inside a web test, such as this may cause issues with your load test. I recommend that you load all test url's during the test instanciation if at all possible.
Essentially minimise the database calls to as few as possible.

Resources