As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
We just added an autoupdater in our software and got some bug report saying
that the autoupdate wouldn't complete properly because the downloaded file's sha1 checksum wasn't matching. We're hosted on Amazon S3...
That's either something wrong with my code or something wrong with S3.
I reread my code for suspicious stuff and wrote a simple script downloading and checking the checksum of the downloaded file, and indeed got a few errors once in while (1 out of 40 yesterday). Today it seems okay.
Did you experience that kind of problem? Is there some kind of workaround ?
extra info: test were ran in Japan.
Amazon's S3 will occasionally fail with errors during uploads or downloads -- generally "500: Internal Server" errors. The error rate is normally pretty low, but it can spike if the service is under heavy load. The error rate is never 0%, so even at the best of times the occasional request will fail.
Are you checking the HTTP response code in your autoupdater? If not, you should check that your download succeeded (HTTP 200) before you perform a checksum. Ideally, your app should retry failed downloads, because transient errors are an unavoidable "feature" of S3 that clients need to deal with.
It is worth noting that if your clients are getting 500 errors, you will probably not see any evidence of these in the S3 server logs. These errors seem to occur before the request reaches the service's logging component.
Other than the downtime a few weeks ago. None that I heard of.
They did a good job considering the one time it was down was because of an obscure server error that cascaded throughout the cloud. They was very open about it and resolve it as soon as they found out.(it happened during a weekend, iirc)
So they are pretty reliable. My advice is double check your code. And bring it up to amazon support if it is still a problem.
ok, this is all a bit old now, but for reference. I've just been running data migration of several gigs of data from an EC2 server directly into s3. I'm getting 500 errors about every 10 minutes or so, representing an error rate of about 1% of PUTs. So, yes, S3 does have a problem with 500 errors.
Haven't done much in the way of GET's though, so cant comment
I agree, quad-checking your code would be a good idea. I'm not saying that it can't happen, but I don't believe that I have ever seen it, and I've used S3 a pretty good bit now. I have, however, mismanaged exceptions/connection breaks a few times and ended up with pieces that didn't match what I was expecting.
I would be pretty surprised if they actually send bad data, but, as always, anything is possible.
Never heard of a problem during download. That's weird. I get TONS of 500 Internal Server Error messages when uploading. That's why I have a daemon that uploads while the user is doing something else.
It doesn't seem to be something in your code, maybe there is really something wrong with S3 (or with S3->Japan.)
You can try firing up an EC2 server, and just run the test from there (traffic won't cost any money, so use as much as you want!) and see if you get errors. If you do, then you're out of luck and S3 isn't for you :)
Good luck!
More than sending bad data, I think I got an ERROR403. If I just try again it's usually ok.
And I agree : I saw a lot of report about people talking about amazon being totally down, but nobody talking about a "sometimes my access is refused" error, so I guess there might be an error on my side. I just set up the log on amazon.
Anyway thank you! I'll follow your advise and stop blaming "the other guy".
I occasionally get unexpected 404 errors with GETs objects that are part of a preceeding LIST but new to the bucket, and other misc. errors (eg: 403 on my access id and secret key), but nothing catastrophic.
My code runs server side, so I've put in some robust error handling and logging. I think this is a wise thing to do anytime you have one server on the net communicating with another server. :P
Related
I'm a bit confused by a problem that has only become more apparently lately and I'm hoping that someone might be able to point me in the direction of either where I might look for appropriate settings, or if I am running into another problem they have come across before.
I have a Laravel application and a private server that I use for our little museum. Now as the application has become more complex, the lag is noticeable and you can see how it almost lines up the connections, finishing one request before moving along to the next, whether it be api, ajax, view responses, whatever.
I am running Apache 2.4.29 and my Ubuntu Server is 18.04.1.
I have been looking around but not much has helped, in regards to connections settings, if I look at my phpinfo() I see this Max Requests Per Child: 0 - Keep Alive: on - Max Per Connection: 100 but I believe these are just fine the way they are.
If I check my memory I think it says I have 65 GB of available memory, with 5 being used in caching. When reviewing the live data, the memory never crosses into the GB territory and solely remains in the MB territory. This server is absolutely only used for this Laravel project, so I don't have to worry about messing with other projects, I'd just like to make sure this application is getting the best use it can for its purpose.
I'd appreciate any suggestions, I know there's a chance the terms I am searching for are incorrect, or maybe just outdated, so if there are any potential useful resources out there, I'd appreciate those as well.
Thank you so much!
It's really hard to be able to tell since there a lot of details lacking but here some things that can give you a direction of where to look:
Try downloading htop via apt-get and see what happens on your CPU/RAM load with each request to the server.
Do you use php-fpm to manage the php requests? This might help in finding out if the problem lies in your PHP code or in apache configuration
Did you try deploying to a different server? Do you still see the lagging on the other server as well? If not, this indicates a misconfiguration problem and not an issue with your code.
Do you have other processes that are running in the background and might slow things down? Cron? Laravel Queue?
If you try to install another app on the server (let's say phpmyadmin) is it slow as well or it works fine?
Try to take it from here. Best of luck.
I was wondering whether it is possible to autoscale if the demand for requests escalates? What do people do if the app they just created goes viral in the middle of the night, and people starts getting error-codes instead of data? Or is such functionality in the pipeline?
If your app hits its request limit, your extra requests will begin to fail with error code 155 (RequestLimitExceeded). To prevent the requests from failing you should adjust the request limit slider for the relevant app on the on the Account Overview page.
Now, coming to your question, Can this be done automatically? As of now, I will say No. Parse currently requires you to manually do that. Having thoroughly gone through all their blog posts, I will say that there are no hints of this functionality coming in near future. Anyways this question can only be answered 100% "correctly" by someone from Parse. We, at stackOverFlow, can only guess.
This is a great question you raised! As I see parse is a good PaaS with all the "cloudy" features. Even the pricing looks new generation type hourly based, however if it is lacking of automation to adjust the limits you will still pay for your unused capacity over a period of time just as in old datacenters which really bothers me (unless you pay someone to continuously monitor the performance and manually set the limits).
I deployed an application on Heroku. I'm using the free service.
Quite frequently, I get the following error.
PG::Error: ERROR: out of memory
If I refresh the browser, it's ok. But then, it happens again randomly.
Why does this happen?
Thanks.
Sam Kong
If you experience these when running queries, your queries are complicated or inefficient. The free tier has no cache, so you're already out there.
If you're getting these errors otherwise, open a support ticket at https://help.heroku.com
heroku restart simply helped me though
If you are not in a free tier, its maybe because you are using too much memory connecting to PG.
Consider an app running on several dynos, with several processes, each with lots of threads, maybe you are filling up the pool.
Also, as it appears in Heroku's Help Center maybe you are caching too many statements that wont be used.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
I was asked this question once at an interview:
"Suppose you own a website where the server is at some remote location. One day, some user calls/emails you saying the site is abominably slow. How would you identify why the site is slow? Also, when you check the website yourself as any user would (using your browser), the site behaves just fine."
I could think of only one thing (which was shot down):
Check the server logs to analyse incoming traffic. Maybe a DoS attack or exceptionally high traffic. Interviewer told me to assume the server has normal traffic and no DoS.
I was kind of lost because I had never thought of this problem. I have almost no idea how running a server/website works. So if someone could highlight a few approaches, it would be nice.
While googling around, I could find only this relevant, wonderful article. That article is kind of too technical for me now, but I'm slowly breaking it down and understanding it.
Since you already said when you check the site yourself the speed is fine, this means that (at least for the pages you checked) there is nothing wrong with the server and it can serve those pages at a good speed. What you should be figuring out at this point is what the difference is between you and the user that reports your site is slow. It might be a lot of different things:
Is the user using a slow network connection (mobile for example)?
Does the user experience the same problems with other websites hosted at the same webhoster? If so, this could indicate a network problem. Normally this could also indicate a resource problem at the webserver, but in that case the site would also be slow for you.
If neither of the above leads to an answer, you could assume that the connection to the server and the server itself are fine. This means the problem must be in the users device. Find out which browser/OS he uses and try to replicate the problem. If that fails find out if he uses any antivirus or similar software that might cause problems.
This is a great tool to find the speed of web pages and tells you what makes it slow: https://developers.google.com/speed/pagespeed/insights
I think one of the important thing that is missing from above answers is the server location, which can play a vital in web performance.
When someone is saying that it is taking a longer time to open a web page that means high latency. High latency can be caused due to server location.
Let's assume as you are the owner of the web page then the server and client are co-located, so it will have a low latency.
But, now if client is across the border, then latency time will increase drastically. And hence a slow perfomance.
Another factor is caching which drastically affects the latency time.
Taking the example of facebook, they have server all over the world to reduce the latency time (and also to provide several other advantages) and they use huge caching system to cache their hot data (trending topics) whereas cold data (old data) are stored in hard disk so it takes a longer time to load an older photo or post.
So, a user might would have complained about this as they were trying load up some cold data.
I can think of these few reasons (first two are already mentioned above):
High Latency due to location of client
Server memory might need to be increased
Number of service calls from the page.
If a service could be down at the time of complaint, it could prevent page from loading.
The server load might be too high at the time of the poor experience. The server might need to increase the resources (e.g. adding another server/web server to the cluster).
Check if there was any background job running on the server at that time.
It is important to check the logs and schedules of the batch jobs to determine what all was running at that time.
Hope this help.
Normally the user takes the page loading time as a measure to find out that the site is slow. But if you really want to know that what is taking the maximum time the you can open the browser debugger by pressing f12. if your browser is chrome the click on network and see what calls your application is making and which are taking maximum time. If you are using Firefox the you need to install firebug. If you have that, then again press f12 and click on Net.
One reason could be the role of the user is different of your role. You might be having suppose an administrator privilege (some thing like super user role) and the code might be just allowing everything for such role that means it does not really do much of conditional checking to see what is allowed or not. Some times, it's a considerable over ahead to get all the privileges of the user and have the conditions checking, how course depends how how the authorization is implemented. That means, the page might be really slow for specific roles. Hence, you should find out the roles of the user and see if that is a reason.
Obviously an issue with the connection of the person connecting to your site OR it's possible it was a temporary issue and by the time you checked your site, everything was dandy. You could check your logs or ask your host if there was an issue at the time the slow down occured.
This is usually a memory issue and it can be resolved by increasing the Heap Size of the Web Server hosting the application. In case the application is running on Weblogic Server. Heap size can be increased in "setEnv" file located in Application Home.
Goodluck!
Michael Orebe
Though your question is quite clear, web site optimisation is a very extensive subject.
The majority of the popular web developing frameworks are for some reason, extremely processor inefficient.
The old fashioned way of developing n-tier web applications is still very relevant and is still considered to be best practice according the W3C. If you take a little time to read the source code structure of the most popular web developing frameworks you will see that they run much more code at the server than is necessary.
This may seem a bit of a simple answer but, the less code you run at the server and the more code you run at the client the faster your servers will work.
Sometimes contrasting framework code against the old fashioned way is the best way to get an understanding of this. Here is a link to a fully working mini web application which represents W3C best practices and runs the minimum amount of code at the server and the maximum amount of code at the client: http://developersfound.com/W3C_MVC_EX.zip this codebases is also MVC compliant.
This codebase comes with a MySQL database dump, php and client side code. To see this code in action you will need to restore the SQL dump to a MySQL instance (sql dump came from MySQL 8 Community) and add the user and schema permissions that are found in the php file (conn_include.php); setting the user to have execute permissions on the schema.
If you contrast this code base against all of the most popular web frameworks, it will really open your eyes to just how inefficient these frameworks are. The popular PHP frameworks that claim to be MVC frameworks aren’t actually MVC compliant at all. This is because they rely on embedding PHP tags inside HTML tags or visa-versa (considered very bad practice according the W3C). Also most popular node frameworks run way more code at the server than is necessary. Embedded tags also stop asynchronous calls from working properly unless the framework supports AJAX dumps such as Yii 2.
Two of the most important rules to follow with MVC compliance is: never embed server side tags (such as PHP tags) in HTML tags or visa-versa (unless there is a very good excuse such as SEO) and religiously never write code to run at the server if it can be run at the client. Also true MVC is based on tier separation, where as the MVC frameworks are based on code separation. True MVC compliance is very processor efficient. Don’t get me wrong MVC frameworks are very useful for a lot of things, but if you’re developing a site that is going to get millions of hits, they are quite useless, or at least they will drive your cloud bills so high that it will really eat into your company’s profits.
In summary frameworks don’t give much control over what code runs at the client or server and are very inefficient but you can get prototypes up and running quicker with less code.
In contrast the old fashioned way takes a bit more elbow grease but you have complete control over what runs at the server and what runs at the client.
As an additional bit of advice for optimisation avoid using pass-through queries and triggers and instead opt for stored procedures. Historically stored procedures weren’t invented at the time MVC was present as a paradigm but it definitely increases separation of concerns between the tiers and is much more processor efficient.
Hope this advice helps.
I have a few sites which are exhibiting a slow load time. All are WordPress 3.5. All are hosted through BlueHost. All are developed by me (built as child-themes of existing WP themes).
Using Safari Developer tools, I see that they average 4–6 seconds (not ms) of latency before anything happens, which appears to be abnormally high. I've tried to wrap my head around latency, and I know I'm not the only one to ask about it here ... but I cannot figure out if the primary culprit is my hosting provider (Bluehost) or with my development.
Here are a couple of my sites with issues:
http://www.HubbardProductions.com
http://www.xla.com
Can anyone point me in the right direction? What can I do to reduce the latency?
you can see from here. your website is responding lately. http://i.imgur.com/VIVoq.png
http://tools.pingdom.com/fpt/#!/jyKI0Kv01/http://hubbardproductions.com/
Chris, same problem here. Also with Bluehost + Wordpress 3.5.
Some minutes ago, my sites even went down, and I was unable even to access cPanel. I received the following error:
Auth failed69.89.31.120:2083 is temporarily down.
I contacted the technical staff and they told me to try again, deleting cookies, and also sent me this url:
https://my.bluehost.com/cgi/help/481
Which, in my case, is of little help, but perhaps it can help you.
I asked them if there was any problem with the servers lately and they said nope, no issues.
So, to answer your question, I would:
Wait a few days, in case it is temporary (I hope).
If not, I would run some tests with simple html pages, then php, then php + simple SQL, etc., to find the bottleneck, and if it is a server issue or a wordpress issue.
If I find it is a server issue, I would complain.
If everything fails, I would move my sites to other hosting. Bye-bye Bluehost. :(
Good luck!