When I run next build in my server, it gets stuck here, and the website also crashes and turns into a 504 Gateway Timeout error:
info - Loaded env from /home/ubuntu/MYPROJECT/.env
info - Using webpack 4. Reason: future.webpack5 option not enabled https://nextjs.org/docs/messages/webpack5
info - Checking validity of types
info - Using external babel configuration from /home/ubuntu/MYPROJECT/.babelrc
info - Creating an optimized production build..
This does not happen all the time, maybe 1 time every 5 to 10 deployments.
I then have to stop and start the EC2 instance. Right after the build command always runs very quick. Which could mean I have memory issues in the server? How could I resolve this?
I have a t2.micro instance with 8GB of storage.
Usage of /: 76.2% of 7.69GB
Memory usage: 50%
Could be that upgrading resolves this issue?
Related
We are constantly getting an error while starting our Beam Golang SDK pipeline (driver program) from a docker image which works when started from local / VM instance. We are using Dataflow runner for our pipeline and Kubernetes to deploy.
LOCAL SETUP:
We have GOOGLE_APPLICATION_CREDENTIALS variable set with service account for our GCP cluster. When running the job from local, job gets submitted to dataflow and completes successfully.
DOCKER SETUP:
Build image used is FROM golang:1.14-alpine. When we pack the same program with Dockerfile and try to run, it fails with error
User program exited: fork/exec /bin/worker: no such file or directory
On checking Stackdriver logs for more details, we see this:
Error syncing pod 00014c7112b5049966a4242e323b7850 ("dataflow-go-job-1-1611314272307727-
01220317-27at-harness-jv3l_default(00014c7112b5049966a4242e323b7850)"),
skipping: failed to "StartContainer" for "sdk" with CrashLoopBackOff:
"back-off 2m40s restarting failed container=sdk pod=dataflow-go-job-1-
1611314272307727-01220317-27at-harness-jv3l_default(00014c7112b5049966a4242e323b7850)"
Found reference to this error in Dataflow common errors doc, but it is too generic to figure out whats failing. After multiple retries, we were able to eliminate any permission / access related issues from pods. Not sure what else could be the problem here.
After multiple attempts, we decided to start the job manually from a new Debian 10 based VM instance and it worked. This brought to our notice that we are using alpine based golang image in Docker which may not have all the required dependencies installed to start the job.
On golang docker hub, we found a golang:1.14-buster where buster is codename for Debian 10. Using that for docker build helped us solve the issue. Self answering here to help anyone else facing the same issues.
I've been running a few containers (approximately a dozen) for awhile now. I've approached whatever the hard limit is on container/image sizes in the past, and had to clean these up to keep it from barfing all over everything, and recently the same has happened again.
I have identified several containers and images I can safely remove to reduce its footprint. But just as I was getting ready to do so, Docker crashed on me. And when I attempt to restart it, it crashes with the error message:
Fatal Error
Docker daemon failed to start
[timestamp] dockerd failed to start daemon: error initializing graphdriver: driver not supported
Thus, I can't use any of the command-line tools to remove these images/containers.
As there are running containers that I don't dare delete at this point, this makes it a little difficult to resolve. Is there a way to start Docker (on the mac) that doesn't actually start any of the containers so that maybe I can avoid this error?
Is the error message even related to my problem? I'm on Docker 2.3.0.4 if it matters.
You could switch to overlay2 driver instead of graph driver
You can follow the document below to switch
https://docs.docker.com/storage/storagedriver/overlayfs-driver/
Hope everyone is keeping safe!
I have a Ruby on Rails application hosted on AWS Beanstalk. I am using CloudFormation template, to update any stack for e.g., Ruby version, Linux Platform upgrade etc.
I was trying to upgrade, Linux box to 2.11.7 and Ruby to 2.6.6 and then ElasticSearch to 7.4
I was doing these changes in CloudFormation YML template and then I ran aws cloudformation update-stack command to apply these changes.
While the changes took time, I accidentally clicked on Rebuild Environment from Web AWS Console as a result, all the previously configured settings like SQS, Load balancer etc., were replaced by new settings.
Now, whenever I am trying to execute the update-stack command, it fails with below errors:
2020-06-09 15:25:44 UTC+0530
WARN
Environment health has transitioned from Info to Degraded. Command failed on all instances.
Incorrect application version found on all instances. Expected version "code-pipeline-xxxxxxxxxx" (deployment 2377). Application update failed 40 seconds ago and took 79 seconds.
2020-06-09 15:25:03 UTC+0530
INFO
The environment was reverted to the previous configuration setting.
2020-06-09 15:24:44 UTC+0530
INFO
Environment health has transitioned from Ok to Info. Application update in progress on 1 instance. 0 out of 1 instance completed (running for 39 seconds).
2020-06-09 15:24:30 UTC+0530
ERROR
During an aborted deployment, some instances may have deployed the new application version.
To ensure all instances are running the same version, re-deploy the appropriate application version.
2020-06-09 15:24:30 UTC+0530
ERROR
Failed to deploy application.
2020-06-09 15:24:30 UTC+0530
ERROR
Unsuccessful command execution on instance id(s) 'i-xxxxxxxxxx'. Aborting the operation.
2020-06-09 15:24:30 UTC+0530
INFO
Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
2020-06-09 15:24:30 UTC+0530
ERROR
[Instance: i-xxxxxxxxxx] Command failed on instance. Return code: 18 Output: (TRUNCATED)...g: the running version of Bundler (1.16.0) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. Your Ruby version is 2.6.6, but your Gemfile specified 2.6.5. Hook /opt/elasticbeanstalk/hooks/appdeploy/pre/10_bundle_install.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
2020-06-09 15:24:19 UTC+0530
INFO
Deploying new version to instance(s).
2020-06-09 15:23:45 UTC+0530
INFO
Updating environment developWeb's configuration settings.
2020-06-09 15:23:36 UTC+0530
INFO
Environment update is starting.
I can confirm that I have Ruby-2.6.6 set. I am not sure from where it is picking up the old version of Ruby?
Is there any way I can fix this? OR forcefully apply template changes?
Any help on this would be highly appreciated.
[UPDATE]: When I try to connect to ElasticSearch from Rails console, I get:
Faraday::ConnectionFailed: Failed to open TCP connection to old-elasticsearch-host-name.es.amazonaws.com:80 (Hostname not known: old-elasticsearch-host-name.es.amazonaws.com)
from /opt/rubies/ruby-2.6.6/lib/ruby/2.6.0/net/http.rb:949:in `rescue in block in connect'
Caused by SocketError: Failed to open TCP connection to old-elasticsearch-host-name.es.amazonaws.com:80 (Hostname not known: old-elasticsearch-host-name.es.amazonaws.com)
from /opt/rubies/ruby-2.6.6/lib/ruby/2.6.0/net/http.rb:949:in `rescue in block in connect'
Caused by Resolv::ResolvError: no address for old-elasticsearch-host-name.es.amazonaws.com
from /opt/rubies/ruby-2.6.6/lib/ruby/2.6.0/resolv.rb:94:in `getaddress'
The new URL of elasticsearch instance is different but it is still picking up the old URL from ELASTICSEARCH_HOST ENV variable.
Information from my CF template:
I can now provide info as per request. please tag me to see what I have in CF template
This was a config issue.
Whenever I was running aws update-stack command, it was going to s3 and pulling the zip (of the source code) code and in Gemfile of that zip code ruby version was set to 2.6.5.
So, I'd uploaded the fresh copy of the source code and then executed the update-stack command and it worked
I deployed my Application in AWS Elastic Beanslack. Intially I deployed my application directly in Aws console. After configuring all the things. I zipped my code and upload it in a console. At that time its working perfect.
But now I tried to deploy with cli, its shows error. I put eb deploy command
Creating application version archive "app-xxxxxxxxx".
Uploading: [##################################################] 100% Done...
2020-03-14 18:51:49 INFO Environment update is starting.
2020-03-14 18:51:55 INFO Deploying new version to instance(s).
2020-03-14 18:52:22 ERROR [Instance: i-xxxxxxxxx] Command failed on instance. Return code: 255 Output: (TRUNCATED)...ar/src/Composer/DependencyResolver/GenericRule.php on line 36
Fatal error: Out of memory (allocated 809508864) (tried to allocate 8192 bytes) in phar:///opt/elasticbeanstalk/support/composer.phar/src/Composer/DependencyResolver/GenericRule.php on line 36.
Hook /opt/elasticbeanstalk/hooks/appdeploy/pre/10_composer_install.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
2020-03-14 18:52:22 INFO Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
2020-03-14 18:52:22 ERROR Unsuccessful command execution on instance id(s) 'i-xxxxxxxxx'. Aborting the operation.
2020-03-14 18:52:23 ERROR Failed to deploy application.
From Internet I tried these all these things
1) I extended memory in php.ini but still its not working
2) I created ebextensions folder and do some configurations but thats also not working
My guess was Initially I deployed manually, so at that time I zipped with vendor folder also. Now when I tried with cli it wont take vendor folder. Instead of its by using composer install
So I think I'm facing this issues due to these things
Please let us know, If any other thing I want to do
This is a configuration inside de ElasticBeanstalk Environment.
To increase de memory limit:
Access the Elastic Beanstalk section
Open to your environment
Go to Configuration , then Software
Find the Memory limit field. The default value is 1024M.
Update the value to what you want
Apply the change
Redeploy your application
When deploy with ElasticBeanstalk (ELB) we should check Environment. With memory_limit, you can config this in Software tab (Configuration)
Follow:
Access the ELB
Open to your environment
Go to Configuration , then Software tab
Find the Memory limit field.
Change the value for what you want
Redeploy
I'm trying to set up TorqueBox inside Vagrant on Ubuntu Quantal. I've deployed my app into TorqueBox, but when I try to run bin/standalone.sh, it hangs for a long time after "Setting up Bundler" and then simply says "Killed".
I'm at a complete loss as to how to debug this.
I followed this guide for the installation of TorqueBox: http://torquebox.org/documentation/2.3.0/production-setup.html
Here's the full log: https://gist.github.com/elabs-dev/5411966
Is there a dump file in $TORQUEBOX_HOME/jboss/standalone/bin ? If so, it could indicate that the JVM is crashing.
Otherwise, it could be that there is insufficient memory available to deploy whatever you're deploying - how large is your app?