CodeDeploy Timeout Error and Corrupted EC2 - shell

I have an issue where my most recent CodeDeploy failed due to a timeout and am now not able to connect or load my EC2 instance. It didn't rollback to the previous working version and since I can't connect I can't take a deeper look at the CodeDeploy logs. When looking at the log tail there isn't anything glaring about the error log and I don't think it is related to the timeout error, but I'm not sure what in my shell script could be creating the timeout. Any ideas of what might be wrong in my setup? Should I do a check for a node_modules folder before running sudo npm install?
Error:
Error Code: ScriptTimedOut
Script Name:scripts/npm-install.sh
Message: Script at specified location: scripts/npm-install.sh failed to complete in 300 seconds
Log Tail:
LifecycleEvent - AfterInstall
Script - scripts/npm-install.sh
[stderr]npm WARN deprecated lodash-node#3.10.2: This package is discontinued. Use lodash#^4.0.0.
[stderr]npm WARN deprecated sendgrid#4.10.0: Please see v6.X+ at https://www.npmjs.com/org/sendgrid
[stderr]npm WARN deprecated nodemailer#2.7.2: All versions below 4.0.1 of Nodemailer are deprecated. See https://nodemailer.com/status/
[stderr]npm WARN deprecated mailcomposer#4.0.1: This project is unmaintained
[stderr]npm WARN deprecated socks#1.1.9: If using 2.x branch, please upgrade to at least 2.1.6 to avoid a serious bug with socket data flow and an import issue introduced in 2.1.0
[stderr]npm WARN deprecated buildmail#4.0.1: This project is unmaintained
[stderr]npm WARN deprecated sendgrid#1.9.2: Please see v6.X+ at https://www.npmjs.com/org/sendgrid
[stderr]npm WARN deprecated mailparser#0.6.2: This project is unmaintained
[stderr]npm WARN deprecated mimelib#0.3.1: This project is unmaintained
[stderr]
Here is my appspec.yml:
version: 0.0
os: linux
files:
- source: /
destination: /var/www/app/
hooks:
AfterInstall:
- location: scripts/npm-install.sh
runas: ec2-user
timeout: 300
ApplicationStart:
- location: scripts/npm-start.sh
runas: ec2-user
timeout: 60
npm-install.sh:
#!/bin/bash
source /home/ec2-user/.bash_profile
cd /var/www/app
sudo npm install
npm-start.sh:
#!/bin/bash
source /home/ec2-user/.bash_profile
cd /var/www/app
npm start

As you probably know, the 300 second timeout is configured in your appspec.yml file. I'm assuming that npm install should finish in less than 300 seconds.
With the information provided, it's impossible to know what went wrong. Per your concern about the node_modules directory not existing, it won't matter because npm install will create that directory if it doesn't already exist. If possible, I would strongly suggest upgrading the deprecated packages. It's possible NPM is not happy about that.
As for your instance, there is nothing in the scripts that you've provided that should kill your instance. If you can't connect to your EC2 instance, you'll probably want to recycle it and get a new one. If you have something actually running on the EC2 instance you can't connect to, you could create a new CodeDeploy deployment group with the new instance, deploy the code and see if the instance remains healthy. This could be an intermittent oddity, and it would be more valuable to troubleshoot if the issue becomes repeatable.

Related

Can't upgrade sonarqube into the latest LTS

I am trying to update sonarqube from 7.9 to 8.9.
After installing the new version, in the logs, I had an error in the logs:
java.lang.RuntimeException: can not run elasticsearch as root
So I modified sonar.sh with RUN_AS_USER=sonar
I have make a chown and a chgrp for the sonar user on opt/sonarqube
I try to restart sonarqube with /opt/sonarqube/bin/linux-x86-64/sonar.sh start, in the log I have the followinf error:
2022.07.08 09:24:14 ERROR app[][o.s.a.ProcessLauncherImpl] Failed to create temporary configuration directory [/opt/sonarqube/temp/conf/es]
2022.07.08 09:24:14 ERROR app[][o.s.a.p.ManagedProcessHandler] Fail to launch process [es]
Yet the rights are still good on the server.
Any ideas please?
Thank you,

Update AWS stack with CF template

Hope everyone is keeping safe!
I have a Ruby on Rails application hosted on AWS Beanstalk. I am using CloudFormation template, to update any stack for e.g., Ruby version, Linux Platform upgrade etc.
I was trying to upgrade, Linux box to 2.11.7 and Ruby to 2.6.6 and then ElasticSearch to 7.4
I was doing these changes in CloudFormation YML template and then I ran aws cloudformation update-stack command to apply these changes.
While the changes took time, I accidentally clicked on Rebuild Environment from Web AWS Console as a result, all the previously configured settings like SQS, Load balancer etc., were replaced by new settings.
Now, whenever I am trying to execute the update-stack command, it fails with below errors:
2020-06-09 15:25:44 UTC+0530
WARN
Environment health has transitioned from Info to Degraded. Command failed on all instances.
Incorrect application version found on all instances. Expected version "code-pipeline-xxxxxxxxxx" (deployment 2377). Application update failed 40 seconds ago and took 79 seconds.
2020-06-09 15:25:03 UTC+0530
INFO
The environment was reverted to the previous configuration setting.
2020-06-09 15:24:44 UTC+0530
INFO
Environment health has transitioned from Ok to Info. Application update in progress on 1 instance. 0 out of 1 instance completed (running for 39 seconds).
2020-06-09 15:24:30 UTC+0530
ERROR
During an aborted deployment, some instances may have deployed the new application version.
To ensure all instances are running the same version, re-deploy the appropriate application version.
2020-06-09 15:24:30 UTC+0530
ERROR
Failed to deploy application.
2020-06-09 15:24:30 UTC+0530
ERROR
Unsuccessful command execution on instance id(s) 'i-xxxxxxxxxx'. Aborting the operation.
2020-06-09 15:24:30 UTC+0530
INFO
Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
2020-06-09 15:24:30 UTC+0530
ERROR
[Instance: i-xxxxxxxxxx] Command failed on instance. Return code: 18 Output: (TRUNCATED)...g: the running version of Bundler (1.16.0) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. Your Ruby version is 2.6.6, but your Gemfile specified 2.6.5. Hook /opt/elasticbeanstalk/hooks/appdeploy/pre/10_bundle_install.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
2020-06-09 15:24:19 UTC+0530
INFO
Deploying new version to instance(s).
2020-06-09 15:23:45 UTC+0530
INFO
Updating environment developWeb's configuration settings.
2020-06-09 15:23:36 UTC+0530
INFO
Environment update is starting.
I can confirm that I have Ruby-2.6.6 set. I am not sure from where it is picking up the old version of Ruby?
Is there any way I can fix this? OR forcefully apply template changes?
Any help on this would be highly appreciated.
[UPDATE]: When I try to connect to ElasticSearch from Rails console, I get:
Faraday::ConnectionFailed: Failed to open TCP connection to old-elasticsearch-host-name.es.amazonaws.com:80 (Hostname not known: old-elasticsearch-host-name.es.amazonaws.com)
from /opt/rubies/ruby-2.6.6/lib/ruby/2.6.0/net/http.rb:949:in `rescue in block in connect'
Caused by SocketError: Failed to open TCP connection to old-elasticsearch-host-name.es.amazonaws.com:80 (Hostname not known: old-elasticsearch-host-name.es.amazonaws.com)
from /opt/rubies/ruby-2.6.6/lib/ruby/2.6.0/net/http.rb:949:in `rescue in block in connect'
Caused by Resolv::ResolvError: no address for old-elasticsearch-host-name.es.amazonaws.com
from /opt/rubies/ruby-2.6.6/lib/ruby/2.6.0/resolv.rb:94:in `getaddress'
The new URL of elasticsearch instance is different but it is still picking up the old URL from ELASTICSEARCH_HOST ENV variable.
Information from my CF template:
I can now provide info as per request. please tag me to see what I have in CF template
This was a config issue.
Whenever I was running aws update-stack command, it was going to s3 and pulling the zip (of the source code) code and in Gemfile of that zip code ruby version was set to 2.6.5.
So, I'd uploaded the fresh copy of the source code and then executed the update-stack command and it worked

Cassandra Detected unreadable sstables(data not caches)

ERROR [main] 2017-08-04 13:24:21,949 CassandraDaemon.java:638 - Detected unreadable sstables /opt/cassandra/data/some_key_space/ep_lc_events-adc44160dbe611e6953689bcd3ed73aa/mc-547-big-Summary.db, and many others...
That has happened after I upgraded Cassandra to 3 version and after a while downgraded it to 2nd version.
When I run this command: sudo service cassandra status
I have got such message:
could not access pidfile for Cassandra
In /var/log/cassandra/system.log I have logs which I wrote at the beginning.
PS: let me pay your attention that everything is happening on EC2 Amazon instance.
Well, I have just upgraded back to 3rd version, used cassandra-unloader to export all data, then downgraded back to 2nd version and used cassandra-loader to import all data. But if you were lucky and had backups and snapshots it would not be an obstacle for you.
PS. Afterwards, I had to run this command nodetool resetlocalschema to reset local schema and resynchronize.
PPS. This you can find how to do that.
https://github.com/brianmhess/cassandra-loader
I also got the same error, but it was due to switching between cassandra 4.0.0 and version 3.11 and back again while using docker.
Update the version to the right one for the ssltable, or delete the data volume:
docker-compose logs cassandra
docker volumes ls
docker ps
docker-compose down
docker volume rm testapp_cassandra
docker volume ls
docker-compose up

Elasticsearch fails to start or run on OSX

I am posting to hopefully help others if they run into this issue on Mac. I recently updated ES to 2.2.x branch using Homebrew:
brew uninstall --force elasticsearch
brew update
brew install elasticsearch
I repeatedly got connection errors trying both localhost and 127.0.0.1 on port 9200.
curl http://localhost:9200
curl: (7) Failed to connect to localhost port 9200: Connection refused
I tried an unload and load.
launchctl unload ~/Library/LaunchAgents/homebrew.mxcl.elasticsearch.plist
launchctl load ~/Library/LaunchAgents/homebrew.mxcl.elasticsearch.plist
Then tried starting manually.
elasticsearch
The following errors appeared indicating that the Java version 1.7.x was an error and why it would not start.
Exception in thread "main" java.lang.RuntimeException: Java version: Oracle Corporation 1.7.0_45 [Java HotSpot(TM) 64-Bit Server VM 24.45-b08] suffers from critical bug https://bugs.openjdk.java.net/browse/JDK-8024830 which can cause data corruption.
Please upgrade the JVM, see http://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html for current recommendations.
If you absolutely cannot upgrade, please add -XX:-UseSuperWord to the JAVA_OPTS environment variable.
Upgrading is preferred, this workaround will result in degraded performance.
at org.elasticsearch.bootstrap.JVMCheck.check(JVMCheck.java:123)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:283)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.
After getting past this error, there were errors as well for previously-installed plugins on the 1.7.x branch.
Exception in thread "main" java.lang.IllegalStateException: Could not load plugin descriptor for existing plugin [bigdesk]. Was the plugin built before 2.0?
Likely root cause: java.nio.file.NoSuchFileException: /usr/local/var/lib/elasticsearch/plugins/bigdesk/plugin-descriptor.properties
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:315)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:380)
at java.nio.file.Files.newInputStream(Files.java:106)
at org.elasticsearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:87)
at org.elasticsearch.plugins.PluginsService.getPluginBundles(PluginsService.java:378)
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:128)
at org.elasticsearch.node.Node.<init>(Node.java:146)
at org.elasticsearch.node.Node.<init>(Node.java:128)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:145)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:285)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.
The solutions to these issues I discovered were the following:
Edit the /usr/local/etc/elasticsearch/elasticsearch.yml file and verify the bind_host configs are commented so it will default to 0.0.0.0.
Edit the /usr/local/Cellar/elasticsearch/YOUR_VERSION/libexec/bin/elasticsearch.in.sh file and add the -XX:-UserSuperWord flag after the other JAVA_OPTS:
JAVA_OPTS="$JAVA_OPTS -XX:-UseSuperWord"
Manually remove the previous plugins so you can install the latest versions for that ES branch:
/usr/local/Cellar/elasticsearch/2.2.0_1/libexec/bin/plugin list
Installed plugins in /usr/local/var/lib/elasticsearch/plugins:
- bigdesk
- head
/usr/local/Cellar/elasticsearch/2.2.0_1/libexec/bin/plugin remove bigdesk
/usr/local/Cellar/elasticsearch/2.2.0_1/libexec/bin/plugin remove head
After these steps I could once again start ES 2.x and then I can re-install any desired plugins. I hope this helps others if they run into similar issues.
I had this issue on MacOs, when I tried to uninstall elasticsearch7 and install elastisearch#6 using homebrew.
I resolved it by manually deleting the following directories and installed elasticsearch#6
Data: /usr/local/var/lib/elasticsearch/
Logs: /usr/local/var/log/elasticsearch/elasticsearch_<<user>>.log
Plugins: /usr/local/var/elasticsearch/plugins/
Config: /usr/local/etc/elasticsearch/
steps:
>brew uninstall elasticsearch
>rm -rf <<above mentioned directories>>
>brew install elasticsearch#6

Is the latest version of docker from Amazon on ec2 broken?

As of last night all our new docker deployments started failing because the latest version of docker (docker-1.3.2-1.0.amzn1.x86_64) in the amazon repo fails to start up.
Steps to reproduce are:
## Launch instance with default amazon AMI
yum install docker-1.3.2-1.0.amzn1.x86_64
service docker restart
### Get the following error in /var/log/docker
2014/11/26 05:14:16 docker daemon: 1.3.2 c78088f/1.3.2; execdriver: native; graphdriver:
[8f6d7cfb] +job serveapi(unix:///var/run/docker.sock)
[info] Listening for HTTP on unix (/var/run/docker.sock)
docker: relocation error: docker: symbol dm_task_get_info_with_deferred_remove,
version Base not defined in file libdevmapper.so.1.02 with link time reference
If I downgrade back to docker-1.3.1-1.0.amzn1.x86_64 everything seems to be fine.
Is the AWS package actually broken, or is it just our setup?
Is there a work around other than downgrading?
Yes, it is broken for me too.
Downgrading has been the solution yet.
The same error was by me on a centos VM provisioned at my workplace - a yum update resolved it.
I suspect a build was broken but went out, and has been fixed subsequently.

Resources