I have a couple of micro instances that have been working fine for weeks. Both are running WordPress blogs. In the last 24 hours one of them has stopped. I can't ssh in even after a reboot. The other instance is working fine.
ssh: connect to host ec2-xxx-xxx-xxx-xxx.ap-southeast-1.compute.amazonaws.com port 22: Operation timed out
There in nothing obvious in the log that looks like a problem. The last few lines are:
cloud-init: runcmd[ OK ]
Mounting other filesystems: [ OK ]
Retrigger failed udev events[ OK ]
Generating SSH1 RSA host key: [ OK ]
Starting sshd: [ OK ]
Starting ntpd: [ OK ]
Starting sendmail: [ OK ]
Starting sm-client: [ OK ]
Starting crond: [ OK ]
[ OK ]
Starting atd: [ OK ]
Starting yum-updatesd: [ OK ]
Running cloud-init user-scripts (none found)[ OK ]
Amazon Linux AMI release 2011.02.1.1 (beta)
Kernel 2.6.35.11-83.9.amzn1.i686 on an i686
ip-xx-xxx-xx-xx login:
The management console states that everything is running and normal.
I use the same security group and .pem file for both instances.
I suspect that this instance has been getting more traffic than the other one. Is there anyway that the micro instance could run out of memory and just stop responding? What could be going wrong?
Here is a screen shot of the Monitoring panel
Thanks
I've seen micro instances lock up for several minutes due to the CPU "stealing" that occurs when you use too much CPU. This is unique to the micro instance. I blogged an example of this (including video) at http://gregsramblings.com/2011/02/07/amazon-ec2-micro-instance-cpu-steal/.
You can move your instance to new resources simply by doing a full STOP and then a START. This will assign it to new hardware and will assign a new IP address (don't forget to re-associate your elastic IP!). A host reboot will not accomplish this. It needs to be stopped via the EC2 console. Terminating it is not necessary.
There are several possibilities, but the two most likely are:
High load on the host that your Micro instance is running on - Micro instances get a small slice of resources anyway, and get scaled back quite harshly when the host is under load.
A fault has occurred on the host which is impacting VM responsiveness - this is actually relatively common, and can exhibit the type of behaviour you're seeing.
In either case, the quickest solution is to nuke the instance and restart it - you'll likely get a new instance on a different host, which may be less stressed or less broken. ;)
I just restarted the instance and it started working again. See the screenshot here
Related
I’m having an issue setting up the Firebase Emulator on a Mac Mini with Apple Silicon (M1 chip).
Every request I send to Firestore (using the emulator) takes a very long time to complete (sometimes it never completes and I get a network error saying the backend couldn’t be reached):
#firebase/firestore: Firestore (8.2.5): Could not reach Cloud Firestore backend. Backend didn't respond within 10 seconds.
This typically indicates that your device does not have a healthy Internet connection at the moment. The client will operate in offline mode until it is able to successfully connect to the backend.
It turns out this is happening because I'm setting host to 0.0.0.0 in the firebase.json file. If I remove the host field or if I set it to be localhost, then I don’t have any issues.
This is my firebase.json file:
{
"firestore": {
"rules": "./firestore.rules"
},
"emulators": {
"firestore": {
"host": "0.0.0.0",
"port": 8080
}
}
}
The reason I'm setting it to 0.0.0.0 is because I need to test my app on a different machine (i.e. on a tablet) and the only way we managed to make it work was setting the host to be 0.0.0.0 since this host will resolve all addresses associated with our machine. This way, I can access the emulator using my local IP address (i.e. 192.168.0.1).
I noticed this issue only happens on the new Mac with the M1 Apple Silicon chip. So, I wonder if this has anything to do with the way M1 is resolving the IP address.
Any ideas?
I'm still not sure if this is M1-specific since I haven't been able to test this on another M1 machine but adding my computer's name next to localhost in the hosts file seems to have fixed this issue:
127.0.0.1 localhost Will.local
::1 localhost Will.local
I had the same issue with my new M1 Macbook Pro. The problem did remaining no matter if there were host info on firebase.json. -file.
I found an exciting solution to this problem. I switched my Mac to Google's DNS -servers (8.8.8.8 and 8.8.4.4), and suddenly everything started to work without any significant delays. Before that, the initial call sometimes took 10-15 seconds or even more.
I trying to setup AWS Cloud9 and am running into a wall each time I try to setup my environments. Once I create the environment and start following this guide https://docs.aws.amazon.com/cloud9/latest/user-guide/sample-lamp.html to configure the LAMP server, through the Cloud9 IDE terminal, the environment will just stop responding. Once I try to reload the IDE I get the follow error;
Cannot connect to instance error message.
Rebooting the instance doesn't seem to resolve the error message. But any time I make a fresh instance it will let me work from anywhere to 30 seconds to 90 seconds before it stops responding.
I have looked through my VPC port settings, as well as security group settings, and they both appear to the correct.
VPC inbound rules VPC outbound rules Security Group inbound
rules Security Group outbound rules
Additionally, I was using the default t2.micro instance until I read this post AWS Cloud9: Cannot open environment and have tried with the t2.small but I am still getting the same results.
Any help with where else to look or what else to try would be much appreciated!
Edit: It appears to be random when it stops and freezes, for example when making a m4.large instance. It froze while I was setting up the sudo mysql_secure_installation.
Once I typed "Y" it wouldn't let me press enter. Reloading the IDE gave me the VPC error.
Welcome to SO! When I use cloud9 I tend to use m4.large for anything that's non-trivial. If you're running Apache and MySQL on the same host I would definitely try the m4.large instance. It's $0.10/hr (pricing) so you could try it out fairly cheaply. I'm guessing that's the root of the issue. If you're still having the issue please repost here and we can check further.
Just to confirm:
- You can connect to the instance at least once (even if for a few seconds)
- You see the IDE and can type for 30-60 seconds before it stops responding
If you can't connect that's likely a different issue.
I've spun up and setup 2 servers now in this AZ and both keep going down shortly after being used. Restarting doesn't fix it and I have to start and stop them.
By going down I mean a timeout error for apache and I can't even SSH. Port 22 is open and I can SSH in once I stop and start the EC2, it doens't run out of memory and only has 2 sites on a t2.medium with the database on RDS.
One of the sites is https://coffeehive.com.au/ if that helps
Any ideas?
Cheers,
Adrian
I booted the DataStax AMI for Amazon EC2, logged in via SSH, but the terminal hangs on "Installation Started":
Cluster started with these options:
--clustername CassandraDev --totalnodes 1 --version enterprise --username **** --password ****
Installation started.
"Installation started" keeps going through suffixes consisting of one, two, and three dots. But nothing happens, I can't quit the installation process, and I can't access any log files to see what might be going on (or I don't know how).
Tried on two separate m3.large instances operating in a VPC subnet, at the us-east-1 region. The exact AMI is datastax_clustering_ami_2.5.1_hvm.manifest.xml (ami-ada2b6c4). On the first instance, I waited about an hour and a half. The second instance I just left online all night, with the same results.
Because this is a VPC, all outbound traffic goes through a NAT server. Security groups allow outbound traffic only on ports 80, 443, and 123. Might there be another outbound port that needs to be opened? Inbound ports do not matter, as the server is not public-facing, but within the subnet I have allowed all traffic on all ports.
Someone else has had a similar issue, but without answers so far: DataStax AMI hangs on
Any help would be appreciated!
Since there were a few tickets that came up recently around the same issue, it seems as though something recently changed within the AMI provisioning side in EC2, or this specific configuration of VPCs had never been used before, which seems a bit unlikely.
The current fix is to add an additional entry into /etc/hostname to get rid of the stderr output that occurs after each sudo command. This in turn doesn't get flagged as an error on the provisioning side.
This has been fixed and patched as documented on this ticket:
https://github.com/riptano/ComboAMI/issues/51.
If you spot any additional issues, feel free to create another ticket there.
Going forward, just launch another set of instances using the same user-data and you should be up and running.
I added a bad script in startup script, turned out this script is blocking and hence ubuntu instance can't boot up to ssh server and I can't ssh into it. Is there a way for me to go to the server console (like from VGA port)? like go to single-user mode or safe mode and fix it?
Thanks in advance.
I don't know of any way to obtain server console access or change boot modes on the EC2 instances...
I had the same issue sometime ago, and I ended up creating a temporary EC2 instance, mounting the root device from the original (failing) EC2 there, modifying the files, then reattaching the device and destroying the temporary instance. Note: you may end up paying more depending on which instance type you launch.