I can launch an on-demand EC2 instance and mount my EFS volume with complete success.
However when I try this using a spot instance using exactly the same configuration I find this in the system log...
mount.nfs4: Failed to resolve server fs-9273b65b.efs.eu-west-1.amazonaws.com: Name or service not known
If I log into the instance and check /etc/fstab the mount is there and if I then execute "sudo mount -a" the volume mounts without a hitch.
It would appear that at the instant the cloud-config script runs to mount that volume the name does not resolve but a few minutes later can be resolved without error.
Any guidance greatly appreciated.
Related
I'm trying to run ElasticSearch in Azure Container Instances. I've created the container like this using the Azure CLI:
az container create --image elasticsearch:7.4.2 --name $containerGroupName -g $resourceGroup --ip-address public --dns-name-label <mydns> --memory 8 --cpu 2 --ports 9200
The container ends up in a waiting state. When I check out the logs in the Azure portal, I'll see the following error:
ERROR: [2] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
Apparently, this has something to do with virtual memory: https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html
But how do I change it inside the container? I cannot connect to it since the container is not running.
Very late to the party but hopefully this will help someone in the future. Elasticsearch if deployed stand-alone requires the environment variable discovery.type to be set. Since Azure does not allow vars with a . (dot) at the moment you'll have to create a custom Docker image and host this e. g. with Azure Container Registry or Dockerhub, ...
Content can be as little as this:
FROM elasticsearch:7.8.1
ENV discovery.type=single-node
I realize that ACI is not the best option to run ElasticSearch. I only need it temporarily for a proof-of-concept while I'm waiting for a stable environment of ElasticSearch elsewhere.
Eventually I got it running by picking an older image of ElasticSearch:
az container create --image elasticsearch:5.6.14-alpine --name $containerGroupName -g $resourceGroup --ip-address public --dns-name-label <mydns> --memory 4 --cpu 2 --ports 9200
The solution to this is simply adding this line to the bound elasticsearch.yml config file:
discovery.type: single-node
For this to be permanent even when you restart the container, the config directory of the elasticsearch container needs to be bound to an Azure Fileshare directory, where you make permanent changes there.
Create a Azure file share and create a elasticsearch folder with another config folder inside it: https://docs.microsoft.com/en-us/azure/storage/files/storage-how-to-create-file-share
Deploy a elasticsearch container with the fileshare volume mounted and bind /mnt/elasticsearch/config to a folder you've created on the new fileshare (template tags: "mountPath": "/mnt/elasticsearch/config", "shareName": "elasticsearch/config"): https://docs.microsoft.com/en-us/azure/container-instances/container-instances-volume-azure-files
While the container doesn't error out and shuts itself down because of this max virtual memory error, copy the entire /usr/share/elasticsearch/config folder and filers to the fileshare folder. As soon as you get terminal access, run cp -R /usr/share/elasticsearch/config /mnt/elasticsearch/config
On your file share folder, you should now have a config directory with files and other folders created by the elastisearch startup process. One of this files is elasticsearch.yml. Open it, add the line discovery.type: single-node and save it.
Finally, change mounting and binding location to the correct location, so elasticsearch starts up and reads our now modified configurations. Bind /usr/share/elasticsearch/config to the fileshare /elasticsearch/config folder (template tags: "mountPath": "/usr/share/elasticsearch/config", "shareName": "elasticsearch/config"). More info here: https://www.elastic.co/guide/en/elasticsearch/reference/master/docker.html#docker-configuration-methods
Start the container with the new mounting locations and now the max virtual memory error should be gone.
Please, only use this trick for proofs-of-concept or very low volume logs. Using network storage like Azure Fileshare in general is not suitable for Elasticsearch and can even crash it.
I have a similar issue to the following link, but in powershell as I am running a clickhouse docker container in windows 10.
Data directory permissions on host for Clickhouse installation via docker
My setup is run as such:
docker run -d -p 8124:8124 --name my_database --ulimit nofile=262144:262144 --volume=E:/:/var/lib/clickhouse yandex/clickhouse-server
E drive is one of the drives on my windows computer.
I cannot seem to access /var/lib/clickhouse/data when running a mergetree table creation. It seems that clickhouse client is not being given adequate permissions to reach this file system. The error looks as such:
Access to file denied: /var/lib/clickhouse/data/default/test_mergetree/tmp_insert_20150731_20150731_8_8_0
Since I am in powershell I am unsure how I might approach solving this. I am attempting to access the file system to give powershell permissions:
Something like this
ICACLS "var/lib/clickhouse/data" /setowner "administrator"
But then since clickhouse is dockerized it seems I cannot find the path:
The system cannot find the path specified.
Would I have to run docker compose? Or am I approaching this the wrong way?
ATTEMPT 1
I've tried running the following:
docker run --rm -i --entrypoint /bin/sh yandex/clickhouse-server -c id clickhouse
#got back:
uid=0(root) gid=0(root) groups=0(root)
#went into the system and ran
docker exec -it container-id bash
chown -R 0:0 /var/lib/clickhouse
#got back
chown: cannot read directory '/var/lib/clickhouse/System Volume Information': Permission denied
You should run docker and clickhouse in Linux instead of Windows.
Turns out this is an issue which has not been repaired in windows docker desktop:
https://github.com/docker/for-win/issues/39
Volume mounts are necesarry. But I got around it by changing the disk image to the target host drive. Under settings -> advanced -> change the virtual hard disk image to the drive you want and you can write within that drive. Note you still won't have access to raw data.
Still an issue but the practical solution is to move to Docker Volumes.
With bind mount there is a problem with wsl2
I am using user-data of ec2 instance to power up my auto scale instances and run the application. I am running node js application.
But it is not working properly. I have debugged and checked the instance cloud monitor output. So it says
pm2 command not found
After reading and investigating a lot I have found that the path for the command as root is not there.
As EC2 user-data when it tries to run it finds the path
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
After ssh as ec2-user it is
/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/ec2-user/.local/bin:/home/ec2-user/bin
After ssh as sudo su it is
/root/.nvm/versions/node/v10.15.3/bin:/sbin:/bin:/usr/sbin:/usr/bin
It works only for the last path.
So what is the way or script to run the command as root during launch of the instance provided by user-data?
All thought to start your application with userdata is not recommended, because as per AWS documentation they are not assuring that instance will only come up after successful execution of user data. Even if user data failed it will spin up your instance.
For your problem, I assume if you give the complete absolute path of the binary, It will work.
/root/.nvm/versions/node/v10.15.3/bin/pm2
Better solution for this approach, create a service file for your application startup and start application with systemd or service.
I have installed salt-master and minion on aws ec2 instances and configured cloud in the below formatenter image description here
and i came up with the error like permission denied(publickey) can any one suggest me a better solution
Owner of the key file must be the same user that start your services (salt-master/minion).
Otherwise OS won't allow you to read such file
The command is:
chown this_maybe_salt_user /path/to/your/key
I did ssh into a remote Ubuntu host (Amazon EC2 instance), then set 777 permission for /home/user. After sometime the connection got closed. Now when I try to do ssh, I get error message Permission denied (publickey). How can I login again?
stop current instance and launch another instance with similar AMI and in the same zone.
detach disk volume from old instance and mount to new instance.
in the new instance, change the required file permissions inside the newly mounted volume
detach volume from new instance and attach to old instance.
Then I was able to SSH. Steps mentioned here in detail