I am using flatcar ami as the base os for my AWS EC2 instances.
I added a journald.conf file under the folder /etc/systemd/ during the baking of the AWS image (I want to increase the journalctl retention).
I was expecting to see the new journald configurations read during the boot of the AWS instance. Checking with the command systemctl status systemd-journald it didn't happen.
What I tried:
Using a journald.conf under the /etc/systemd/journald.conf.d/ path;
Restarting the systemd-journald service after the boot with a systemd service.
Important notes:
If I run systemctl restart systemd-journald inside the instance, the configurations are updated;
I did a cat in ExecStartPre before restarting the systemd-journald using an auxiliary service and confirmed the contents. Yet it doesn't pick up the configuration
The file is present, right and under the right path during the boot.
Related
I'm trying to launch Neo4J graph database on AWS using their AIM image (enteprise 3.3.9)
However, the server fails to launch the instance automatically how it's supposed to.
When I try to relaunch it using
systemctl restart neo4j
It also fails.
When I do
systemctl cat neo4j
I find the /etc/neo4j/pre-neo4j.sh file, which is apparently launched on the instance's startup, which, in turn launches Neo4J (when it's supposed to work):
[Unit]
Description=Neo4j Graph Database
After=network-online.target
Wants=network-online.target
[Service]
ExecStart=/etc/neo4j/pre-neo4j.sh
Restart=on-failure
User=neo4j
Group=neo4j
Environment="NEO4J_CONF=/etc/neo4j" "NEO4J_HOME=/var/lib/neo4j"
LimitNOFILE=60000
TimeoutSec=120
SuccessExitStatus=143
[Install]
WantedBy=multi-user.target
So then I launch it manually via the bash script using the sudo prefix and then it starts up fine.
sudo /etc/neo4j/pre-neo4j.sh
The documentation on deploying Neo4J on an AWS server doesn't mention anything about permissions if you use their image. So what can be the problem?
I don't want to have manually launch the DB using the sudo — is it possible to resolve this problem by modifying the bash script itself?
..
The file /etc/neo4j/pre-neo4j.sh sets some environmental parameters and then launches neo4j via:
/usr/share/neo4j/bin/neo4j console
Based on the comments.
The solution was to use
journalctl -u neo4j
to inspect the logs associated with the failed start of neo4j. This enabled to identify the root cause, and subsequently, to fix the issue.
I'm using Ubuntu 18.04.4 LTS which uses systemd, but the squid package packaged with this version of Ubuntu is configured to start via init.d. It starts and runs via systemctl start squid.service if I start it manually after the system has booted.
However, I'm using a ZFS mount point ("/media") to store the cache data, and during the boot process squid is starting before this mount point is active. Consequently I'm getting the error "Failed to verify one of the swap directories". Full output of systemctl status squid is here
I'd like to tell systemd to wait until after media.mount has completed in the most minimally invasive way possible (e.g. without modifying the /etc/init.d/squid file that is maintained by the package). To that end I created the /etc/systemd/system/squid.service.d/override.conf file like so:
% cat /etc/systemd/system/squid.service.d/override.conf
[Unit]
Wants=network.target network-online.target nss-lookup.target media.mount
After=network.target network-online.target nss-lookup.target media.mount
[Install]
WantedBy=multi-user.target
But squid is still starting too early.
Is what I want to do possible? Or do I have to bite the bullet and define a native /etc/systemd/system/squid.service file and remove the /etc/init.d/squid init script?
I am using user-data of ec2 instance to power up my auto scale instances and run the application. I am running node js application.
But it is not working properly. I have debugged and checked the instance cloud monitor output. So it says
pm2 command not found
After reading and investigating a lot I have found that the path for the command as root is not there.
As EC2 user-data when it tries to run it finds the path
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
After ssh as ec2-user it is
/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/ec2-user/.local/bin:/home/ec2-user/bin
After ssh as sudo su it is
/root/.nvm/versions/node/v10.15.3/bin:/sbin:/bin:/usr/sbin:/usr/bin
It works only for the last path.
So what is the way or script to run the command as root during launch of the instance provided by user-data?
All thought to start your application with userdata is not recommended, because as per AWS documentation they are not assuring that instance will only come up after successful execution of user data. Even if user data failed it will spin up your instance.
For your problem, I assume if you give the complete absolute path of the binary, It will work.
/root/.nvm/versions/node/v10.15.3/bin/pm2
Better solution for this approach, create a service file for your application startup and start application with systemd or service.
I've created a cloud-init file using Terraform's aws_instance.user_data parameter.
This then is executed on start up, on Centos machines, by the cloud-init systemd service.
I would like to edit this file for dev/testing purposes on the fly then simply restart the said service.
To this end, where can I find the cloud-init file that contains the commands run by the cloud-init service?
On my Centos machine, this was found in the file:
/var/lib/cloud/instances/<instance-id>/user-data.txt
Where instance-id is found in the file /var/lib/cloud/data/instance-id.
I have installed Mesosphere DCOS in AWS using the provided template. Now I would like to restart all the nodes but adding the --insecure-registry parameter to all the slave nodes (and master as well) so that they communicate with my docker registry. So I was reading the best way to do this is on the cloud-config script for the AWS template.
So in the AWS EC2 Launch Configurations I copied the configuration of the master node, then adjusted the User Data then updated the auto scaling groups and restarted the master. (awesome answer how to do this at the end of How do I use insecure docker registries with Amazon EC2 Container Service (ECS)?)
The lines were added to the end of the units section in cloud-config as suggested by CoreOS docs:
https://coreos.com/os/docs/latest/cloud-config.html
units
....Many lines here
- name: docker.service
drop-ins: |-
- name: 50-insecure-registry.conf
content: |
[Service]
Environment=DOCKER_OPTS='--insecure-registry="10.0.1.0/24"'
But then, the master wouldn't restart. So I had to revert my change.
So many questions:
a. Why is there no docker.service block in this template cloud-config? How and when docker starts?
b. Do I need to edit the flannel_docker_opts.env file? Again there is no mention to such file in this cloud-config. But there is mention in this page:
https://coreos.com/flannel/docs/latest/flannel-config.html
Of particular interest at the end of that page:
ExecStartPost in flanneld.service converts information in /run/flannel/subnet.env into Docker daemon command line args (such as --bip and --mtu), storing them in /run/flannel_docker_opts.env
...
docker.service sources in /run/flannel_docker_opts.env which contains env variables with command line options and starts the Docker with them.
And in fact I can see the mentioned files like early-docker.service, but again no mention of flannel in the cloud-config.
But indeed I found the service files mentioned in the page above:
/usr/lib64/udev/rules.d/80-docker.rules
/usr/lib64/systemd/system/early-docker.service
/usr/lib64/systemd/system/early-docker.socket
/usr/lib64/systemd/system/docker.service
/usr/lib64/systemd/system/docker.socket
/usr/lib64/systemd/system/sockets.target.wants/docker.socket
/usr/lib64/systemd/system/early-docker.target
And indeed the /run/flannel_docker_opts.env file is mentioned in the docker.service file, but does not exist in the /run folder:
vi /usr/lib64/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=docker.socket early-docker.target network.target
Requires=docker.socket early-docker.target
[Service]
Environment=TMPDIR=/var/tmp
EnvironmentFile=-/run/flannel_docker_opts.env <<<<<<<<<< HERE!!!!!
MountFlags=slave
LimitNOFILE=1048576
LimitNPROC=1048576
ExecStart=/usr/lib/coreos/dockerd --daemon --host=fd:// $DOCKER_OPTS $DOCKER_OPT_BIP $DOCKER_OPT_MTU $DOCKER_OPT_IPMASQ
[Install]
c. Where does this docker.service file comes from! There is no mention to it in the cloud-config. Is it part of the CoreOS architecture?
d. If this docker.service is integral part of CoreOS, why are all the files in this /usr/lib64 path? CoreOS docs mentions other path locations for all the files.
Any suggestion would be appreciated. I'm going blind now. I will try and create to this non existent flannel_docker_conf.env file. But I'm not sure if what I'm doing is the correct way.
Thanks!