I have installed Mesosphere DCOS in AWS using the provided template. Now I would like to restart all the nodes but adding the --insecure-registry parameter to all the slave nodes (and master as well) so that they communicate with my docker registry. So I was reading the best way to do this is on the cloud-config script for the AWS template.
So in the AWS EC2 Launch Configurations I copied the configuration of the master node, then adjusted the User Data then updated the auto scaling groups and restarted the master. (awesome answer how to do this at the end of How do I use insecure docker registries with Amazon EC2 Container Service (ECS)?)
The lines were added to the end of the units section in cloud-config as suggested by CoreOS docs:
https://coreos.com/os/docs/latest/cloud-config.html
units
....Many lines here
- name: docker.service
drop-ins: |-
- name: 50-insecure-registry.conf
content: |
[Service]
Environment=DOCKER_OPTS='--insecure-registry="10.0.1.0/24"'
But then, the master wouldn't restart. So I had to revert my change.
So many questions:
a. Why is there no docker.service block in this template cloud-config? How and when docker starts?
b. Do I need to edit the flannel_docker_opts.env file? Again there is no mention to such file in this cloud-config. But there is mention in this page:
https://coreos.com/flannel/docs/latest/flannel-config.html
Of particular interest at the end of that page:
ExecStartPost in flanneld.service converts information in /run/flannel/subnet.env into Docker daemon command line args (such as --bip and --mtu), storing them in /run/flannel_docker_opts.env
...
docker.service sources in /run/flannel_docker_opts.env which contains env variables with command line options and starts the Docker with them.
And in fact I can see the mentioned files like early-docker.service, but again no mention of flannel in the cloud-config.
But indeed I found the service files mentioned in the page above:
/usr/lib64/udev/rules.d/80-docker.rules
/usr/lib64/systemd/system/early-docker.service
/usr/lib64/systemd/system/early-docker.socket
/usr/lib64/systemd/system/docker.service
/usr/lib64/systemd/system/docker.socket
/usr/lib64/systemd/system/sockets.target.wants/docker.socket
/usr/lib64/systemd/system/early-docker.target
And indeed the /run/flannel_docker_opts.env file is mentioned in the docker.service file, but does not exist in the /run folder:
vi /usr/lib64/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=docker.socket early-docker.target network.target
Requires=docker.socket early-docker.target
[Service]
Environment=TMPDIR=/var/tmp
EnvironmentFile=-/run/flannel_docker_opts.env <<<<<<<<<< HERE!!!!!
MountFlags=slave
LimitNOFILE=1048576
LimitNPROC=1048576
ExecStart=/usr/lib/coreos/dockerd --daemon --host=fd:// $DOCKER_OPTS $DOCKER_OPT_BIP $DOCKER_OPT_MTU $DOCKER_OPT_IPMASQ
[Install]
c. Where does this docker.service file comes from! There is no mention to it in the cloud-config. Is it part of the CoreOS architecture?
d. If this docker.service is integral part of CoreOS, why are all the files in this /usr/lib64 path? CoreOS docs mentions other path locations for all the files.
Any suggestion would be appreciated. I'm going blind now. I will try and create to this non existent flannel_docker_conf.env file. But I'm not sure if what I'm doing is the correct way.
Thanks!
Related
I am using flatcar ami as the base os for my AWS EC2 instances.
I added a journald.conf file under the folder /etc/systemd/ during the baking of the AWS image (I want to increase the journalctl retention).
I was expecting to see the new journald configurations read during the boot of the AWS instance. Checking with the command systemctl status systemd-journald it didn't happen.
What I tried:
Using a journald.conf under the /etc/systemd/journald.conf.d/ path;
Restarting the systemd-journald service after the boot with a systemd service.
Important notes:
If I run systemctl restart systemd-journald inside the instance, the configurations are updated;
I did a cat in ExecStartPre before restarting the systemd-journald using an auxiliary service and confirmed the contents. Yet it doesn't pick up the configuration
The file is present, right and under the right path during the boot.
I'm trying to launch Neo4J graph database on AWS using their AIM image (enteprise 3.3.9)
However, the server fails to launch the instance automatically how it's supposed to.
When I try to relaunch it using
systemctl restart neo4j
It also fails.
When I do
systemctl cat neo4j
I find the /etc/neo4j/pre-neo4j.sh file, which is apparently launched on the instance's startup, which, in turn launches Neo4J (when it's supposed to work):
[Unit]
Description=Neo4j Graph Database
After=network-online.target
Wants=network-online.target
[Service]
ExecStart=/etc/neo4j/pre-neo4j.sh
Restart=on-failure
User=neo4j
Group=neo4j
Environment="NEO4J_CONF=/etc/neo4j" "NEO4J_HOME=/var/lib/neo4j"
LimitNOFILE=60000
TimeoutSec=120
SuccessExitStatus=143
[Install]
WantedBy=multi-user.target
So then I launch it manually via the bash script using the sudo prefix and then it starts up fine.
sudo /etc/neo4j/pre-neo4j.sh
The documentation on deploying Neo4J on an AWS server doesn't mention anything about permissions if you use their image. So what can be the problem?
I don't want to have manually launch the DB using the sudo — is it possible to resolve this problem by modifying the bash script itself?
..
The file /etc/neo4j/pre-neo4j.sh sets some environmental parameters and then launches neo4j via:
/usr/share/neo4j/bin/neo4j console
Based on the comments.
The solution was to use
journalctl -u neo4j
to inspect the logs associated with the failed start of neo4j. This enabled to identify the root cause, and subsequently, to fix the issue.
I'm using Ubuntu 18.04.4 LTS which uses systemd, but the squid package packaged with this version of Ubuntu is configured to start via init.d. It starts and runs via systemctl start squid.service if I start it manually after the system has booted.
However, I'm using a ZFS mount point ("/media") to store the cache data, and during the boot process squid is starting before this mount point is active. Consequently I'm getting the error "Failed to verify one of the swap directories". Full output of systemctl status squid is here
I'd like to tell systemd to wait until after media.mount has completed in the most minimally invasive way possible (e.g. without modifying the /etc/init.d/squid file that is maintained by the package). To that end I created the /etc/systemd/system/squid.service.d/override.conf file like so:
% cat /etc/systemd/system/squid.service.d/override.conf
[Unit]
Wants=network.target network-online.target nss-lookup.target media.mount
After=network.target network-online.target nss-lookup.target media.mount
[Install]
WantedBy=multi-user.target
But squid is still starting too early.
Is what I want to do possible? Or do I have to bite the bullet and define a native /etc/systemd/system/squid.service file and remove the /etc/init.d/squid init script?
I've created a cloud-init file using Terraform's aws_instance.user_data parameter.
This then is executed on start up, on Centos machines, by the cloud-init systemd service.
I would like to edit this file for dev/testing purposes on the fly then simply restart the said service.
To this end, where can I find the cloud-init file that contains the commands run by the cloud-init service?
On my Centos machine, this was found in the file:
/var/lib/cloud/instances/<instance-id>/user-data.txt
Where instance-id is found in the file /var/lib/cloud/data/instance-id.
I'm very new to kubernetes and trying to conceptualize it as well as set it up locally in order to try developing something on it.
There's a confound though that I am running on a windows machine.
Their "getting started" documentation in github says you have to run Linux to use kubernetes.
As docker runs on windows, I was wondering if it was possible to create a kubernetes instance as a container in windows docker and use it to manage the rest of the cluster in the same windows docker instance.
From reading the setup instructions, it seems like docker, kubernetes, and something called etcd all have to run "in parallel" on a single host operating system... But part of me thinks it might be possible to
Start docker, boot 'default' machine.
Create kubernetes container - configure to communicate with the existing docker 'default' machine
Use kubernetes to manage existing docker.
Pipe dream? Wrongheaded foolishness? I see there are some options around running it in a vagrant instance. Does that mean docker, etcd, & kubernetes together in a single VM (which in turn creates a cluster of virtual machines inside it?)
I feel like I need to draw a picture of what this all looks like in terms of physical hardware and "memory boxes" to really wrap my head around this.
With Windows, you need docker-machine and boot2docker VMs to run anything docker related.
There is no (not yet) "docker for Windows".
Note that issue 7428 mentioned "Can't run kubernetes within boot2docker".
So even when you follow instructions (from a default VM created with docker-machine), you might still get errors:
➜ workspace docker run --net=host -d -v /var/run/docker.sock:/var/run/docker.sock gcr.io/google_containers/hyperkube:v0.14.2 /hyperkube kubelet --api_servers=http://localhost:8080 --v=2 --address=0.0.0.0 --enable_server --hostname_override=127.0.0.1 --config=/etc/kubernetes/manifests
ee0b490f74f6bc9b70c1336115487b38d124bdcebf09b248cec91832e0e9af1d
➜ workspace docker logs -f ee0b490f74f6bc9b70c1336115487b38d124bdcebf09b248cec91832e0e9af1d
W0428 09:09:41.479862 1 server.go:249] Could not load kubernetes auth path: stat : no such file or directory. Continuing with defaults.
I0428 09:09:41.479989 1 server.go:168] Using root directory: /var/lib/kubelet
The alternative would be to try on a full-fledge Linux VM (like the latest Ubuntu), instead of a boot2docker-like VM (based on a TinyCore distro).
All k8s components can be raised up with hyperkube, which helps you bring up a containerized one.
If you're able to run docker on windows, it would probably work. I haven't tried it on windows personally.