I'm trying to run ElasticSearch in Azure Container Instances. I've created the container like this using the Azure CLI:
az container create --image elasticsearch:7.4.2 --name $containerGroupName -g $resourceGroup --ip-address public --dns-name-label <mydns> --memory 8 --cpu 2 --ports 9200
The container ends up in a waiting state. When I check out the logs in the Azure portal, I'll see the following error:
ERROR: [2] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
Apparently, this has something to do with virtual memory: https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html
But how do I change it inside the container? I cannot connect to it since the container is not running.
Very late to the party but hopefully this will help someone in the future. Elasticsearch if deployed stand-alone requires the environment variable discovery.type to be set. Since Azure does not allow vars with a . (dot) at the moment you'll have to create a custom Docker image and host this e. g. with Azure Container Registry or Dockerhub, ...
Content can be as little as this:
FROM elasticsearch:7.8.1
ENV discovery.type=single-node
I realize that ACI is not the best option to run ElasticSearch. I only need it temporarily for a proof-of-concept while I'm waiting for a stable environment of ElasticSearch elsewhere.
Eventually I got it running by picking an older image of ElasticSearch:
az container create --image elasticsearch:5.6.14-alpine --name $containerGroupName -g $resourceGroup --ip-address public --dns-name-label <mydns> --memory 4 --cpu 2 --ports 9200
The solution to this is simply adding this line to the bound elasticsearch.yml config file:
discovery.type: single-node
For this to be permanent even when you restart the container, the config directory of the elasticsearch container needs to be bound to an Azure Fileshare directory, where you make permanent changes there.
Create a Azure file share and create a elasticsearch folder with another config folder inside it: https://docs.microsoft.com/en-us/azure/storage/files/storage-how-to-create-file-share
Deploy a elasticsearch container with the fileshare volume mounted and bind /mnt/elasticsearch/config to a folder you've created on the new fileshare (template tags: "mountPath": "/mnt/elasticsearch/config", "shareName": "elasticsearch/config"): https://docs.microsoft.com/en-us/azure/container-instances/container-instances-volume-azure-files
While the container doesn't error out and shuts itself down because of this max virtual memory error, copy the entire /usr/share/elasticsearch/config folder and filers to the fileshare folder. As soon as you get terminal access, run cp -R /usr/share/elasticsearch/config /mnt/elasticsearch/config
On your file share folder, you should now have a config directory with files and other folders created by the elastisearch startup process. One of this files is elasticsearch.yml. Open it, add the line discovery.type: single-node and save it.
Finally, change mounting and binding location to the correct location, so elasticsearch starts up and reads our now modified configurations. Bind /usr/share/elasticsearch/config to the fileshare /elasticsearch/config folder (template tags: "mountPath": "/usr/share/elasticsearch/config", "shareName": "elasticsearch/config"). More info here: https://www.elastic.co/guide/en/elasticsearch/reference/master/docker.html#docker-configuration-methods
Start the container with the new mounting locations and now the max virtual memory error should be gone.
Please, only use this trick for proofs-of-concept or very low volume logs. Using network storage like Azure Fileshare in general is not suitable for Elasticsearch and can even crash it.
Related
In my k8s environment where spring-boot applications runs, I checked log location in /var/log and /var/lib but both empty. Then I found log location in /tmp/spring.log . It seems this the default log location. My problem are
How kubectl log knows it should read logs from /tmp location. I get log output on kubectl logs command.
I have fluent-bit configured where it has input as
following
[INPUT]
Name tail
Tag kube.dev.*
Path /var/log/containers/*dev*.log
DB /var/log/flb_kube_dev.db
This suggest it should reads logs from /var/log/containers/ but it does not have logs. However i am getting fluent-bit logs successfully. What am i missing here ?
Docker logs only contain the logs that are dumped on STDOUT by your container's process with PID 1 (your container's entrypoint or cmd process).
If you want to see the logs via kubectl logs or docker logs, you should redirect your application logs to STDOUT instead of file /tmp/spring.log. Here's an excellent example of how this can achieved with minimal effort.
Alternatively, you can also use hostPath volumeMount. This way, you can directly access the log from the path on the host.
Warning when using hostPath volumeMount
If the pod is shifted to another host due to some reason, you logs will not move along with it. A new log file will be created on this new host at the same path.
If you are searching for the actual location of the logs outside the containers (and on the host nodes of the cluster), this depends on a couple things. I suppose you are using Docker to run your containers under Kubernetes, which is the most common setup.
On each node of your Kubernetes cluster, you can use the following command to check what is the logging driver being currently used:
docker info | grep -i logging
The default value should be json-file, which means that logs are being written as jsons from the containers, to a certain location on your host nodes.
If you find another driver, such as for example journald, then that means Docker logging driver is sending logs directly to the systemd journal. There are many logging drivers, so as a first check, you should be sure that all yours Kubernetes nodes are configured to log as json files (or, in the way you need to harvest them).
Once this is done, you can start checking where your containers are logging their own log. Choose a Pod to analyze, then:
Identify on which Kubernetes node it is running on
kubectl get pod pod-name -owide
Grab the container ID with something like the following
kubectl get pod pod-name -ojsonpath='{.status.containerStatuses[0].containerID}'
Where the id should be something in the shape of docker://f834508490bd2b248a2bbc1efc4c395d0b8086aac4b6ff03b3cc8fd16d10ce2c
Remove the docker:// part and SSH on the Kubernetes node on which this container is running, then do a
docker inspect container-id | grep -i logpath
Which should give you the log locations for that particular container. You can try tail on the file to check if the logs are really there or not.
In my case, the container I tried this procedure on, was logging inside:
/var/lib/docker/containers/289271086d977dc4e2e0b80cc28a7a6aca32c888b7ea5e1b5f24b28f7601ff63/289271086d977dc4e2e0b80cc28a7a6aca32c888b7ea5e1b5f24b28f7601ff63-json.log
I have Windows 10 OS, docker-compose and want to work with Apache Flink tutorial Playground, docker-compose starting correctly starting docker-compose but after several minutes of work,
Apache Flink has to create checkpoints, but there is some problem with access to the file system.
Exception:
org.apache.flink.runtime.checkpoint.CheckpointException: Could not finalize the pending checkpoint 104. Failure reason: Failure to finalize checkpoint.
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:1216) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
…..
Caused by: org.apache.flink.util.SerializedThrowable: Mkdirs failed to create file:/tmp/flink-checkpoints-directory/d73c2f87b0d7ea6748a1913ee4b50afe/chk-104
at org.apache.flink.core.fs.local.LocalFileSystem.create(LocalFileSystem.java:262) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
Could you please help me with the correct path and docker access?
state.backend: filesystem
state.checkpoints.dir: file:///tmp/flink-checkpoints-directory
state.savepoints.dir: file:///tmp/flink-savepoints-directory
Also I tried use full Windows path but got the same error.
Are you using Windows Docker containers or Linux Docker containers ?
Right-click on the Docker Desktop icon to see your current configuration.
Switch to Windows containers
OR
Switch to Linux containers
You have to configure Flink paths according to you target Docker container type.
Note: You cannot use Windows and Linux containers at the same time.
When I create python buy dashboard and go to CLI in jeastic and run lsblk
I can see that the / is mount. is python container or CentOS?
If it is container how can they mount /?because in kubernetes we can not mount /?
If it is CentOS why I do not have full permission?!
Jelastic is full on containers, no VMs are used there for now. In container it is possible to mount / if you have enough privileges for that, that works for both Kubernetes and Jelastic. Python image is also working inside container, by default you are connecting to SSH over jelastic user that has limited permissions that is why you cannot mount /. If you contact your hosting service provider I believe they will be able to provide you root access.
I'm very new to kubernetes and trying to conceptualize it as well as set it up locally in order to try developing something on it.
There's a confound though that I am running on a windows machine.
Their "getting started" documentation in github says you have to run Linux to use kubernetes.
As docker runs on windows, I was wondering if it was possible to create a kubernetes instance as a container in windows docker and use it to manage the rest of the cluster in the same windows docker instance.
From reading the setup instructions, it seems like docker, kubernetes, and something called etcd all have to run "in parallel" on a single host operating system... But part of me thinks it might be possible to
Start docker, boot 'default' machine.
Create kubernetes container - configure to communicate with the existing docker 'default' machine
Use kubernetes to manage existing docker.
Pipe dream? Wrongheaded foolishness? I see there are some options around running it in a vagrant instance. Does that mean docker, etcd, & kubernetes together in a single VM (which in turn creates a cluster of virtual machines inside it?)
I feel like I need to draw a picture of what this all looks like in terms of physical hardware and "memory boxes" to really wrap my head around this.
With Windows, you need docker-machine and boot2docker VMs to run anything docker related.
There is no (not yet) "docker for Windows".
Note that issue 7428 mentioned "Can't run kubernetes within boot2docker".
So even when you follow instructions (from a default VM created with docker-machine), you might still get errors:
➜ workspace docker run --net=host -d -v /var/run/docker.sock:/var/run/docker.sock gcr.io/google_containers/hyperkube:v0.14.2 /hyperkube kubelet --api_servers=http://localhost:8080 --v=2 --address=0.0.0.0 --enable_server --hostname_override=127.0.0.1 --config=/etc/kubernetes/manifests
ee0b490f74f6bc9b70c1336115487b38d124bdcebf09b248cec91832e0e9af1d
➜ workspace docker logs -f ee0b490f74f6bc9b70c1336115487b38d124bdcebf09b248cec91832e0e9af1d
W0428 09:09:41.479862 1 server.go:249] Could not load kubernetes auth path: stat : no such file or directory. Continuing with defaults.
I0428 09:09:41.479989 1 server.go:168] Using root directory: /var/lib/kubelet
The alternative would be to try on a full-fledge Linux VM (like the latest Ubuntu), instead of a boot2docker-like VM (based on a TinyCore distro).
All k8s components can be raised up with hyperkube, which helps you bring up a containerized one.
If you're able to run docker on windows, it would probably work. I haven't tried it on windows personally.
I have installed Mesosphere DCOS in AWS using the provided template. Now I would like to restart all the nodes but adding the --insecure-registry parameter to all the slave nodes (and master as well) so that they communicate with my docker registry. So I was reading the best way to do this is on the cloud-config script for the AWS template.
So in the AWS EC2 Launch Configurations I copied the configuration of the master node, then adjusted the User Data then updated the auto scaling groups and restarted the master. (awesome answer how to do this at the end of How do I use insecure docker registries with Amazon EC2 Container Service (ECS)?)
The lines were added to the end of the units section in cloud-config as suggested by CoreOS docs:
https://coreos.com/os/docs/latest/cloud-config.html
units
....Many lines here
- name: docker.service
drop-ins: |-
- name: 50-insecure-registry.conf
content: |
[Service]
Environment=DOCKER_OPTS='--insecure-registry="10.0.1.0/24"'
But then, the master wouldn't restart. So I had to revert my change.
So many questions:
a. Why is there no docker.service block in this template cloud-config? How and when docker starts?
b. Do I need to edit the flannel_docker_opts.env file? Again there is no mention to such file in this cloud-config. But there is mention in this page:
https://coreos.com/flannel/docs/latest/flannel-config.html
Of particular interest at the end of that page:
ExecStartPost in flanneld.service converts information in /run/flannel/subnet.env into Docker daemon command line args (such as --bip and --mtu), storing them in /run/flannel_docker_opts.env
...
docker.service sources in /run/flannel_docker_opts.env which contains env variables with command line options and starts the Docker with them.
And in fact I can see the mentioned files like early-docker.service, but again no mention of flannel in the cloud-config.
But indeed I found the service files mentioned in the page above:
/usr/lib64/udev/rules.d/80-docker.rules
/usr/lib64/systemd/system/early-docker.service
/usr/lib64/systemd/system/early-docker.socket
/usr/lib64/systemd/system/docker.service
/usr/lib64/systemd/system/docker.socket
/usr/lib64/systemd/system/sockets.target.wants/docker.socket
/usr/lib64/systemd/system/early-docker.target
And indeed the /run/flannel_docker_opts.env file is mentioned in the docker.service file, but does not exist in the /run folder:
vi /usr/lib64/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=docker.socket early-docker.target network.target
Requires=docker.socket early-docker.target
[Service]
Environment=TMPDIR=/var/tmp
EnvironmentFile=-/run/flannel_docker_opts.env <<<<<<<<<< HERE!!!!!
MountFlags=slave
LimitNOFILE=1048576
LimitNPROC=1048576
ExecStart=/usr/lib/coreos/dockerd --daemon --host=fd:// $DOCKER_OPTS $DOCKER_OPT_BIP $DOCKER_OPT_MTU $DOCKER_OPT_IPMASQ
[Install]
c. Where does this docker.service file comes from! There is no mention to it in the cloud-config. Is it part of the CoreOS architecture?
d. If this docker.service is integral part of CoreOS, why are all the files in this /usr/lib64 path? CoreOS docs mentions other path locations for all the files.
Any suggestion would be appreciated. I'm going blind now. I will try and create to this non existent flannel_docker_conf.env file. But I'm not sure if what I'm doing is the correct way.
Thanks!