I am starting an EC2 test cluster of 3-minions of type 'm3.large'
want to use 32G SSD storage available on these machine for a pod running on it
want to use using 'hostPath' to mount some portion of this storage to pod so that it is available for the life of minions
However the issue is majority of this storage is allocated for "/mnt/ephemeral/docker" not available on host for 'hostPath' use. On minion it looks like
$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 3.7G 0 3.7G 0% /dev
tmpfs 748M 75M 674M 10% /run
/dev/xvda1 7.8G 2.0G 5.5G 27% /
tmpfs 3.7G 828K 3.7G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.7G 0 3.7G 0% /sys/fs/cgroup
/dev/mapper/vg--ephemeral-docker 30G 5.0G 23G 18% /mnt/ephemeral/docker
/dev/mapper/vg--ephemeral-kubernetes 1.4G 1.4G 0 100% /mnt/ephemeral/kubernetes
tmpfs 748M 0 748M 0% /run/user/1000
Here 30G is allocated to "/mnt/ephemeral/docker" and is only available in docker container as a transient storage. I understand the EBS volumes for persistent store, however the SSD/ephemeral storage is persistent for the life of the EC2 instance and can be used for needs like data store for a Cassandra pod running in Kubernetes
The main issue is not being able to use the instance storage from within pods as instance persistence storage.
Configuring 'emptyDir' in pod uses 1.4G from '/mnt/ephemeral/kubernetes' and fills quickly
volumes:
- name: data
- emptyDir: {}
Configuring 'hostPath' in pod uses 7.8G from '/' and fills quickly
volumes:
- name: data
- hostPath:
path: /cassandra_data
Major chunk of instance storage 30G is allocated to '/mnt/ephemeral/docker' and is not avilable for Kubernates pods as instance persistent storage. What is the best way to use (Kubernetes volumes) the SSD storage that comes built-in with the EC2 instances.
There's discussion on
https://github.com/kubernetes/kubernetes/issues/15055
so I'm going to mark this answered.
Related
I have been trying to install elasticsearch without success. Every single guide regardless of if I used the .deb or install from apt fails in the same way. I am using ubuntu 20.04 as an LXC on proxmox. After installing the service start fails and here is the log file
[2022-06-10T12:06:13.884+0000][22320][gc,init] CardTable entry size: 512
[2022-06-10T12:06:13.885+0000][22320][gc ] Using G1
[2022-06-10T12:06:13.957+0000][22320][gc,init] Version: 18.0.1.1+2-6 (release)
[2022-06-10T12:06:13.957+0000][22320][gc,init] CPUs: 24 total, 4 available
[2022-06-10T12:06:13.957+0000][22320][gc,init] Memory: 96593M
[2022-06-10T12:06:13.957+0000][22320][gc,init] Large Page Support: Disabled
[2022-06-10T12:06:13.957+0000][22320][gc,init] NUMA Support: Disabled
[2022-06-10T12:06:13.958+0000][22320][gc,init] Compressed Oops: Enabled (Non-zero disjoint base)
[2022-06-10T12:06:13.958+0000][22320][gc,init] Heap Region Size: 16M
[2022-06-10T12:06:13.958+0000][22320][gc,init] Heap Min Capacity: 31G
[2022-06-10T12:06:13.958+0000][22320][gc,init] Heap Initial Capacity: 31G
[2022-06-10T12:06:13.958+0000][22320][gc,init] Heap Max Capacity: 31G
[2022-06-10T12:06:13.958+0000][22320][gc,init] Pre-touch: Disabled
[2022-06-10T12:06:13.958+0000][22320][gc,init] Parallel Workers: 4
[2022-06-10T12:06:13.958+0000][22320][gc,init] Concurrent Workers: 1
[2022-06-10T12:06:13.958+0000][22320][gc,init] Concurrent Refinement Workers: 4
[2022-06-10T12:06:13.958+0000][22320][gc,init] Periodic GC: Disabled
[2022-06-10T12:06:13.958+0000][22320][gc,metaspace] CDS archive(s) not mapped
[2022-06-10T12:06:13.958+0000][22320][gc,metaspace] Compressed class space mapped at: 0x0000000080000000-0x00000000c0000000, reserved size: 1073741>
[2022-06-10T12:06:13.958+0000][22320][gc,metaspace] Narrow klass base: 0x0000000000000000, Narrow klass shift: 0, Narrow klass range: 0xc0000000
[2022-06-10T12:06:14.152+0000][22320][gc,heap,exit] Heap
[2022-06-10T12:06:14.152+0000][22320][gc,heap,exit] garbage-first heap total 32505856K, used 24578K [0x0000001001000000, 0x00000017c1000000)
[2022-06-10T12:06:14.152+0000][22320][gc,heap,exit] region size 16384K, 2 young (32768K), 0 survivors (0K)
[2022-06-10T12:06:14.152+0000][22320][gc,heap,exit] Metaspace used 3525K, committed 3584K, reserved 1114112K
[2022-06-10T12:06:14.152+0000][22320][gc,heap,exit] class space used 271K, committed 320K, reserved 1048576K
So pretty much all online guides to install elastic search are useless except this one:
https://techviewleo.com/install-elastic-stack-elk-8-on-ubuntu/
For anyone attempting to install elastic search and comes here the main thing this above guide does that the other ones dont is it disables all the security in the yaml file:
Here are the settings that worked for me
network.host: localhost
cluster.name: my-application
node.name: node-1
# Enable security features
xpack.security.enabled: false
xpack.security.enrollment.enabled: false
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
enabled: false
keystore.path: certs/http.p12
# Enable security features
xpack.security.enabled: false
xpack.security.enrollment.enabled: false
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
enabled: false
keystore.path: certs/http.p12
As warning this config should not be used for a publicly exposed instance
I have set up a band new cluster using ecs-cli on AWS with following commands:
Configure cluster : ecs-cli configure --cluster cluster_name --region region_name --default-launch-type EC2 --config-name config_name
Use default profile : ecs-cli configure default --config-name config_name
Create Cluster : ecs-cli up --keypair key_name --capability-iam --size 1 --instance-type t2.micro --security-group sg_id --vpc vpc_id --subnets subnet_id --cluster-config config_name
The cluster was created successfully on ECS. But when I am trying to run my docker-compose file to run jenkins and jenkins data volume containers (already pushed to ECR) I am getting "RESOURCE:MEMORY" error even though the CPU and memory utilisation is 0%.
Deploy docker compose file to cluster : ecs-cli compose up --cluster-config config_id
Actual Result:
WARN[0000] Skipping unsupported YAML option for service... option name=networks service name=jenkins
WARN[0000] Skipping unsupported YAML option for service... option name=networks service name=jenkins_dv
INFO[0000] Using ECS task definition TaskDefinition="aws-infra:4"
INFO[0000] Couldn't run containers reason="RESOURCE:MEMORY"
jenkins:
image: jenkins:latest
cpu_shares: 50
mem_limit: 524288000
ports: ["8080:8080", "50000:50000"]
volumes_from: ['jenkins_dv']
jenkins_dv:
image: jenkins_dv:latest
cpu_shares: 50
mem_limit: 524288000
Even when I am running the docker compose file after deleting cpu_shares and mem_limits (as it is not required for EC2 instances), I am getting same error. Since the cluster is new and does not have any CPU or memory being utilised the tasks should be created successfully. What am I doing wrong here?
I have got the solution to this issue. I have allocated memory limit as 500MB (in bytes) to both the containers. As per AWS documentation tc.micro has 1GB memory but if you open your instance (Cluster> EC2 Instance > container instance) and view the memory allocation, the actual memory allocated is slightly less than 1GB. I updated my file and gave memory limit as 250MB (in bytes) to both the containers and it worked.
I have tried all options like
Mount /dev/xvdf /vol -t xfs
Mount /dev/xvdf1 /vol -t xfs
Mount /dev/xvdf2 /vol -t xfs
Same repeated with ext4
But I keep on getting the error :
Wrong FS type
This was a volume attached to another ec2 instance and I detached it then attached to new instance. It was boot volume.
When you attach an EBS volume to a different EC2 instance, it may not have the same device name as on the previous instance. Make sure you are trying to mount the proper device, use 'lsblk' to determine the block device name.
More details are documented in the AWS EC2 User Guide:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html
I run hadoop cluster in Docker by mount a local folder by -v.
Then I login the hadoop cluster and 'cd' to the mount folder and execute hdfs dfs -put ./data/* input/. It works.
But my problem is that I cannot delete the data that I copied to hdfs. I delete containers by docker rm ,but the data still exist. Now I only can reset Docker and the data can be deleted.
Is there any other solution?
This is my docker info
➜ hadoop docker info
Containers: 5
Running: 5
Paused: 0
Stopped: 0
Images: 1
Server Version: 1.12.3
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 22
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null bridge host overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.27-moby
Operating System: Alpine Linux v3.4
OSType: linux
Architecture: x86_64
CPUs: 5
Total Memory: 11.71 GiB
Name: moby
ID: NPR6:2ZTU:CREI:BHWE:4TQI:KFAC:TZ4P:S5GM:5XUZ:OKBH:NR5C:NI4T
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 56
Goroutines: 81
System Time: 2016-11-22T08:10:37.120826598Z
EventsListeners: 2
Username: chaaaa
Registry: https://index.docker.io/v1/
WARNING: No kernel memory limit support
Insecure Registries:
127.0.0.0/8
This is an issue. https://github.com/docker/for-mac/issues/371
If you can remove all images/containers then:
Stop Docker.
run
docker rm $(docker ps -a -q)
docker rmi $(docker images -q)
docker volume rm $(docker volume ls |awk '{print $2}')
rm -rf ~/Library/Containers/com.docker.docker/Data/*
Start Docker, you have yours GB back.
To delete data in a HDFS you need to make a similar call like the one you did to put the file, in this case:
hdfs dfs -rm ./data/*
If there are directories you should add -r
hdfs dfs -rm -R ./data/*
And finally, by default Hadoop move deleted files/directories to a trash directory, which would be in the home of the hadoop user you're using for this requests, something like /user/<you>/.Trash/
About HDFS
Usually in the namenode there is some metadata about the structure of the HDFS, like the directories or files in it and where are the blocks forming it stored (Which datanodes). While datanodes will keep blocks of HDFS data, the data stored usually is not usable, since it will usually be just part of the data blocks in the HDFS.
Because of this, all operations with the HDFS are done through the namenode using hdfs calls, like put, get, rm, mkdir... instead of regular operating system command line tools.
I've been reading a lot about this issue in here and other websites, but I haven't manage to find a proper solution on how to increase the images size limit which is set to 10GB by default.
A bit of background informations.
I'm building a docker container:
https://bitbucket.org/efestolab/docker-buildgaffer
Which download and builds a consistent set of libraries on top of a centos image. (takes a horrible amount of time and space to build)
The problem is that every single time I try to build it I hit this error :
No space left on device
Docker version:
Docker version 1.7.1, build 786b29d
Docker Info :
Containers: 1
Images: 76
Storage Driver: devicemapper
Pool Name: docker-8:7-12845059-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 11.28 GB
Data Space Total: 107.4 GB
Data Space Available: 96.1 GB
Metadata Space Used: 10.51 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.137 GB
Udev Sync Supported: false
Deferred Removal Enabled: false
Data loop file: /home/_varlibdockerfiles/devicemapper/devicemapper/data
Metadata loop file: /home/_varlibdockerfiles/devicemapper/devicemapper/metadata
Library Version: 1.02.82-git (2013-10-04)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.15.9-031509-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 8
Total Memory: 15.58 GiB
Name: hdd-XPS-15-9530
ID: 2MEF:IYLS:MCN5:AR5O:6IXJ:3OB3:DGJE:ZC4N:YWFD:7AAB:EQ73:LKXQ
Username: efesto
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
After stopping the service and nuking the /var/lib/docker folder,
I've been updating by docker startup script
/lib/systemd/system/docker.service
with these flags :
ExecStart=/usr/bin/docker -d --storage-opt dm.basesize=20G --storage-opt dm.loopdatasize=256G -H fd:// $DOCKER_OPTS
and restarted docker service, but still fails with the same error.
I've also been reading that might be due to the original image I'm rely on (centos:6), which might have been built with 10GB limit.
So I rebuild the centos6 image, and used that as base for mine, but I did hit the same error.
Does anyone have a reliable way to make me able to build this docker image fully ?
If there's any other informations which might be useful, just feel free to ask.
Thanks for any reply or suggestions !
L.
Found this article
Basically edit /etc/docker/daemon.json file to include
"storage-opts": [
"dm.basesize=40G"
]
Restart the docker service, and it will enable to create/import images larger than 10Gb
Thanks to the test of #user2915097, I've been updating kernel version 3.16.0, installed the kernel extras, and removed and re installed docker.
the problem seems to be addressable to devicemapper, now without any change in the docker command I get:
Containers: 0
Images: 94
Storage Driver: aufs
Root Dir: /home/_varlibdockerfiles/aufs
Backing Filesystem: extfs
Dirs: 94
Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.16.0-45-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 8
Total Memory: 15.58 GiB
Name: hdd-XPS-15-9530
ID: 2MEF:IYLS:MCN5:AR5O:6IXJ:3OB3:DGJE:ZC4N:YWFD:7AAB:EQ73:LKXQ
Username: efesto
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
and it finally builds images > 10GB.
L.
Since this question has been asked, the storage driver here:
Storage Driver: devicemapper
is no longer used by default, and not recommended. That also means the settings for the 10GB limit no longer apply.
The overlay2 storage driver (currently enabled by default) does not have size limits of it's own. Instead, the underlying filesystem you use for /var/lib/docker is used for any available free space and inodes there. You can check that free space with:
df -h /var/lib/docker
df -ih /var/lib/docker
after modifing the docker daemon startup parameters do the following
systemctl daemon-reload
systemctl stop docker
rm -rf /var/lib/docker/*
systemctl start docker
This will remove all your images, make sure you save them before
eg docker save -o something.tar.gz image_name
and reload them after starting docker
eg docker load -i something.tar.gz