I'm facing a really strange issue with my cluster.
Whenever I'm trying to load any file into HDFS that is larger than 100 MB(104857600 bytes) it fails with the following error:
All datanodes are bad... Aborting.
This is really strange as 100 MB has become the threshold for filesize.
Even if i try to increase the file size by 1 single byte (104857601 bytes), and try to load it in HDFS, it fails with a long stacktrace. Principally saying "All datanodes are bad... Aborting"
Has anybody faced similar situation earlier?
Is it possible that there's some configuration change by mistake which has led to this behaviour? If yes, any configuration that limits the size of data that can be ingested that i can change?
Thanks
"Has anybody faced similar situation earlier?"
Yes I had. You should decrease the limit for the user who runs hadoop.
I installed hadoop on a linux box downloading it from apache website, and my system was not tuned for it, I got your message. These are the settings from cloudera quick start, compare your limit with these.
[cloudera#quickstart ~]$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 30494
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
If the you are able to get more details of the data node from the stack trace, you should be able to stop the datanode which would be one the daemons.
By killing or stopping the datanode you would get the data node which is likely to be corrupted, re-instantiated. Basically you are restarting jvm of that particular datanode.
Command:
To stop: bin/hadoop-daemon.sh stop datanode
To start: bin/hadoop-daemon.sh start datanode
Related
The error appears in /var/log/syslog while creating new OSDs to a Ceph cluster. The symptoms are silent and ansible shows the following error:
monclient: authenticate NOTE: no keyring found; disabled cephx
authentication", "librados: client.admin authentication error (95)
Operation not supported", " stderr: [errno 95] error connecting to the cluster"
The solution is with fixing the resource limit on the host.
Find out the requested limits by the service.
$ vim /lib/systemd/system/ceph-osd#.service
Check out the limits
[Service]
LimitNOFILE=1048576
LimitNPROC=1048576
Check the max limits of your system.
# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 148786
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 145383
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 148786
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Set the limits expected by the host.
[Service]
LimitNOFILE=145383
LimitNPROC=148786
Reload systemctl daemon
$ sudo systemctl daemon-reload
Restart your service
Recently we have seen some shared memory read/write performance issue with our application.
In our application, we copy data from shared memory /run/shm to pinned memory, so as to achieve higher GPU transfer throughput. We observed that the copy speed is so different when copying from different files (same size).
To reproduce the issue, we copied the same source file to two files (i.e. file1, file2) in shared memory. Then we confirmed the problem by counting the time of copying file to /dev/null, and the results are as below.
numactl --cpubind=0 cp /run/shm/file1 /dev/null 0.05s user 0.28s system 94% cpu 0.355 total
numactl --cpubind=1 cp /run/shm/file1 /dev/null 0.11s user 0.22s system 91% cpu 0.360 total
numactl --cpubind=0 cp /run/shm/file2 /dev/null 0.08s user 2.15s system 93% cpu 2.383 total
numactl --cpubind=1 cp /run/shm/file2 /dev/null 0.16s user 5.44s system 97% cpu 5.748 total
We have checked the meminfo, we only used 50% of memory. Nothing was swapped out (and we explicitly disabled it).
Searched around but could not identify the issue. Finally, we rebooted the system and the issue was gone.
Does anyone know what might be the root cause?
Thanks!
I have some problems. When I run the command start-dfs.sh, I get this error:
master: Error: Could not find or load main class master.log”.
The hadoop version is 2.7.2 and 2.6.4.
OS is centos 7.
Error: Could not find or load main class master.log
ulimit -a for user hadoop
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 14631
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4096
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
I had faced a similar issue and would like to share the thing that
worked for me. There seems to be an issue related to the hostname. It
had been changed due to some reason and I just cross-checked
hostname(/etc/hostname) file, reverted it to the earlier name and
just restarted my machine with 'init 6'(reboot command didn't work
for me) and the issue was resolved. Hope this may help youin some
way.
is there any way to set resource limits for each job that gnu parallel is executing?
I am interested in a memory limit or a time limit.
Look at --timeout (especially --timeout 1000% seems useful). For memory limit try:
ulimit -m 1000000
ulimit -v 1000000
to limit to 1 GB.
See also niceload and
https://unix.stackexchange.com/questions/173762/gnu-parallel-limit-memory-usage/173860?noredirect=1#comment287065_173860
I am new to Hadoop, I config cluster follow the instruction
After config, I start HDFS daemons with /bin/start-dfs.sh
I check log file /home/deploy/hadoop/libexec/../logs/hadoop-deploy-datanode-slave1.out to make sure is run, but I see only text as below:
ulimit -a for user deploy
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 63524
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 16384
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Hope anyone can help?
#Minh, you are looking at /home/deploy/hadoop/libexec/../logs/hadoop-deploy-datanode-slave1.out, instead see /home/deploy/hadoop/libexec/../logs/hadoop-deploy-datanode-slave1.log.
Similarly there would be other log files in /home/deploy/hadoop/libexec/../logs/ folder.
Let me explain more about .log and .out files.
Some of the files in logs folder end with .log, and others end with .out.
The .out files are only written to when daemons are starting.
After daemons have started successfully, .out files are truncated.
By contrasts, all log messages can be found in the .log files, including the daemon start-up messages that are sent to the .out files.