How to get my hdfs docker client running? - hadoop

Question

I'm starting an hdfs server with:
docker run -d sequenceiq/hadoop-docker:2.6.0
I'm observing the running docker processes with
docker ps
Which gets the following result:
6bfa4f2fd3b5 sequenceiq/hadoop-docker:2.6.0 "/etc/bootstrap.sh -d"
31 minutes ago Up 31 minutes 22/tcp, 8030-8033/tcp, 8040/tcp,
8042/tcp, 8088/tcp, 49707/tcp, 50010/tcp, 50020/tcp, 50070/tcp, 50075/tcp,
50090/tcp kind_hawking
I'm trying to connect via hdfs to my docker container with:
sudo docker run -ti davvdg/hdfs-client hadoop fs -fs hdfs://localhost:50075 -ls /
This gives the following resut:
ls: Call From a48f81b8e1bb/172.17.0.3 to localhost:50075 failed on
connection exception: java.net.ConnectException: Connection refused; For
more details see: http://wiki.apache.org/hadoop/ConnectionRefused
My question is: How to get my hdfs docker client running?
Edit:
Thanks to some helpful feedback from #shizhz I'm updating the question.
Here is my Dockerfile
FROM sequenceiq/hadoop-docker:2.6.0
CMD ["/etc/bootstrap.sh", "-d"]
# Hdfs ports
EXPOSE 50010 50020 50070 50075 50090 8020 9000
# Mapred ports
EXPOSE 10020 19888
#Yarn ports
EXPOSE 8030 8031 8032 8033 8040 8042 8088
#Other ports
EXPOSE 49707 2122
EXPOSE 9000
EXPOSE 2022
Here is how I'm building the image:
sudo docker build -t my-hdfs .
Here is how I'm running the image:
sudo docker run -d -p my-hdfs
Here is how I'm checking the processes:
sudo docker ps
with a result like:
d9c9855cfaf0 my-hdfs "/etc/bootstrap.sh -d" 2 minutes ago
Up 2 minutes 0.0.0.0:32801->22/tcp, 0.0.0.0:32800->2022/tcp,
0.0.0.0:32799->2122/tcp, 0.0.0.0:32798->8020/tcp, 0.0.0.0:32797->8030/tcp,
0.0.0.0:32796->8031/tcp, 0.0.0.0:32795->8032/tcp, 0.0.0.0:32794->8033/tcp,
0.0.0.0:32793->8040/tcp, 0.0.0.0:32792->8042/tcp, 0.0.0.0:32791->8088/tcp,
0.0.0.0:32790->9000/tcp, 0.0.0.0:32789->10020/tcp, 0.0.0.0:32788->19888/tcp,
0.0.0.0:32787->49707/tcp, 0.0.0.0:32786->50010/tcp, 0.0.0.0:32785->50020/tcp,
0.0.0.0:32784->50070/tcp, 0.0.0.0:32783->50075/tcp, 0.0.0.0:32782->50090/tcp
agitated_curran
Here is how I'm getting the IP address:
docker inspect --format '{{ .NetworkSettings.IPAddress }}' d9c9855cfaf0
with a result like:
172.17.0.3
Here is how I'm running the test:
sudo docker run --rm sequenceiq/hadoop-docker:2.6.0 /usr/local/hadoop-2.6.0/bin/hadoop fs -fs hdfs://192.168.0.3:9000 -ls /
With a result like:
17/04/08 19:51:54 INFO ipc.Client: Retrying connect to server:
192.168.0.3/192.168.0.3:9000. Already tried 0 time(s); maxRetries=45
ls: Call From fafcd377f4a0/172.17.0.2 to 192.168.0.3:9000 failed on connection
exception: java.net.ConnectException: Connection refused; For more details
see: http://wiki.apache.org/hadoop/ConnectionRefused
My question is: How to get my hdfs docker client running?

Answer 1

By default, each container will use bridge network driver and has its own isolation network environment. It's not the same thing but you can just simply think they are different servers and has their own private IPs. So when you started a client container and try to connect the address hdfs://localhost:50075, it'll actually try to connect the port 50075 of itself, rather than connecting the hadoop server container, obviously it'll be refused. Please refer to their official network docs for more info.
Containers on the same host can communicate with each other by their private IPs, so to connect to your hadoop server container, you can firstly find out it private IP by:
$> docker inspect --format '{{ .NetworkSettings.IPAddress }}' 378
192.168.0.2
And then I can use the client by(And I think the port should be 9000):
$> docker run --rm sequenceiq/hadoop-docker:2.6.0 /usr/local/hadoop-2.6.0/bin/hadoop fs -fs hdfs://192.168.0.2:9000 -ls /
Found 1 items
drwxr-xr-x - root supergroup 0 2015-01-15 04:04 /user
If you want to run client container on another host, then you need to know multi-host networking.

Answer 2

for the client, you could use the thin docker image just for client like hdfs client.
one of them are pretty good that I've used before: https://hub.docker.com/r/ryneyang/hadoop-utils/