How to setup mesos-dns resolve crossing cluster - mesos

I setup a Mesos cluster with vm machines, machine A (master + agent) and machine B(agent only), and I also run Marathon and Mesos-dns on machine A, both agents support docker.
I startup a web application with docker via Marathon, the docker container run with bridge network mode.
When I run one instance, the container startup normally and Mesos-dns resolves it correct with the docker service internal IP (example, 172.17.0.2), but because I only run one instance and there are two agents in mesos cluster, so only one agent gets the container, the other one there is nothing, if client accesses the mesos agent which there nothing running, there is error appeared.
That means, the container is running on machine B, does not run on machine A, my docker application named test, and listen with port 5000, once I run "curl http://test.marathon.mesos:5000/" on machine B, I get the correct response, but when I run same command on machine A, there is an error appeared "curl: (7) Failed to connect to test.marathon.mesos port 5000: No route to host", the mesos-dns resolve domain to docker internal ip 172.17.0.2, but this ip is not appeared on machine A, because there is not any container running on machine A.
I also can run many instances on same Agent node without any problems, but as I know, mesos cluster and marathon are running application on agent node randomize, so all agent nodes behind load balancer could be accessed, if client access to agent node without container via load balancer, that's a problem for client.
my mesos-dns config file like below:
{
"zk":"zk://10.11.54.103:2181,10.11.54.103:2182,10.11.54.103:2183/mesos",
"masters": ["10.11.54.103:5050"],
"refreshSeconds": 60,
"ttl": 60,
"domain": "mesos",
"port": 53,
"resolvers": ["10.11.255.1","10.11.255.2","4.2.2.2"],
"timeout": 5,
"httpon": true,
"dnson": true,
"httpport": 8123,
"externalon": true,
"listener": "10.11.54.103",
"SOAMname": "ns1.mesos",
"SOARname": "root.ns1.mesos",
"SOARefresh": 60,
"SOARetry": 600,
"SOAExpire": 86400,
"SOAMinttl": 60,
"IPSources": ["netinfo", "mesos", "host"]
}
I wish the Mesos-dns can resole domain cross the whole mesos cluster, is there any idea?

From the documentation:
Mesos-DNS with Docker
If you choose to use Mesos-DNS with Docker, with a version of Mesos after 0.25, be aware that there are some caveats. By default the Docker executor publishes the IP of the Docker container into the NetworkInfo field. Unfortunately, unless you're running some kind of SDN solution, bridged, or host networking with Docker, this can prove to make the containers unreachable.
The default configuration that Mesos-DNS ships with in config.json.sample omits netinfo from the sources. The default options if you omit this field from the configuration includes netinfo. If you have trouble with Docker, ensure you check the IPSources field to omit netinfo.
IPSources defines a fallback list of IP sources for task records,
sorted by priority. If you use Docker, and enable the netinfo IPSource, it may cause tasks to become unreachable, because after Mesos 0.25, the Docker executor publishes the container's internal IP in NetworkInfo.

Related

Connect to a MariaDB Docker Container in a own Docker network remotly

Hi what I am actually trying is to connect remotly from a MySQL Client in Windows Subsystem for Linux mysql -h 172.18.0.2 -P 3306 -u root -p and before that I started the Docker Container as follows: docker container run --name testdb --network testnetwork -p 3306:3306 -e MYSQL_ROOT_PASSWORD=mysqlRootPassword -e MYSQL_DATABASE=localtestdb -d mariadb/server.
The purpose why I put the container in a own network, is because I also have a dockerized Spring Boot Application (GraphQL-Server) which shall communicated with this db. But always when I try to connect from my built-in mysql client, in my Windows Subsystem for Linux, with the above shown command. I got the error message: ERROR 2002 (HY000): Can't connect to MySQL server on '172.18.0.2' (115).
What I already tried, to solve the problem on my own is, look up whether the configuration file line (bind-address) is commented out. But it wont work. Interestingly it already worked to set up a docker container with MariaDB and connect from the outside, but now when I try exactly the same, only with the difference that I now put the container in a own existing network, it wont work.
Hopefully there some one out there which is able to help me with this annonying problem.
Thanks!
So far,
Daniel
//edit:
Now I tried the solution advice from a guy from this topic: How to configure containers in one network to connect to each other (server -> mysql)?. Futhermore I linked my Spring Boot (server) application with the "--link databaseContainerName" parameter to the MariaDB container.
Now I am able to start both containers without any error, but I am still not able to connect remotly to the MariaDB container. Which is now running in a virtual docker network with his own subnet.
I explored this recently - this is by design - container isolation. Usually only main (service httpd) host is accessible externally, hiding internal connections (hosts it communicates to deliver response).
Container created in own network is not accessible from external adresses, even from containers in the same bridge but other network (172.19.0.0/16).
Your container should be accessible on docker host address (127.0.0.1 if run locally) and mapped ("-p 3306:3306") port - 3306. But of course it won't work if many running db containers have the same mapping to the same host port.
Isolation is done using firewall - iptables. You can list rules (iptables -L) to see that - from docker host level.
You can modify firewall to allow external access to internal networks. I used this rule:
iptables -A DOCKER -d 172.16.0.0/12 -j ACCEPT
After that your MySQL containerized engine should be accessible using internal address 172.18.0.2 and source (not mapped) port 3306.
Warnings
it disables all isolation, dont't use it on production;
you have to run this after every docker start - rules created/modified by docker on the fly
not every docker container will respond on ping, check it from docker host (linux subsystem in this case) first, from windows cmd later
I used this option (in docker.service) to make rule permanent:
ExecStartPost=/bin/sh -c '/etc/iptables/accept172_16.sh'
For docker on external(shared in lan) host you should use route add (or hosts file on your machine or router) to forward 172.x.x.x addresses into lan docker host.
Hint: use portainer project (with restart policy - always) to manage docker containers. It's easier to see config errors, too.

How to access a port on the host machine when running docker container on MacOS with --network=host?

I have set up a couple of containers that interact with each other. The main application container runs on --network = host because it queries several mySQL containers running on different ports exposed on the host network.
I am trying to hit the application on the host but get an error:
curl: (7) Failed to connect to 0.0.0.0 port 36081: Connection refused
I am working on Docker installed on MacOS.
I have read several questions that indicate that docker on MacOS runs on a VM. But what is the workaround to access the application from the host? Any way to get the IP of the said VM?
You cannot use --network=host on Mac to connect via host ports but binding to host port using -p options works.
https://docs.docker.com/docker-for-mac/networking/#/there-is-no-docker0-bridge-on-osx
I WANT TO CONNECT TO A CONTAINER FROM THE MAC
Port forwarding works
for localhost; --publish, -p, or -P all work. Ports exposed from Linux
are forwarded to the host.
Our current recommendation is to publish a port, or to connect from
another container. This is what you need to do even on Linux if the
container is on an overlay network, not a bridge network, as these are
not routed.
For your use case,
You need to create a docker network and attach both the DB and application containers to this network. Then the containers will be able to talk to each other by their name. You can also publish the application container port so that you can access it from your host.
https://docs.docker.com/network/bridge/
Instead of creating the network, attaching the containers to the network etc manually, you can use docker-compose.
https://docs.docker.com/compose/

Running multiple, independent, Docker daemon on Windows with Hyper-V isolation and LCOW

I am trying to run multiple Docker daemon configured to run containers with Hyper-V isolation and LCOW on the same Windows 10 machine.
I was able to configure the daemons to manage their own data files, but I am still struggling to get the network configuration clean.
When the first daemon start, it binds to the local "nat" network for DNS resolution. When the second daemon starts, it tries to bind to the same "nat" network then fails as port 53 is already being used by first daemon.
ERRO[2019-02-15T15:50:58.194988300Z] Resolver Setup/Start failed for container nat, "error in opening name server socket listen udp 172.18.64.1:53: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted."
Containers started by this daemon then cannot perform any name resolution. Access through IP still works properly.
Here is the dockerd configuration I am currently using:
{
"registry-mirrors": [],
"insecure-registries": [],
"bridge": "mydaemon1",
"data-root": "C:\\Users\\myuser\\Desktop\\Docker\\Docker",
"deprecated-key-path": "C:\\Users\\myuser\\Desktop\\Docker\\Docker\\config\\key.json",
"debug": true,
"exec-root": "C:\\Users\\myuser\\Desktop\\Docker\\Docker\\exec-root",
"exec-opts": [
"isolation=hyperv"
],
"experimental": true,
"group": "mydaemon-docker",
"hosts": [
"npipe:////./pipe/mydaemon1_engine"
],
"pidfile": "C:\\Users\\myuser\\Desktop\\Docker\\Docker\\docker.pid",
"storage-opts": [
"lcow.kirdpath=C:\\Users\\myuser\\Desktop\\Docker\\server\\resources",
"lcow.kernel=lcow-kernel",
"lcow.initrd=lcow-initrd.img"
]
}
I tried to tweak the bridge configuration, but it didn't change anything. Daemon always tries to connect to nat network. It looks like the only supported value is none, which removes the default eth0 in the containers and any DNS support.
Is it possible to configure the network used for DNS resolution, ie nat here?
Ideally I want the daemon to have its own, dedicated, nat network.
I know it is not possible to do it in Docker for Windows while using the MobyVM as WinNAT, which is used in that case, does not support it.
While using Hyper-V isolation and LCOW, it seems WinNAT is not used anymore as Get-NetNat does not return any NAT network configuration despite DNS working properly. I am not sure I am right on anything, whether this is possible neither if any other Windows limitation applies...

docker beta on osx dns for links between containers not available

Yesterday I installed the docker-beata (https://beta.docker.com/) for osx. So far it seems great but the links defined between containers still do not to work out of the box, e.g their respective DNS name does not seem to be resolved.
How can I change this to make the dns-name of the linked container available on the (osx/ windows)host using docker-beta?
links:
- someName
A pinata list returns the following, and a ping to docker.local fails with unknown host
🐳 hostname = docker
Hostname of the virtual machine endpoint, where container ports will be
exposed if using nat networking. Access it via 'docker.local'.
🐳 hypervisor = native (memory=8, ncpu=4)
The Docker.app includes embedded hypervisors that run the virtual machines
that power the containers. This setting allows you to control which the
default one used for Linux is.
▸ native: a version of the xhyve hypervisor that uses the MacOSX
Hypervisor.framework to run container VMs. Parameters:
memory (VM memory in gigabytes), ncpu (vCPUs)
🐳 network = hostnet (docker-ipv4=192.168.65.2, host-ipv4=192.168.65.1)
Controls how local containers can access the external network via the
MacOS X host. This includes outbound traffic as well as publishing ports
for external access to the local containers.
▸ hostnet: a mode that helps if you are using a VPN that restricts
connectivity. Activating this mode will proxy container network
packets via the Docker.app process as host socket traffic.
Parameters: docker-ipv4 (docker node), host-ipv4 (host node)
▸ nat: a mode that uses the MacOS X vmnet.framework to route container
traffic to the host network via a NAT.
🐳 filesystem = osxfs
Controls the mode by which files from the MacOS X host and the container
filesystem are shared with each other.
▸ osxfs: a FUSE-based filesystem that bidirectionally forwards OSX
filesystem events into the container.
🐳 native/port-forwarding = true
Expose container ports on the Mac, rather than the VM
▸ true: Container ports will be exposed on the Mac
▸ false: Container ports will be exposed on the VM
🐳 daemon = run 'pinata get daemon' or 'pinata set daemon [#file|-]>
JSON configuration of the local Docker daemon. Configure any custom
options you need as documented in:
https://docs.docker.com/engine/reference/commandline/daemon/. Set it
directly, or a #file or - for stdin.
Described here with the old docker https://github.com/databricks/spark-integration-tests
With boot2docker, the Docker containers will be run inside of a
VirtualBox VM, which creates some difficulties for communication
between the Mac host and the containers. Follow these instructions to
work around those issues:
Network access: Our tests currently run the SparkContext from outside
of the containers, so we need both host <-> container and container
<-> container networking to work properly. This is complicated by the
fact that boot2docker runs the containers behind a NAT in VirtualBox.
One workaround is to add a routing table entry that routes traffic to
containers to the VirtualBox VM's IP address:
sudo route -n add 172.17.0.0/16 boot2docker ip You'll have to
re-run this command if you restart your computer or assign a new IP to
the VirtualBox VM.
but this no longer works, as the docker-beata runs already in hostnet
I had this problem too, it seems to be fixed after disabling OS X firewall
I could swear I saw some documentation of an executable to whitelist instead of disabling the firewall, but I can't find it...

How to update Kafka config file with Docker IP address

I am running Kafka inside a Docker container. Kafka requires a connection to Zookeeper, and so I am running Zookeeper in another container. I am running Docker on OSX and so my VM has the IP address: 192.168.99.99.
What I can't figure out, is how do I update my Kafka Docker installation to point to the instance of Zookeeper running inside its own separate Docker container, i.e. with IP address of 192.168.99.9 and port 2181?
Kafka has a config file called server.properties which has a property of zookeeper.connect which I can set, but I want this value to be overridden dynamically, rather than hard-coding the IP here. How do I achieve this?
And, as an additional question, I want my Docker file to work across OS's - so whatever I do should work on Linux too..
You should not need to set an ip in that config file:
Through docker-compose v2 (docker 1.10+), a bridge network is created which means both containers are in that network and see each other.
See more at "Networking in Compose".
If Zookeeper expose its port 2181, the config file from Kafka can simply reference zookeeper by its container name.
And that will work on any docker (boot2docker on Mac or native docker on Linux)

Resources