I'm having problems when setting up replication in openshift/docker cluster.
In openshift, each opendj server will have two ips: service ip and pod id. So when I setup two opendj service, two service ip and two pod ip will be there.
I want to set up the replication by service ip, because pod is is not accessible from other pod, but apparently OpenDJ think there are four replication server there, with each two server having same ServerId.
Log snippet:
category=SYNC severity=ERROR msgID=org.opends.messages.replication.55 msg=In Replication server Replication Server 8989 31635: replication servers 172.30.244.127(service ip):8989 and 10.129.0.1:8989(pod ip) have the same ServerId : 11281
My Question is: is it possible to just build the Replication Server cluster by Service IP, not Pod id?
Thanks a lot.
PS: seems this issue is similar with this https://bugster.forgerock.org/jira/browse/OPENDJ-567
Wayne
For anyone having the same issue, please config your opendj service to headless service, that will solve the problem
Related
I have a mesos master behind a load balancer and a mesos agent that tries to connect to the mesos master via the load balancer
Everything is good when the agent directly connects to the master by providing the --master flag but as soon as I change the --master to point to the load balancer (dns entry not the LB IP) I keep getting the following error repeatedly on my agent
I0223 11:16:55.776448 4945 slave.cpp:1416] Detecting new master
I0223 11:16:55.796245 4947 slave.cpp:6456] Got exited event for master#xx.xx.xx.xx:8082
W0223 11:16:55.796283 4947 slave.cpp:6461] Master disconnected! Waiting for a new master to be elected
I don't see any logs in master
mesos master port:8082
load balancer listener:8082->8082
mesos agent port:5052
We use the classic load balancer that does not preserve the IP
I then tried advertising the agent IP & Port but that didn't help either
I also tried setting --hostname, --advertise_ip & --advertise_port on master but that didn't help either
Has anyone faced this issue? What should be the right values for --advertise_ip, --advertise_port
I'm not using the standard mesos master/agent ports FYI
At this point I've tried all sorts of combinations
hostname = DNS name
advertise_ip = IP DNS resolves to
advertise_port = External port
I want to run minio cluster for tests, this cluster should contain 2 servers with 4 drives each.
For this purpose was selected minio setup as systemd service.
Both servers has same configuration in /etc/default/minio file:
# Volume to be used for MinIO server.
MINIO_VOLUMES="http://10.24.36.82/tmp/minio/srv/d{1...4} http://10.24.36.83/tmp/minio/srv/d{1...4}"
# Use if you want to run MinIO on a custom port.
#MINIO_OPTS="--address :9199"
# Root user for the server.
#MINIO_ROOT_USER=Root-User
# Root secret for the server.
Minio start is ok, cluster is working, but for some reason admin console shows that there're 3 servers in the cluster and one is always offine. When I open minio console on 10.24.36.82, it shows third server with same ip:
Server 10.24.36.83 has same picture but this time it has its own clone:
Lots of errors produced in minio logs about third server offline.
My question is why minio duplicates its instance and how to fix this?
The problem was in server url definition. MINIO_VOLUMES must contain port for every server pool address. If its not it starts somehow and tries to find extra server on port 80
im just getting started learning nomad and consul
i have several servers without a Local Area Network and they are connected through wam (which i think you mean by datacenters) every server is a datacenter
i found in the docs https://www.consul.io/docs/architecture that each datacenter should have 3 to 5 consul servers so is my case applicable with consul and nomad
should i make all of the consul servers or 3 servers as consul and the rest are consul clients
You could use:
3*consul servers or
2*consul clients and one server
I have every tested all these cases they work correctly
Should I run consul slaves alongside nomad slaves or inside them?
The later might not make sense at all but I'm asking it just in case.
I brought my own nomad cluster up with consul slaves running alongside nomad slaves (inside worker nodes), my deployable artifacts are docker containers (java spring applications).
The issue with my current setup is that my applications can't access consul slaves (to read configurations) (none of 0.0.0.0, localhost, worker node ip worked)
Lets say my service exposes 8080, I configured docker part (in hcl file) to use bridge as network mode. Nomad maps 8080 to 43210.
Everything is fine until my service tries to reach the consul slave to read configuration. Ideally giving nomad worker node IP as consul host to Spring should suffice. But for some reason it's not.
I'm using latest version of nomad.
I configured my nomad slaves like https://github.com/bmd007/statefull-geofencing-faas/blob/master/infrastructure/nomad/client1.hcl
And the link below shows how I configured/ran my consul slave:
https://github.com/bmd007/statefull-geofencing-faas/blob/master/infrastructure/server2.yml
Note: if I use static port mapping and host as the network mode for docker (in nomad) I'll be fine but then I can't deploy more than one instance of each application in each worker node (due to port conflic)
Nomad jobs listen on a specific host/port pair.
You might want to ssh into the server and run docker ps to see what host/port pair the job is listening on.
a93c5cb46a3e image-name bash 2 hours ago Up 2 hours 10.0.47.2:21435->8000/tcp, 10.0.47.2:21435->8000/udp foo-bar
Additionally, you will need to ensure that the consul nomad job is listening on port 0.0.0.0, or the specific ip of the machine. I believe that is this config value: https://www.consul.io/docs/agent/options.html#_bind
All those will need to match up in order to consul to be reachable.
More generally, I might recommend: if you're going to run consul with nomad, you might want to switch to host networking, so that you don't have to deal with the specifics of the networking within a container. Additionally, you could schedule consul as a system job so that it is automatically present on every host.
So I managed to solve the issue like this:
nomad.job.group.network.mode = host
nomad.job.group.network.port: port "http" {}
nomad.job.group.task.driver = docker
nomad.job.group.task.config.network_mode = host
nomad.job.group.task.config.ports = ["http"]
nomad.job.group.task.service.connect: connect { native = true }
nomad.job.group.task.env: SERVER_PORT= "${NOMAD_PORT_http}"
nomad.job.group.task.env: SPRING_CLOUD_CONSUL_HOST = "localhost"
nomad.job.group.task.env: SPRING_CLOUD_SERVICE_REGISTRY_AUTO_REGISTRATION_ENABLED = "false"
Running consul agent (slaves) using docker-compose alongside nomad agent (slave) with host as network mode + exposing all required ports.
Example of nomad job: https://github.com/bmd007/statefull-geofencing-faas/blob/master/infrastructure/nomad/location-update-publisher.hcl
Example of consul agent config (docker-compose file): https://github.com/bmd007/statefull-geofencing-faas/blob/master/infrastructure/server2.yml
Disclaimer: The LAB is part of Cluster Visualization Framework called: LiteArch Trafik which I have created as an interesting exercise to understand Nomad and Consul.
It took me long time to shift my mind from K8S to Nomad and Consul,
Integration them was one of my effort I spent in the last year.
When service resolution doesn't work, I found out it's more or less the DNS configuration on servers.
There is a section for it on Hashicorp documentation called DNS Forwarding
Hashicorp DNS Forwarding
I have created a LAB which explains how to set up Nomad and Consul.
But you can use the LAB seperately.
I created the LAB after learning the hard way how to install the cluster and how to integrate Nomad and Consul.
With the LAB you need Ubuntu Multipass installed.
You execute one script and you will get full functional Cluster locally with three servers and three nodes.
It shows you as well how to install docker and integrate the services with Consul and DNS services on Ubuntu.
After running the LAB you will get the links to Nomad, Fabio, Consul.
Hopefully it will guide you through the learning process of Nomad and Consul
LAB: LAB
Trafik:Trafik Visualizer
I don't see a way to configure the cluster FQDN for On Premise installation.
I create a 6 nodes cluster (each nodes running on a physical server) and I'm only able to contact each node on their own IP instead of contacting the cluster on a "general FQDN". With this model, I'm to be are of which node is up, and which node is down.
Does somebody know how to achieve it, based on the sample configurations files provided with Service Fabric standalone installation package?
You need to add a network load balancer to your infrastructure for that. This will be used to route traffic to healthy nodes.