I have a kubernetes cluster running on GKE and a Jenkins server running on a GCP instance.
I am using the Kubernetes plugin to dynamically create pods on the kubernetes cluster. I created a pipeline(Declarative syntax) for the same.
So I am aware that the Jenkins slave agents communicates with the Jenkins master on port 50000.
A snip of the configuration
But for some reason when I viewed the logs for the JNLP container creates by Jenkins, I received an exception - tcpSlaveAgentListener not found.
A snip of the container log
According to the above image, I assume the tunneling is unsuccessful as it is trying to connect to http://34.90.46.204:8080/tcpSlaveAgentListener/ whereas it should connect to http://34.90.46.204:50000/tcpSlaveAgentListener/.
It was a lazy question for me to ask, but I solved the issue.
In the Manage Jenkins-> Configure Global Security settings:
For the option on setting a port for TCP inbound agents: unselect the disable option which is selected by default and then provide a port for the inbound agents to interact on (50000).
A snip of the configuration
Jenkins uses a TCP port to communicate with agents connected inbound. If you're going to use inbound agents, you can allow the system to randomly select a port at launch (this avoids interfering with other programs, including other Jenkins instances). As it's hard for firewalls to secure a random port, you can instead specify a fixed port number and configure your firewall accordingly.
Hope this helps someone.
Related
I just used Ansible to monitor my servers with Prometheus, Grafana and Node Exporter. I have one monitoring server (Prometheus) & one webserver (Node Exporter).
I followed a tutorial for the setup. The thing is that it does not provide any information about security. For the moment any one is able to listen on the node_exporter port of my webserver.
I thought about iptable to protect my webserver from external calls on node_exporter port. Then I will only give access to my Promotheus server.
Is it the way to do?
There might be mainly three options
Local or external firewall, as already mentioned in your question
Setting up an encryption proxy (sshified) on your Prometheus Server, which encrypts the outgoing session over SSH to the node_exporter nodes
Setting up an encryption proxy (stunnel) on your Prometheus Nodes, which let you only make an encrypted session, see Authentication and encryption for Prometheus and its exporters
Option 3 can be easily added to the solution of Running node_exporter with Ansible.
Option 2 is also quite simple and can be done via Ansible easily.
Option 1 can be done via Ansible modules available for (local) firewall configuration like firewalld.
There might be more solutions possible like ghostunnel, ...
Should I run consul slaves alongside nomad slaves or inside them?
The later might not make sense at all but I'm asking it just in case.
I brought my own nomad cluster up with consul slaves running alongside nomad slaves (inside worker nodes), my deployable artifacts are docker containers (java spring applications).
The issue with my current setup is that my applications can't access consul slaves (to read configurations) (none of 0.0.0.0, localhost, worker node ip worked)
Lets say my service exposes 8080, I configured docker part (in hcl file) to use bridge as network mode. Nomad maps 8080 to 43210.
Everything is fine until my service tries to reach the consul slave to read configuration. Ideally giving nomad worker node IP as consul host to Spring should suffice. But for some reason it's not.
I'm using latest version of nomad.
I configured my nomad slaves like https://github.com/bmd007/statefull-geofencing-faas/blob/master/infrastructure/nomad/client1.hcl
And the link below shows how I configured/ran my consul slave:
https://github.com/bmd007/statefull-geofencing-faas/blob/master/infrastructure/server2.yml
Note: if I use static port mapping and host as the network mode for docker (in nomad) I'll be fine but then I can't deploy more than one instance of each application in each worker node (due to port conflic)
Nomad jobs listen on a specific host/port pair.
You might want to ssh into the server and run docker ps to see what host/port pair the job is listening on.
a93c5cb46a3e image-name bash 2 hours ago Up 2 hours 10.0.47.2:21435->8000/tcp, 10.0.47.2:21435->8000/udp foo-bar
Additionally, you will need to ensure that the consul nomad job is listening on port 0.0.0.0, or the specific ip of the machine. I believe that is this config value: https://www.consul.io/docs/agent/options.html#_bind
All those will need to match up in order to consul to be reachable.
More generally, I might recommend: if you're going to run consul with nomad, you might want to switch to host networking, so that you don't have to deal with the specifics of the networking within a container. Additionally, you could schedule consul as a system job so that it is automatically present on every host.
So I managed to solve the issue like this:
nomad.job.group.network.mode = host
nomad.job.group.network.port: port "http" {}
nomad.job.group.task.driver = docker
nomad.job.group.task.config.network_mode = host
nomad.job.group.task.config.ports = ["http"]
nomad.job.group.task.service.connect: connect { native = true }
nomad.job.group.task.env: SERVER_PORT= "${NOMAD_PORT_http}"
nomad.job.group.task.env: SPRING_CLOUD_CONSUL_HOST = "localhost"
nomad.job.group.task.env: SPRING_CLOUD_SERVICE_REGISTRY_AUTO_REGISTRATION_ENABLED = "false"
Running consul agent (slaves) using docker-compose alongside nomad agent (slave) with host as network mode + exposing all required ports.
Example of nomad job: https://github.com/bmd007/statefull-geofencing-faas/blob/master/infrastructure/nomad/location-update-publisher.hcl
Example of consul agent config (docker-compose file): https://github.com/bmd007/statefull-geofencing-faas/blob/master/infrastructure/server2.yml
Disclaimer: The LAB is part of Cluster Visualization Framework called: LiteArch Trafik which I have created as an interesting exercise to understand Nomad and Consul.
It took me long time to shift my mind from K8S to Nomad and Consul,
Integration them was one of my effort I spent in the last year.
When service resolution doesn't work, I found out it's more or less the DNS configuration on servers.
There is a section for it on Hashicorp documentation called DNS Forwarding
Hashicorp DNS Forwarding
I have created a LAB which explains how to set up Nomad and Consul.
But you can use the LAB seperately.
I created the LAB after learning the hard way how to install the cluster and how to integrate Nomad and Consul.
With the LAB you need Ubuntu Multipass installed.
You execute one script and you will get full functional Cluster locally with three servers and three nodes.
It shows you as well how to install docker and integrate the services with Consul and DNS services on Ubuntu.
After running the LAB you will get the links to Nomad, Fabio, Consul.
Hopefully it will guide you through the learning process of Nomad and Consul
LAB: LAB
Trafik:Trafik Visualizer
I am trying to set up my JMeter master/slave set up on AWS EC2 instances using windows. Out of the box integration didn't work (connection refused errors) and after some investigation I got to know that RMI communication only works if the machines are on same subnet (is this true?). I found this great article and I tried to follow it as is on windows (tried running ssh port forward tunneling via cygwin) but not have any luck. So I was hoping if someone already done this with Windows and can share their experience.
Out of the box integration didn't work (connection refused errors) and
after some investigation I got to know that RMI communication only
works if the machines are on same subnet (is this true?)
It is not necessary. Even if the machines are in a different subnet and addressable via the network, RMI should work. Here make sure the Security Groups and NACLs are properly set so that both EC2 instances can communicate with each other via the network.
You can check whether you can ping(For this you need to enable ICMP in your Security Groups and also make sure other needed ports are opened for RMI) from one EC2 instance to another. If the servers can communicate with each other, then you need to troubleshoot internal firewall configuration in windows EC2 instances.
We have used https://testable.io/ AMI's from the marketplace. It takes care of all configuration and just needs to worry only about testing and the results.
https://aws.amazon.com/marketplace/pp/B073JNTVKP
All our endpoints are secured to certain IP-ranges from testable.io. If you have internal, still you need to setup those security settings.
Hope it helps.
I spun up a Mesosphere cluster on Digital Ocean (development) and it's not allowing me to allow external (non vpn) connections to containers or apps. How can this be solved ?
To ensure that the world doesn't have access to your cluster normally, there have been iptables rules installed. By default, these allow full access inside the cluster and nothing externally.
If you're interested in running real applications, I'd recommend the following:
Put HAProxy on a single node.
Setup the haproxy-marathon-bridge script.
On the same box that you installed HAProxy on, setup iptables to allow access to the port that HAProxy is listening on.
By doing this, you'll have a single place to refer to when giving access to applications running on your Mesos cluster. No matter where the app or container is scheduled (with marathon), you'll always be able to reach it via. haproxy.
I am having trouble connecting to an Amazon Elastic Cloud Computer Instance via a browser.
I attempted going to ********.compute-1.amazonaws.com , but the browser returns that the connection has timed out.
I can connect via ssh and winscp. That is how I uploaded a web app I developer. I have also created a security group and added rules to open ports 22 and 80.
Do I have to assign the security group to the instance somehow?
The security group's rules also do not have a source IP, well they do its 0.0.0.0/0
I would really appreciate any and all help in getting this site ' viewable ' via a browser.
By default, your instances will only be in the default security group. If it's an EC2 instance you cannot change security groups while the instance is running, you'll have to specify them in advance. If it's a VPC instance you can change security groups at runtime.
Add the rule to the default group
You can however add the rule to allow port 80 to that default security group; just don't create a new security group as it can not be associated with the running instance.
Is the web server up?
Also, make sure that your web server is up and running. From your instance (using SSH shell access), check if the right process is listening on port 80, using the command netstat -lnp. You should then see a row with proto tcp and a Local Address ending in :80. The IP Address listed should be either 0.0.0.0 (meaning 'any IP') or a specific IP of a listening network interface.
Web server not up
If you are in need of a web server, take a look at Apache or Nginx. They both support PHP.
Hope this helps.
I had also faced similar issue with ec2 micro instance. I was using Red-Hat AMI. Despite of opening ports 8081 in security group, I was not able to a telnet to the host port. Disabling the iptable did the trick for me:
sudo /etc/init.d/iptables stop
Do not forget to disable firewall if you use windows for your server.
I faced the same issue while setting up redash AMI image on AWS. Inbound security rules should be changed when instance is not running. Let's say if the instance is running (meaning it's active and started); If you change the inbound rules of that machine you'll still face firewall issue. So Stop the machine on which you want to change the inbound rules on. Change the inbound rules. Start the machine now. Now you can hit the machine url from the ip you just opened the access to the machine to.
The EC2 instance firewall is maybe enabled.
Check it with this command:
sudo systemctl status firewalld
if enabled you can disable it with :
sudo systemctl disable firewalld
or setup rules to allow port 80 trafic