KUBE_SERVICE inserting REJECT rules in iptables - proxy

I have kubernetes running on 4 centos 7 boxes, master and minions. I also have flannel and skydns installed. flannel overlay ip is 172.17.0.0/16 and my service cluster ip is 10.254.0.0/16. I'm running spinnaker pods on the k8 cluster. what I see is that the spinnaker services are unable to find each other. Each pod gets an ip from the 172.17 slice and I can ping the pods from any of the nodes using that ip. However the services themselves uses the cluser ip and are unable to talk to each other. Since Kube-proxy is the one that should be forwarding this traffic, I looked at the iptable rules and I see this:
[root#MultiNode4 ~$]iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
DOCKER-ISOLATION all -- anywhere anywhere
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- anywhere anywhere /* kubernetes service portals */
Chain DOCKER (1 references)
target prot opt source destination
Chain DOCKER-ISOLATION (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere
Chain KUBE-SERVICES (1 references)
target prot opt source destination
REJECT tcp -- anywhere 10.254.206.105 /* spinnaker/spkr-clouddriver: has no endpoints */ tcp dpt:afs3-prserver reject-with icmp-port-unreachable
REJECT tcp -- anywhere 10.254.162.75 /* spinnaker/spkr-orca: has no endpoints */ tcp dpt:us-srv reject-with icmp-port-unreachable
REJECT tcp -- anywhere 10.254.62.109 /* spinnaker/spkr-rush: has no endpoints */ tcp dpt:8085 reject-with icmp-port-unreachable
REJECT tcp -- anywhere 10.254.68.125 /* spinnaker/spkr-echo: has no endpoints */ tcp dpt:8089 reject-with icmp-port-unreachable
REJECT tcp -- anywhere 10.254.123.127 /* spinnaker/spkr-front50: has no endpoints */ tcp dpt:webcache reject-with icmp-port-unreachable
REJECT tcp -- anywhere 10.254.36.197 /* spinnaker/spkr-gate: has no endpoints */ tcp dpt:8084 reject-with icmp-port-unreachable
Seems like kube-proxy is unable to forward. I have no errors on kube-proxy startup:
[root#MultiNode4 ~$]systemctl status kube-proxy -l
kube-proxy.service - Kubernetes Kube-Proxy Server
Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2016-07-07 02:54:54 EDT; 1h 10min ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 7866 (kube-proxy)
Memory: 3.6M
CGroup: /system.slice/kube-proxy.service
โ””โ”€7866 /usr/bin/kube-proxy --logtostderr=true --v=0 --master=http://centos-master:8080
Jul 07 02:54:54 clm-aus-015349.bmc.com systemd[1]: Started Kubernetes Kube-Proxy Server.
Jul 07 02:54:54 clm-aus-015349.bmc.com systemd[1]: Starting Kubernetes Kube-Proxy Server...
Jul 07 02:54:54 clm-aus-015349.bmc.com kube-proxy[7866]: E0707 02:54:54.754845 7866 server.go:340] Can't get Node "multiNode4", assuming iptables proxy: nodes "MultiNode4" not found
Jul 07 02:54:54 clm-aus-015349.bmc.com kube-proxy[7866]: I0707 02:54:54.756460 7866 server.go:200] Using iptables Proxier.
Jul 07 02:54:54 clm-aus-015349.bmc.com kube-proxy[7866]: I0707 02:54:54.756527 7866 proxier.go:208] missing br-netfilter module or unset br-nf-call-iptables; proxy may not work as intended
Jul 07 02:54:54 clm-aus-015349.bmc.com kube-proxy[7866]: I0707 02:54:54.756551 7866 server.go:213] Tearing down userspace rules.
Jul 07 02:54:54 clm-aus-015349.bmc.com kube-proxy[7866]: I0707 02:54:54.770100 7866 conntrack.go:36] Setting nf_conntrack_max to 262144
Jul 07 02:54:54 clm-aus-015349.bmc.com kube-proxy[7866]: I0707 02:54:54.770145 7866 conntrack.go:41] Setting conntrack hashsize to 65536
Jul 07 02:54:54 clm-aus-015349.bmc.com kube-proxy[7866]: I0707 02:54:54.771445 7866 conntrack.go:46] Setting nf_conntrack_tcp_timeout_established to 86400
What am I missing?

I have the same problem, and it turns out that the selector is wrong.
After fixing that, everything works just fine.

The REJECT is inserted when a particular service has 0 endpoints. The selector in your Service.spec must be wrong or you don't have any pods running.

Related

Unable to start Elasticsearch on Ubuntu 20

I am trying to start elasticsearch after installation. It throws error
Job for elasticsearch.service failed because a fatal signal was delivered to the control process.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
After running: systemctl status elasticsearch.service .
โ— elasticsearch.service - Elasticsearch
Loaded: loaded (/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
Active: failed (Result: signal) since Mon 2021-05-17 14:30:02 IST; 1min 56s ago
Docs: https://www.elastic.co
Process: 94558 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=killed, signal=KILL)
Main PID: 94558 (code=killed, signal=KILL)
May 17 14:29:58 rohit-Lenovo-Legion-Y540-15IRH-PG0 systemd[1]: Starting Elasticsearch...
May 17 14:30:02 rohit-Lenovo-Legion-Y540-15IRH-PG0 systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
May 17 14:30:02 rohit-Lenovo-Legion-Y540-15IRH-PG0 systemd[1]: elasticsearch.service: Failed with result 'signal'.
May 17 14:30:02 rohit-Lenovo-Legion-Y540-15IRH-PG0 systemd[1]: Failed to start Elasticsearch.
In journalctl -xe, I am getting this
May 17 14:30:02 rohit-Lenovo-Legion-Y540-15IRH-PG0 kernel: Out of memory: Killed process 94558 (java) total-vm:9804148kB, anon-rss:5809744kB, file-rss:0kB, shmem-rss:0kB, UID:129 pgtables:11660kB oom_sc>
May 17 14:30:01 rohit-Lenovo-Legion-Y540-15IRH-PG0 CRON[94743]: pam_unix(cron:session): session opened for user root by (uid=0)
May 17 14:30:01 rohit-Lenovo-Legion-Y540-15IRH-PG0 CRON[94744]: (root) CMD ([ -x /etc/init.d/anacron ] && if [ ! -d /run/systemd/system ]; then /usr/sbin/invoke-rc.d anacron start >/dev/null; fi)
May 17 14:30:01 rohit-Lenovo-Legion-Y540-15IRH-PG0 CRON[94743]: pam_unix(cron:session): session closed for user root
May 17 14:30:02 rohit-Lenovo-Legion-Y540-15IRH-PG0 kernel: oom_reaper: reaped process 94558 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
May 17 14:30:02 rohit-Lenovo-Legion-Y540-15IRH-PG0 systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit elasticsearch.service has exited.
--
-- The process' exit code is 'killed' and its exit status is 9.
May 17 14:30:02 rohit-Lenovo-Legion-Y540-15IRH-PG0 systemd[1]: elasticsearch.service: Failed with result 'signal'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit elasticsearch.service has entered the 'failed' state with result 'signal'.
May 17 14:30:02 rohit-Lenovo-Legion-Y540-15IRH-PG0 systemd[1]: Failed to start Elasticsearch.
-- Subject: A start job for unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit elasticsearch.service has finished with a failure.
--
-- The job identifier is 13124 and the job result is failed.
May 17 14:30:02 rohit-Lenovo-Legion-Y540-15IRH-PG0 sudo[94552]: pam_unix(sudo:session): session closed for user root
May 17 14:31:34 rohit-Lenovo-Legion-Y540-15IRH-PG0 kernel: [UFW BLOCK] IN=wlp0s20f3 OUT= MAC=90:78:41:e1:0c:67:ec:0d:e4:f9:4a:71:08:00 SRC=192.168.1.102 DST=192.168.1.108 LEN=390 TOS=0x00 PREC=0x00 TTL=>
May 17 14:31:35 rohit-Lenovo-Legion-Y540-15IRH-PG0 kernel: [UFW BLOCK] IN=wlp0s20f3 OUT= MAC=90:78:41:e1:0c:67:ec:0d:e4:f9:4a:71:08:00 SRC=192.168.1.102 DST=192.168.1.108 LEN=390 TOS=0x00 PREC=0x00 TTL=>
May 17 14:31:35 rohit-Lenovo-Legion-Y540-15IRH-PG0 kernel: [UFW BLOCK] IN=wlp0s20f3 OUT= MAC=90:78:41:e1:0c:67:ec:0d:e4:f9:4a:71:08:00 SRC=192.168.1.102 DST=192.168.1.108 LEN=390 TOS=0x00 PREC=0x00 TTL=>
May 17 14:31:37 rohit-Lenovo-Legion-Y540-15IRH-PG0 kernel: [UFW BLOCK] IN=wlp0s20f3 OUT= MAC=90:78:41:e1:0c:67:ec:0d:e4:f9:4a:71:08:00 SRC=192.168.1.102 DST=192.168.1.108 LEN=390 TOS=0x00 PREC=0x00 TTL=>
May 17 14:31:59 rohit-Lenovo-Legion-Y540-15IRH-PG0 systemd[1]: Started Run anacron jobs.
-- Subject: A start job for unit anacron.service has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit anacron.service has finished successfully.
--
-- The job identifier is 13197.
May 17 14:31:59 rohit-Lenovo-Legion-Y540-15IRH-PG0 anacron[94906]: Anacron 2.3 started on 2021-05-17
May 17 14:31:59 rohit-Lenovo-Legion-Y540-15IRH-PG0 anacron[94906]: Normal exit (0 jobs run)
May 17 14:31:59 rohit-Lenovo-Legion-Y540-15IRH-PG0 systemd[1]: anacron.service: Succeeded.
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit anacron.service has successfully entered the 'dead' state.
My ES Configuration
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
Please help me to resolve this issue.

Peer disconnecting when setting Mikrotik as OpenVPN client

I want to connect my OpenVPN server (Ubuntu 16.4) in my office to my Mikrotik at home as client.
I already have OpenVPN server set based on this tutorial (link). If I try to connect it connects with OpenVPN client Windows app (no errors), and asks for username and password, with Client.ovpn added in Program Files/OpenVPN/config.
Here is my server.conf in OpenVPN server:
port 51333
proto tcp
dev tun5
ca ca.crt
cert server.crt
key server.key
dh dh2048.pem
server 10.8.101.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "route 192.168.20.10 255.255.255.255" #This is my server that I want to connect in Office
keepalive 10 120
tls-auth /etc/openvpn/ta.key
key-direction 0
cipher AES-256-CBC
auth SHA1
max-clients 10
user nobody
group nogroup
persist-key
persist-tun
username-as-common-name
plugin /usr/lib/openvpn/openvpn-plugin-auth-pam.so /etc/pam.d/openvpn
status /etc/openvpn/mikrotik.log
verb 5
mute-replay-warnings
client-config-dir ccd
management localhost 7505
Here is my configuration of base.conf in OpenVPN server (Ubuntu) for creating client.ovpn files:
client
dev tun
proto tcp
remote mydomain.com 51333 #in my DNS I redirect this domain to my public static domain in office there in firewall I am portforwarding this port to my server in office
resolv-retry infinite
nobind
user nobody
group nogroup
auth-user-pass
#ca ca.crt
#cert client.crt
#key client.key
remote-cert-tls server
tls-auth ta.key
cipher AES-256-CBC
auth SHA1
# More reliable detection when a system loses its connection.
ping 15
ping-restart 45
ping-timer-rem
persist-tun
persist-key
verb 5
key-direction 1
Now in Mikrotik (OS is: v6.42.12 in hAP lite (simps) I have:
Import my Client.ovpn in Files
Import Certificate Client.ovpn and set for T (name: ca.crt_0) and for KT (name: client.crt_0).
Create new PPP Profile: ppp profile add name=OVPN-client change-tcp-mss=yes only-one=yes use-encryption=required use-mpls=no
Create new interface: interface ovpn-client add connect-to=mydomain.com port 51333 add-default-route=no auth=sha1 certificate=client.crt_0 disabled=no user=vpnuser password=vpnpass name=myvpn profile=OVPN-client
But with this configuration, I cannot establish a connection. I cannot get "R - status" on OVPN-client, I only get this error:
* ovpn-out1: connecting
* ovpn-out1: terminating - peer disconnected
* ovpn-out1: disconnected
If I check logs in server I get this:
openVPN1 ovpn-server[2050]: MULTI: multi_create_instance called
Oct 26 09:33:03 openVPN1 ovpn-server[2050]: Re-using SSL/TLS context
Oct 26 09:33:03 openVPN1 ovpn-server[2050]: Control Channel MTU parms
Oct 26 09:33:03 openVPN1 ovpn-server[2050]: Data Channel MTU parms
Oct 26 09:33:03 openVPN1 ovpn-server[2050]: Local Options String: 'V4,dev-type tun,link-mtu 1559,tun-mtu 1500,proto TCPv4_SERVER,keydir 0,cipher AES-256-CBC,auth SHA1,keysize 256,tls-auth,key-method 2,tls-server'
Oct 26 09:33:03 openVPN1 ovpn-server[2050]: Expected Remote Options String: 'V4,dev-type tun,link-mtu 1559,tun-mtu 1500,proto TCPv4_CLIENT,keydir 1,cipher AES-256-CBC,auth SHA1,keysize 256,tls-auth,key-method 2,tls-client'
Oct 26 09:33:03 openVPN1 ovpn-server[2050]: Local Options hash (VER=V4): '7ac8f09f'
Oct 26 09:33:03 openVPN1 ovpn-server[2050]: Expected Remote Options hash (VER=V4): '53276059'
Oct 26 09:33:03 openVPN1 ovpn-server[2050]: TCP connection established with [AF_INET]XX.XXX.XXX.XX:60345
Oct 26 09:33:03 openVPN1 ovpn-server[2050]: TCPv4_SERVER link local: [undef]
Oct 26 09:33:03 openVPN1 ovpn-server[2050]: TCPv4_SERVER link remote: [AF_INET]XX.XXX.XXX.XX:60345
Oct 26 09:33:03 openVPN1 ovpn-server[2050]: XX.XXX.XXX.XX:60345 TLS: Initial packet from [AF_INET]XX.XXX.XXX.XX:60345
Oct 26 09:33:03 openVPN1 ovpn-server[2050]: XX.XXX.XXX.XX:60345 TLS Error: cannot locate HMAC in incoming packet from [AF_INET]XX.XXX.XXX.XX:60345
Oct 26 09:33:03 openVPN1 ovpn-server[2050]: XX.XXX.XXX.XX:60345 Fatal TLS error (check_tls_errors_co), restarting
Oct 26 09:33:03 openVPN1 ovpn-server[2050]: XX.XXX.XXX.XX:60345 SIGUSR1[soft,tls-error] received, client-instance restarting
Oct 26 09:33:03 openVPN1 ovpn-server[2050]: TCP/UDP: Closing socket
MikroTik does not support UDP in OpenVPN only TCP.
ROS 7 will support it but it's still in development phase.
As of 2022 - you would need to use ROS version 7 or later, where UPP support for OpenVPN is implemented. Still, Mikrotik's implementation of OpenVPN is limited, eg. no support for TLS auth with a static key.

How to set up autosearch nodes in Elasticsearch 6.1

I have created cluster of 5 nodes in ES 6.1. I am able to create cluster when I added line with all ip addresses of other nodes into configuration file elasticsearch.yaml as discovery.zen.ping.unicast.hosts. It looks like this:
discovery.zen.ping.unicast.hosts: ["10.206.81.241","10.206.81.238","10.206.81.237","10.206.81.239"]
When I have this line in my config file, everything works well.
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.206.81.241 9 54 0 0.03 0.05 0.05 mi * master4
10.206.81.239 10 54 0 0.00 0.01 0.05 mi - master1
10.206.81.238 14 54 0 0.00 0.01 0.05 mi - master3
10.206.81.240 15 54 0 0.00 0.01 0.05 mi - master5
10.206.81.237 10 54 0 0.00 0.01 0.05 mi - master2
When I added discovery.zen.ping.multicast.enabled: true elasticsearch will not start.
I would like to have more nodes and if I will have to configure each file separately and add new address to each configuration every time, it is not proper way. So is there any way how to set up ES6 to find new nodes automatically?
EDIT:
journalctl -f output:
led 08 10:43:04 elk-prod3.user.dc.company.local polkitd[548]: Registered Authentication Agent for unix-process:23395:23676999 (system bus name :1.162 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
led 08 10:43:04 elk-prod3.user.dc.company.local systemd[1]: Stopping Elasticsearch...
led 08 10:43:04 elk-prod3.user.dc.company.local systemd[1]: Started Elasticsearch.
led 08 10:43:04 elk-prod3.user.dc.company.local systemd[1]: Starting Elasticsearch...
led 08 10:43:04 elk-prod3.user.dc.company.local polkitd[548]: Unregistered Authentication Agent for unix-process:23395:23676999 (system bus name :1.162, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
led 08 10:43:07 elk-prod3.user.dc.company.local systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
led 08 10:43:07 elk-prod3.user.dc.company.local systemd[1]: Unit elasticsearch.service entered failed state.
led 08 10:43:07 elk-prod3.user.dc.company.local systemd[1]: elasticsearch.service failed.
Basically you should have "stable" nodes. What i mean is that you should have IPs which are always part of cluster
discovery.zen.ping.unicast.hosts: [MASTER_NODE_IP_OR_DNS, MASTER2_NODE_IP_OR_DNS, MASTER3_NODE_IP_OR_DNS]
Then if you use autoscaling or add nodes they must "talk" to that ips to let them know that they are joining the cluster.
You haven't mentioned anything about your network setup so i can say you for sure what is wrong. But as I recall unicast hosts is recommended approach
PS. If you are using azure, there is feature called VM scaleset I modified template to my needs. Idea is that by default I am always using 3 nodes, and if my cluster is loaded scale set will add dynamically more nodes.
discovery.zen.ping.multicast has been removed from elasticsearch, see: https://www.elastic.co/guide/en/elasticsearch/plugins/6.1/discovery-multicast.html

Shiny's websockets don't work on port 80 of an AWS EC2 instance

I have set up a licensed Shiny Server Pro on an AWS server. It seems to work basically, but I would like to change the default port from 3838 to 80, and this causes me grief, probably because port 80 doesn't handle the websockets as I expect.
So after installation the server dutifully delivers the default welcome.html page with the two embedded apps hello and rmd and everything works as expected. Bus as soon as I change the port in /etc/shiny-server/shiny-server.conf from 3838 to 80, the two embedded apps won't work anymore. They seem to load, but immediately after loading they are greyed out and a message is displayed on top "Disconnected from the Server / Reload".
I now believe that this is caused by the choice of port 80. The AWS instance's security group has a "Custom TCP rule" on port 3838 and allows all incoming traffic on this port. OTOH, port 80 is of type "HTTP" and I seem to be unable to change this to a "Custom TCP rule" (which makes sense, sort of).
When configuring the Shiny server for port 80 and looking at Chrome's dev console after loading the demo page I see websockets appear on the network tab, when the page is fully loaded. However, they're there only for a few milliseconds. Conversely, when using port 3838, the same websockets are persistent and are not terminated.
At the same time, when using port 80, a message appears on the console tab that the connection has been terminated:
Thu Mar 09 2017 08:47:46 [INF]: Connection opened. http://10.43.190.69/sample-apps/rmd/
Thu Mar 09 2017 08:47:46 [DBG]: Open channel 0
Thu Mar 09 2017 08:47:46 [INF]: Connection closed. Info:{"type":"close","code":4705,"reason":"Unable to open connection","wasClean":true}
Thu Mar 09 2017 08:47:46 [DBG]: SockJS connection closed
This message is absent when using port 3838:
Thu Mar 09 2017 09:09:28 [INF]: Connection opened. http://10.43.190.69:3838/sample-apps/hello/
Thu Mar 09 2017 09:09:28 [DBG]: Open channel 0
Thu Mar 09 2017 09:09:28 [INF]: Connection opened. http://10.43.190.69:3838/sample-apps/rmd/
Thu Mar 09 2017 09:09:28 [DBG]: Open channel 0
Thu Mar 09 2017 09:09:30 [DBG]: Open channel 1
Thu Mar 09 2017 09:09:30 [DBG]: 2 message(s) discarded from buffer
Thu Mar 09 2017 09:09:30 [DBG]: 4 message(s) discarded from buffer
To circumvent this problem I have also tried using nginx and apache2 as a reverse proxy, with very similar results, so I tried to use port 80 without a proxy in the first place.
Edit: netstat output when shiny server on port 3838 (all good):
/home/ubuntu# netstat -tlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 *:ssh *:* LISTEN 1451/sshd
tcp 0 0 *:4151 *:* LISTEN 6996/shiny-server
tcp 0 0 *:3838 *:* LISTEN 6996/shiny-server
tcp6 0 0 [::]:ssh [::]:* LISTEN 1451/sshd
netstat output when Shiny server on port 80 (embedded apps not working, no websockets):
/home/ubuntu# netstat -tlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 *:http *:* LISTEN 11116/shiny-server
tcp 0 0 *:ssh *:* LISTEN 1451/sshd
tcp 0 0 *:4151 *:* LISTEN 11116/shiny-server
tcp6 0 0 [::]:ssh [::]:* LISTEN 1451/sshd
So I believe nothing else is listening on port 80.
Can anyone give a hint?
Cheers,
Enno

Facing issues in DC/OS Installation in On-Premises

I am Trying to install DC/OS but facing issues in On-Premises
Below is the Error Log
Connection to 172.XX.XXX.XX closed.
Starting DC/OS Install Process
Running preflight checks
Checking if DC/OS is already installed: PASS (Not installed)
PASS Is SELinux disabled?
Checking if docker is installed and in PATH: PASS
Checking docker version requirement (>= 1.6): PASS (1.11.1)
Checking if curl is installed and in PATH: PASS
Checking if bash is installed and in PATH: PASS
Checking if ping is installed and in PATH: PASS
Checking if tar is installed and in PATH: PASS
Checking if xz is installed and in PATH: PASS
Checking if unzip is installed and in PATH: PASS
Checking if ipset is installed and in PATH: PASS
Checking if systemd-notify is installed and in PATH: PASS
Checking if systemd is installed and in PATH: PASS
Checking systemd version requirement (>= 200): PASS (219)
Checking if group 'nogroup' exists: PASS
Checking if port 80 (required by mesos-ui) is in use: PASS
Checking if port 53 (required by mesos-dns) is in use: PASS
Checking if port 15055 (required by dcos-history) is in use: PASS
Checking if port 5050 (required by mesos-master) is in use: PASS
Checking if port 2181 (required by zookeeper) is in use: PASS
Checking if port 8080 (required by marathon) is in use: PASS
Checking if port 3888 (required by zookeeper) is in use: PASS
Checking if port 8181 (required by exhibitor) is in use: PASS
Checking if port 8123 (required by mesos-dns) is in use: PASS
Checking Docker is configured with a production storage driver: WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
PASS (overlay)
Creating directories under /etc/mesosphere
Creating role file for master
Configuring DC/OS
Setting and starting DC/OS
Job for dcos-setup.service failed because the control process exited with error code. See "systemctl status dcos-setup.service" and "journalctl -xe" for details.
172.XX.XXX.XX:XX
Connection to 172.XX.XXX.XXX closed.
Starting DC/OS Install Process
Running preflight checks
Checking if DC/OS is already installed: PASS (Not installed)
PASS Is SELinux disabled?
Checking if docker is installed and in PATH: PASS
Checking docker version requirement (>= 1.6): PASS (1.11.1)
Checking if curl is installed and in PATH: PASS
Checking if bash is installed and in PATH: PASS
Checking if ping is installed and in PATH: PASS
Checking if tar is installed and in PATH: PASS
Checking if xz is installed and in PATH: PASS
Checking if unzip is installed and in PATH: PASS
Checking if ipset is installed and in PATH: PASS
Checking if systemd-notify is installed and in PATH: PASS
Checking if systemd is installed and in PATH: PASS
Checking systemd version requirement (>= 200): PASS (219)
Checking if group 'nogroup' exists: PASS
Checking if port 80 (required by mesos-ui) is in use: PASS
Checking if port 53 (required by mesos-dns) is in use: PASS
Checking if port 15055 (required by dcos-history) is in use: PASS
Checking if port 5050 (required by mesos-master) is in use: PASS
Checking if port 2181 (required by zookeeper) is in use: PASS
Checking if port 8080 (required by marathon) is in use: PASS
Checking if port 3888 (required by zookeeper) is in use: PASS
Checking if port 8181 (required by exhibitor) is in use: PASS
Checking if port 8123 (required by mesos-dns) is in use: PASS
Checking Docker is configured with a production storage driver: WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
PASS (overlay)
Creating directories under /etc/mesosphere
Creating role file for slave
Configuring DC/OS
Setting and starting DC/OS
Job for dcos-setup.service failed because the control process exited with error code. See "systemctl status dcos-setup.service" and "journalctl -xe" for details.
172.XX.XX.XXX:22
As suggested in Error Log we looked into "systemctl status dcos-setup.service" and "journalctl -xe" but not able to determine the actual cause
[root#macXX ~]# systemctl status dcos-setup.service
รข dcos-setup.service
Loaded: not-found (Reason: No such file or directory)
Active: inactive (dead)
[root#macXX ~]# journalctl -xe
May 27 09:40:01 mac58 kernel: SELinux: initialized (dev overlay, type overlay), uses xattr
May 27 09:40:01 mac58 kernel: device veth0db88ce entered promiscuous mode
May 27 09:40:01 mac58 kernel: IPv6: ADDRCONF(NETDEV_UP): veth0db88ce: link is not ready
May 27 09:40:01 mac58 NetworkManager[776]: <warn> (veth3d6605b): failed to find device 58 'veth3d6605b' with udev
May 27 09:40:01 mac58 NetworkManager[776]: <info> (veth3d6605b): new Veth device (carrier: OFF, driver: 'veth', ifindex: 58)
May 27 09:40:01 mac58 NetworkManager[776]: <warn> (veth0db88ce): failed to find device 59 'veth0db88ce' with udev
May 27 09:40:01 mac58 NetworkManager[776]: <info> (veth0db88ce): new Veth device (carrier: OFF, driver: 'veth', ifindex: 59)
May 27 09:40:01 mac58 NetworkManager[776]: <info> (docker0): bridge port veth0db88ce was attached
May 27 09:40:01 mac58 NetworkManager[776]: <info> (veth0db88ce): enslaved to docker0
May 27 09:40:02 mac58 kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
May 27 09:40:02 mac58 kernel: SELinux: initialized (dev mqueue, type mqueue), uses transition SIDs
May 27 09:40:02 mac58 kernel: SELinux: initialized (dev proc, type proc), uses genfs_contexts
May 27 09:40:02 mac58 kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
May 27 09:40:02 mac58 kernel: SELinux: initialized (dev devpts, type devpts), uses transition SIDs
May 27 09:40:02 mac58 kernel: SELinux: initialized (dev sysfs, type sysfs), uses genfs_contexts
May 27 09:40:02 mac58 kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
May 27 09:40:02 mac58 avahi-daemon[739]: Withdrawing workstation service for veth3d6605b.
May 27 09:40:02 mac58 NetworkManager[776]: <warn> (veth3d6605b): failed to disable userspace IPv6LL address handling
May 27 09:40:02 mac58 NetworkManager[776]: <info> (veth0db88ce): link connected
May 27 09:40:02 mac58 NetworkManager[776]: <info> (docker0): link connected
May 27 09:40:02 mac58 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth0db88ce: link becomes ready
May 27 09:40:02 mac58 kernel: docker0: port 1(veth0db88ce) entered forwarding state
May 27 09:40:02 mac58 kernel: docker0: port 1(veth0db88ce) entered forwarding state
May 27 09:40:03 mac58 avahi-daemon[739]: Registering new address record for fe80::4410:e6ff:fedd:be2a on veth0db88ce.*.
May 27 09:40:17 mac58 kernel: docker0: port 1(veth0db88ce) entered forwarding state
May 27 09:47:10 mac58 kernel: traps: python3[17232] general protection ip:7f2be234423 sp:7f2beceb0394 error:0 in libc-2.22.s
May 27 09:47:11 mac58 kernel: docker0: port 1(veth0db88ce) entered disabled state
May 27 09:47:11 mac58 NetworkManager[776]: <info> (veth0db88ce): link disconnected
May 27 09:47:11 mac58 NetworkManager[776]: <warn> (veth3d6605b): failed to find device 58 'veth3d6605b' with udev
May 27 09:47:11 mac58 NetworkManager[776]: <info> (veth3d6605b): new Veth device (carrier: OFF, driver: 'veth', ifindex: 58)
May 27 09:47:11 mac58 NetworkManager[776]: <info> (docker0): link disconnected (deferring action for 4 seconds)
May 27 09:47:11 mac58 abrt-server[17352]: Executable '/opt/mesosphere/packages/python--e3169ded66609d3cb4055a3f9f8f0b1113a557
May 27 09:47:11 mac58 abrt-server[17352]: 'post-create' on '/var/spool/abrt/ccpp-2016-05-27-09:47:10-17214' exited with 1
May 27 09:47:11 mac58 abrt-server[17352]: Deleting problem directory '/var/spool/abrt/ccpp-2016-05-27-09:47:10-17214'
May 27 09:47:11 mac58 avahi-daemon[739]: Withdrawing address record for fe80::4410:e6ff:fedd:be2a on veth0db88ce.
May 27 09:47:11 mac58 kernel: docker0: port 1(veth0db88ce) entered disabled state
May 27 09:47:11 mac58 avahi-daemon[739]: Withdrawing workstation service for veth3d6605b.
May 27 09:47:11 mac58 avahi-daemon[739]: Withdrawing workstation service for veth0db88ce.
May 27 09:47:11 mac58 NetworkManager[776]: <warn> (veth3d6605b): failed to disable userspace IPv6LL address handling
May 27 09:47:11 mac58 NetworkManager[776]: <info> (docker0): bridge port veth0db88ce was detached
May 27 09:47:11 mac58 NetworkManager[776]: <info> (veth0db88ce): released from master docker0
May 27 09:47:11 mac58 NetworkManager[776]: <warn> (veth0db88ce): failed to disable userspace IPv6LL address handling
May 27 09:47:11 mac58 kernel: device veth0db88ce left promiscuous mode
May 27 09:47:11 mac58 kernel: docker0: port 1(veth0db88ce) entered disabled state
May 27 09:47:11 mac58 docker[10803]: time="2016-05-27T09:47:11.828750505+05:30" level=error msg="Handler for POST /v1.23/cont
May 27 09:47:16 mac58 NetworkManager[776]: <info> (docker0): link disconnected (calling deferred action)
Your Help is Highly Appreciated!!!...Thank you

Resources