consul agent could only bind to 127.0.0.1 - consul

After installing consul on a Debian virtualbox VM, I ran "consul agent -dev" to start the agent. The stdout looked fine.
Then I typed "consul members" in another terminal, and found it's listening on 127.0.0.1:8301 instead of a non-loopback address.
root#debian02:/home/steven# consul members
Node Address Status Type Build Protocol DC Segment
debian02 127.0.0.1:8301 alive server 1.0.1 2 dc1 <all>
I could run another program that can listen on all addresses, so the network configuration should be fine for the VM.
What could be wrong?
Here's the output of ip address:
root#debian02:/home/steven# ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:9d:a7:7e brd ff:ff:ff:ff:ff:ff
inet 10.0.2.4/24 brd 10.0.2.255 scope global dynamic enp0s3
valid_lft 812sec preferred_lft 812sec
inet6 fe80::a00:27ff:fe9d:a77e/64 scope link
valid_lft forever preferred_lft forever
Output of netstat:
root#debian02:/home/steven# netstat -lntu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 127.0.0.1:8302 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:8500 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:8600 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:8300 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:8301 0.0.0.0:* LISTEN
tcp6 0 0 ::1:631 :::* LISTEN
udp 0 0 0.0.0.0:631 0.0.0.0:*
udp 0 0 0.0.0.0:5353 0.0.0.0:*
udp 0 0 0.0.0.0:1900 0.0.0.0:*
udp 0 0 0.0.0.0:47046 0.0.0.0:*
udp 0 0 0.0.0.0:68 0.0.0.0:*
udp 0 0 127.0.0.1:8301 0.0.0.0:*
udp 0 0 127.0.0.1:8302 0.0.0.0:*
udp 0 0 127.0.0.1:8600 0.0.0.0:*
udp6 0 0 :::52337 :::*
udp6 0 0 :::5353 :::*
Thanks,
Steven


have you read the documentation? https://www.consul.io/docs/agent/options.html#_client
it'll bind to 127.0.0.1 by default and its changeable. you can use 0.0.0.0

Related

Cannot bind to some ports due to permission denied

For the last 3 months or so I'm having random errors where I can't bind a specific port where our Identity server is running on my local development workstation. At first I thought it's my broken machine, so I reset everything, which kinda fixed the issue for 2 months and now it is back.
In the meanwhile other developers saw the same issue. All of us who experience the issue are running Windows 10, but not everyone with Windows 10 has that issue.
Restarting windows after it was shut down with shutdown /s /f /t 0 is the only reliable solution to open up the port again. After I restart normally (due to windows updates or similar) the issue comes up again an I need to shut down windows with that command.
I wrote a small f# script to test which ports are affected. The exact error message I get is
An attempt was made to access a socket in a way forbidden by its access permissions
When I looked for it, the suggested solutions where to restart the machine, but that seems more like a work around, not a real solution. I've also found https://stackoverflow.com/a/10461978/621366 - but netstat -o doesn't list the port, neither does the tool TCPView. All of us are pretty sure that the port isn't occupied by anything. I also tried running netstat in an administrator console and similar commands inside bash on windows, but couldn't find anything. I can't even connect to the port via telnet, it says nothing is listening on the port.
those ports in question are for instance:
49670 - 49689
49710 - 49749
49760 - 49779
49811 - 49830
49843 - 49882
50197 - 50216
None of us modified anything on the windows firewall or has any additional anti virus tools installed except the windows 10 default ones. So everything should be on default values. And it also worked normally for ~10 months before it broke the first time and afterwards for 2 months. In both cases after some windows updates where installed. The last time it was a bios update (probably due to the meltdown / spectre issues?).
Also trying to open up the ports explicitly on the firewall didn't help.
According to this answer https://stackoverflow.com/a/23982901/621366 TCPView and netstat shouldn't miss any occupied ports, but even when I enable showing unconnected endpoints in TCPView, I don't see any of the ports where I get permission denied when trying to bind them. Here a screen from the occupied ports (I marked the bordering occupied ports which are right before or after the group of permission-denied-ports)
UPDATE:
I've noticed that it always seems to be 160 or 180 (exact numbers) of ports which have permission denied in the ranges of 40,000+ This seems oddly coincidental to me, so obviously something is occupying the ports on purpose, but what? I can't seem to find anything in the windows event logs (although I wouldn't know what to look for exactly) and none of those ports shows up any any of my firewall rules. Also shutting down docker for windows doesn't make any difference and when a colleague mentioned that for them it's enough to restart docker for windows (in the UI go to Reset->Restart) and right now for me, even restarting with the shutdown command doesn't work anymore.
UPDATE 2:
The output of netstat -ano run from an administrator powershell:
Proto Local Address Foreign Address State PID
TCP 0.0.0.0:135 0.0.0.0:0 LISTENING 1152
TCP 0.0.0.0:445 0.0.0.0:0 LISTENING 4
TCP 0.0.0.0:2179 0.0.0.0:0 LISTENING 4696
TCP 0.0.0.0:5040 0.0.0.0:0 LISTENING 6616
TCP 0.0.0.0:5357 0.0.0.0:0 LISTENING 4
TCP 0.0.0.0:5432 0.0.0.0:0 LISTENING 11100
TCP 0.0.0.0:7680 0.0.0.0:0 LISTENING 7056
TCP 0.0.0.0:17500 0.0.0.0:0 LISTENING 9668
TCP 0.0.0.0:49664 0.0.0.0:0 LISTENING 784
TCP 0.0.0.0:49665 0.0.0.0:0 LISTENING 1628
TCP 0.0.0.0:49666 0.0.0.0:0 LISTENING 2028
TCP 0.0.0.0:49667 0.0.0.0:0 LISTENING 3560
TCP 0.0.0.0:49800 0.0.0.0:0 LISTENING 856
TCP 0.0.0.0:49821 0.0.0.0:0 LISTENING 892
TCP 0.0.0.0:50000 0.0.0.0:0 LISTENING 11100
TCP 0.0.0.0:50001 0.0.0.0:0 LISTENING 11100
TCP 0.0.0.0:51000 0.0.0.0:0 LISTENING 11100
TCP 10.0.75.1:139 0.0.0.0:0 LISTENING 4
TCP 10.0.75.1:445 10.0.75.2:44848 ESTABLISHED 4
TCP 127.0.0.1:843 0.0.0.0:0 LISTENING 9668
TCP 127.0.0.1:944 0.0.0.0:0 LISTENING 688
TCP 127.0.0.1:944 127.0.0.1:50968 TIME_WAIT 0
TCP 127.0.0.1:944 127.0.0.1:50970 TIME_WAIT 0
TCP 127.0.0.1:944 127.0.0.1:50973 TIME_WAIT 0
TCP 127.0.0.1:944 127.0.0.1:50977 TIME_WAIT 0
TCP 127.0.0.1:944 127.0.0.1:50981 TIME_WAIT 0
TCP 127.0.0.1:944 127.0.0.1:50990 TIME_WAIT 0
TCP 127.0.0.1:944 127.0.0.1:50992 TIME_WAIT 0
TCP 127.0.0.1:944 127.0.0.1:50996 TIME_WAIT 0
TCP 127.0.0.1:944 127.0.0.1:51005 TIME_WAIT 0
TCP 127.0.0.1:944 127.0.0.1:51007 TIME_WAIT 0
TCP 127.0.0.1:944 127.0.0.1:51009 TIME_WAIT 0
TCP 127.0.0.1:944 127.0.0.1:51015 TIME_WAIT 0
TCP 127.0.0.1:944 127.0.0.1:51017 ESTABLISHED 688
TCP 127.0.0.1:4380 0.0.0.0:0 LISTENING 11024
TCP 127.0.0.1:6942 0.0.0.0:0 LISTENING 9296
TCP 127.0.0.1:17600 0.0.0.0:0 LISTENING 9668
TCP 127.0.0.1:49668 127.0.0.1:49669 ESTABLISHED 688
TCP 127.0.0.1:49669 127.0.0.1:49668 ESTABLISHED 688
TCP 127.0.0.1:50076 127.0.0.1:50077 ESTABLISHED 8828
TCP 127.0.0.1:50077 127.0.0.1:50076 ESTABLISHED 8828
TCP 127.0.0.1:50173 127.0.0.1:50174 ESTABLISHED 9668
TCP 127.0.0.1:50174 127.0.0.1:50173 ESTABLISHED 9668
TCP 127.0.0.1:50175 127.0.0.1:50176 ESTABLISHED 9668
TCP 127.0.0.1:50176 127.0.0.1:50175 ESTABLISHED 9668
TCP 127.0.0.1:50197 127.0.0.1:50198 ESTABLISHED 9668
TCP 127.0.0.1:50198 127.0.0.1:50197 ESTABLISHED 9668
TCP 127.0.0.1:50335 127.0.0.1:50336 ESTABLISHED 6424
TCP 127.0.0.1:50336 127.0.0.1:50335 ESTABLISHED 6424
TCP 127.0.0.1:50346 127.0.0.1:50347 ESTABLISHED 11100
TCP 127.0.0.1:50347 127.0.0.1:50346 ESTABLISHED 11100
TCP 127.0.0.1:51011 127.0.0.1:51012 ESTABLISHED 9296
TCP 127.0.0.1:51012 127.0.0.1:51011 ESTABLISHED 9296
TCP 127.0.0.1:51013 127.0.0.1:51014 ESTABLISHED 9296
TCP 127.0.0.1:51014 127.0.0.1:51013 ESTABLISHED 9296
TCP 127.0.0.1:51016 0.0.0.0:0 LISTENING 9296
TCP 127.0.0.1:51017 127.0.0.1:944 ESTABLISHED 8828
TCP 127.0.0.1:63342 0.0.0.0:0 LISTENING 9296
TCP 127.94.0.1:946 0.0.0.0:0 LISTENING 688
TCP 127.94.0.2:946 0.0.0.0:0 LISTENING 688
TCP 127.94.0.3:946 0.0.0.0:0 LISTENING 688
TCP 127.94.0.4:946 0.0.0.0:0 LISTENING 688
TCP 169.254.105.83:139 0.0.0.0:0 LISTENING 4
TCP 192.168.0.107:139 0.0.0.0:0 LISTENING 4
TCP 192.168.0.107:49415 111.221.29.134:443 ESTABLISHED 4316
TCP 192.168.0.107:49417 111.221.29.127:443 ESTABLISHED 4316
TCP 192.168.0.107:50185 162.125.66.3:443 CLOSE_WAIT 9668
TCP 192.168.0.107:50246 52.70.31.26:443 CLOSE_WAIT 9668
TCP 192.168.0.107:50253 35.177.204.73:443 ESTABLISHED 2804
TCP 192.168.0.107:50254 35.177.204.73:443 ESTABLISHED 2804
TCP 192.168.0.107:50256 35.177.204.73:443 ESTABLISHED 2804
TCP 192.168.0.107:50257 158.85.224.175:443 ESTABLISHED 10836
TCP 192.168.0.107:50258 13.69.14.160:443 ESTABLISHED 8620
TCP 192.168.0.107:50310 66.102.1.188:443 ESTABLISHED 11184
TCP 192.168.0.107:50329 157.240.20.15:443 ESTABLISHED 10836
TCP 192.168.0.107:50331 111.221.29.74:443 ESTABLISHED 10072
TCP 192.168.0.107:50332 162.125.18.133:443 ESTABLISHED 9668
TCP 192.168.0.107:50351 40.77.226.194:443 ESTABLISHED 8620
TCP 192.168.0.107:50460 66.102.1.189:443 ESTABLISHED 10836
TCP 192.168.0.107:50470 66.102.1.189:443 ESTABLISHED 10836
TCP 192.168.0.107:50501 192.30.253.125:443 ESTABLISHED 11184
TCP 192.168.0.107:50513 40.77.226.194:443 ESTABLISHED 8620
TCP 192.168.0.107:50529 87.98.218.198:443 ESTABLISHED 12540
TCP 192.168.0.107:50530 172.217.21.46:443 ESTABLISHED 10836
TCP 192.168.0.107:50616 172.217.21.46:443 ESTABLISHED 10836
TCP 192.168.0.107:50630 162.125.18.133:443 ESTABLISHED 9668
TCP 192.168.0.107:50641 172.217.21.37:443 ESTABLISHED 10836
TCP 192.168.0.107:50645 162.125.66.4:443 CLOSE_WAIT 9668
TCP 192.168.0.107:50668 87.98.218.198:443 ESTABLISHED 12540
TCP 192.168.0.107:50703 151.101.0.133:443 ESTABLISHED 11184
TCP 192.168.0.107:50709 192.30.253.125:443 ESTABLISHED 11184
TCP 192.168.0.107:50744 87.98.218.198:443 ESTABLISHED 12540
TCP 192.168.0.107:50828 162.125.66.3:443 CLOSE_WAIT 9668
TCP 192.168.0.107:50830 151.101.0.133:443 ESTABLISHED 11184
TCP 192.168.0.107:50831 151.101.0.133:443 ESTABLISHED 11184
TCP 192.168.0.107:50832 151.101.0.133:443 ESTABLISHED 11184
TCP 192.168.0.107:50834 151.101.0.133:443 ESTABLISHED 11184
TCP 192.168.0.107:50835 151.101.0.133:443 ESTABLISHED 11184
TCP 192.168.0.107:50836 151.101.0.133:443 ESTABLISHED 11184
TCP 192.168.0.107:50837 151.101.0.133:443 ESTABLISHED 11184
TCP 192.168.0.107:50839 192.30.253.125:443 ESTABLISHED 11184
TCP 192.168.0.107:50844 151.101.0.133:443 ESTABLISHED 11184
TCP 192.168.0.107:50847 192.30.253.124:443 ESTABLISHED 11184
TCP 192.168.0.107:50857 192.30.253.124:443 ESTABLISHED 11184
TCP 192.168.0.107:50863 162.125.34.137:443 CLOSE_WAIT 9668
TCP 192.168.0.107:50865 172.217.21.46:443 TIME_WAIT 0
TCP 192.168.0.107:50866 172.217.21.46:443 ESTABLISHED 10836
TCP 192.168.0.107:50910 35.186.213.138:443 TIME_WAIT 0
TCP 192.168.0.107:50923 172.217.21.46:443 ESTABLISHED 10836
TCP 192.168.0.107:50925 40.117.190.72:443 ESTABLISHED 4040
TCP 192.168.0.107:50927 172.217.21.42:443 ESTABLISHED 11184
TCP 192.168.0.107:50949 151.101.0.133:443 ESTABLISHED 11184
TCP 192.168.0.107:50950 151.101.0.133:443 ESTABLISHED 11184
TCP 192.168.0.107:50951 151.101.0.133:443 ESTABLISHED 11184
TCP 192.168.0.107:50954 192.30.253.124:443 ESTABLISHED 11184
TCP 192.168.0.107:50959 40.117.190.72:443 TIME_WAIT 0
TCP 192.168.0.107:50969 192.30.253.113:22 TIME_WAIT 0
TCP 192.168.0.107:50978 87.98.218.198:443 ESTABLISHED 12540
TCP 192.168.0.107:50984 151.101.1.69:443 ESTABLISHED 11184
TCP 192.168.0.107:50985 192.0.73.2:443 ESTABLISHED 11184
TCP 192.168.0.107:50986 104.16.112.18:443 ESTABLISHED 11184
TCP 192.168.0.107:50991 198.252.206.25:443 ESTABLISHED 11184
TCP 192.168.0.107:50993 192.168.0.10:3910 TIME_WAIT 0
TCP 192.168.0.107:50994 192.168.0.10:3910 TIME_WAIT 0
TCP 192.168.0.107:50997 23.210.254.37:443 ESTABLISHED 912
TCP 192.168.0.107:50998 23.210.254.37:443 ESTABLISHED 912
TCP 192.168.0.107:50999 23.210.254.37:443 ESTABLISHED 912
TCP 192.168.0.107:51001 23.210.254.37:443 ESTABLISHED 912
TCP 192.168.0.107:51006 40.117.190.72:443 ESTABLISHED 11992
TCP 192.168.0.107:51008 40.69.218.62:443 ESTABLISHED 7056
TCP 192.168.0.107:51010 172.217.21.46:443 ESTABLISHED 11184
TCP [::]:135 [::]:0 LISTENING 1152
TCP [::]:445 [::]:0 LISTENING 4
TCP [::]:2179 [::]:0 LISTENING 4696
TCP [::]:5357 [::]:0 LISTENING 4
TCP [::]:7680 [::]:0 LISTENING 7056
TCP [::]:17500 [::]:0 LISTENING 9668
TCP [::]:49664 [::]:0 LISTENING 784
TCP [::]:49665 [::]:0 LISTENING 1628
TCP [::]:49666 [::]:0 LISTENING 2028
TCP [::]:49667 [::]:0 LISTENING 3560
TCP [::]:49800 [::]:0 LISTENING 856
TCP [::]:49821 [::]:0 LISTENING 892
TCP [::1]:5432 [::]:0 LISTENING 11100
TCP [::1]:50000 [::]:0 LISTENING 11100
TCP [::1]:50001 [::]:0 LISTENING 11100
TCP [::1]:51000 [::]:0 LISTENING 11100
UDP 0.0.0.0:53 *:* 5620
UDP 0.0.0.0:3702 *:* 2084
UDP 0.0.0.0:3702 *:* 2084
UDP 0.0.0.0:5050 *:* 6616
UDP 0.0.0.0:5353 *:* 11184
UDP 0.0.0.0:5353 *:* 11184
UDP 0.0.0.0:5353 *:* 11184
UDP 0.0.0.0:5353 *:* 3080
UDP 0.0.0.0:5353 *:* 11184
UDP 0.0.0.0:5353 *:* 11184
UDP 0.0.0.0:5353 *:* 11184
UDP 0.0.0.0:5353 *:* 11184
UDP 0.0.0.0:5355 *:* 3080
UDP 0.0.0.0:17500 *:* 9668
UDP 0.0.0.0:49670 *:* 2084
UDP 0.0.0.0:57329 *:* 5620
UDP 0.0.0.0:57330 *:* 5620
UDP 0.0.0.0:59529 *:* 8620
UDP 0.0.0.0:60605 *:* 11184
UDP 10.0.75.1:137 *:* 4
UDP 10.0.75.1:138 *:* 4
UDP 10.0.75.1:1900 *:* 2620
UDP 10.0.75.1:61326 *:* 2620
UDP 127.0.0.1:1900 *:* 2620
UDP 127.0.0.1:60816 *:* 4616
UDP 127.0.0.1:61328 *:* 2620
UDP 169.254.105.83:137 *:* 4
UDP 169.254.105.83:138 *:* 4
UDP 169.254.105.83:1900 *:* 2620
UDP 169.254.105.83:61330 *:* 2620
UDP 172.30.146.241:67 *:* 5620
UDP 172.30.146.241:68 *:* 5620
UDP 172.30.146.241:1900 *:* 2620
UDP 172.30.146.241:61329 *:* 2620
UDP 192.168.0.107:137 *:* 4
UDP 192.168.0.107:138 *:* 4
UDP 192.168.0.107:1900 *:* 2620
UDP 192.168.0.107:61327 *:* 2620
UDP [::]:3702 *:* 2084
UDP [::]:3702 *:* 2084
UDP [::]:5353 *:* 11184
UDP [::]:5353 *:* 11184
UDP [::]:5353 *:* 3080
UDP [::]:5353 *:* 11184
UDP [::]:5355 *:* 3080
UDP [::]:49671 *:* 2084
UDP [::]:57331 *:* 5620
UDP [::]:59529 *:* 8620
UDP [::1]:1900 *:* 2620
UDP [::1]:61323 *:* 2620
UDP [fe80::30eb:ad8f:f94a:b774%26]:1900 *:* 2620
UDP [fe80::30eb:ad8f:f94a:b774%26]:61324 *:* 2620
UDP [fe80::718c:22bb:fd97:c06c%23]:1900 *:* 2620
UDP [fe80::718c:22bb:fd97:c06c%23]:61322 *:* 2620
UDP [fe80::85d0:3b5c:7746:6953%5]:1900 *:* 2620
UDP [fe80::85d0:3b5c:7746:6953%5]:61325 *:* 2620
The f# code I used to test for open ports:
open System.Net
open System.Net.Sockets
let ipAddress = IPAddress([| (byte)0; (byte)0; (byte)0; (byte)0 |])
let ipEndpoint portNumber = (IPEndPoint(ipAddress, portNumber), portNumber)
let getPorts = seq { for i in 1 .. 65535 -> i }
let checkIfPortAvailable (endpoint, portNumber) =
use listener = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp)
try
listener.Bind(endpoint)
(portNumber, true, null)
with
| ex -> (portNumber, false, ex)
[<EntryPoint>]
let main argv =
getPorts
|> Seq.map ipEndpoint
|> Seq.map checkIfPortAvailable
|> Seq.where (fun (_, works, _) -> not works)
|> Seq.where (fun (_, _, ex) -> ex.Message.Contains("An attempt was made to access a socket in a way forbidden by its access permissions"))
|> Seq.iteri (fun index (port, _, _) -> printfn "%5d: %d" index port)
0
and the output when executing the application (it was executed right after the netstat command):
0: 445
1: 5357
2: 7680
3: 49670
4: 49671
5: 49672
6: 49673
7: 49674
8: 49675
9: 49676
10: 49677
11: 49678
12: 49679
13: 49680
14: 49681
15: 49682
16: 49683
17: 49684
18: 49685
19: 49686
20: 49687
21: 49688
22: 49689
23: 49710
24: 49711
25: 49712
26: 49713
27: 49714
28: 49715
29: 49716
30: 49717
31: 49718
32: 49719
33: 49720
34: 49721
35: 49722
36: 49723
37: 49724
38: 49725
39: 49726
40: 49727
41: 49728
42: 49729
43: 49730
44: 49731
45: 49732
46: 49733
47: 49734
48: 49735
49: 49736
50: 49737
51: 49738
52: 49739
53: 49740
54: 49741
55: 49742
56: 49743
57: 49744
58: 49745
59: 49746
60: 49747
61: 49748
62: 49749
63: 49750
64: 49751
65: 49752
66: 49753
67: 49754
68: 49755
69: 49756
70: 49757
71: 49758
72: 49759
73: 49770
74: 49771
75: 49772
76: 49773
77: 49774
78: 49775
79: 49776
80: 49777
81: 49778
82: 49779
83: 49780
84: 49781
85: 49782
86: 49783
87: 49784
88: 49785
89: 49786
90: 49787
91: 49788
92: 49789
93: 49790
94: 49791
95: 49792
96: 49793
97: 49794
98: 49795
99: 49796
100: 49797
101: 49798
102: 49799
103: 49825
104: 49826
105: 49827
106: 49828
107: 49829
108: 49830
109: 49831
110: 49832
111: 49833
112: 49834
113: 49835
114: 49836
115: 49837
116: 49838
117: 49839
118: 49840
119: 49841
120: 49842
121: 49843
122: 49844
123: 49845
124: 49846
125: 49847
126: 49848
127: 49849
128: 49850
129: 49851
130: 49852
131: 49853
132: 49854
133: 49855
134: 49856
135: 49857
136: 49858
137: 49859
138: 49860
139: 49861
140: 49862
141: 49863
142: 49864
143: 49866
144: 49867
145: 49868
146: 49869
147: 49870
148: 49871
149: 49872
150: 49873
151: 49874
152: 49875
153: 49876
154: 49877
155: 49878
156: 49879
157: 49880
158: 49881
159: 49882
160: 49883
161: 49884
162: 49885
163: 50274
164: 50275
165: 50276
166: 50277
167: 50278
168: 50279
169: 50280
170: 50281
171: 50282
172: 50283
173: 50284
174: 50285
175: 50286
176: 50287
177: 50288
178: 50289
179: 50290
180: 50291
181: 50292
182: 50293
The port which I'd actually want to have opened is 49786 but the amount of ports reported with permission denied by my small testing algorithm, but not allocated when checking netstat is confusing me
UPDATE 3:
Now that simply running shutdown /s /f /t 0 doesn't work anymore, I've found another workaround which seems to be working fairly reliably in https://social.technet.microsoft.com/Forums/windows/en-US/e0cc1b3e-2c63-451a-abc8-8e1e66a23287/an-attempt-was-made-to-access-a-socket-in-a-way-forbidden-by-its-access-permissions?forum=w7itpronetworking - netsh int ip reset and restarting the computer (normal windows 10 restart is enough) and the port was consistently available on the next start so far (but it was just 3 tries)
UPDATE 4:
Summary of the comments:
setting SO_REUSEADDR in the small f# script makes things only worse
trying to close skype, VPN applications, dropbox and similar didn't have any effect (yes I actually made sure that the processes are terminated)

The reason is that Hyper-V takes over these ports, to prevent it from happening do the following:
dism.exe /Online /Disable-Feature:Microsoft-Hyper-V (will have to restart)
netsh int ipv4 add excludedportrange protocol=tcp startport=<your port> numberofports=1
dism.exe /Online /Enable-Feature:Microsoft-Hyper-V /All
The original solution is here.
If after that you still can't bind to the port do the following:
netsh http add iplisten 0.0.0.0 for IPv4
netsh http add iplisten :: for IPv6
More information is here.

I also had the Hyper-v/WSL problem above, but found a simpler solution:
I had stopped docker before doing this, not sure if it's necessary.
Run as admin:
net stop winnat
net start winnat

A faulty Windows upgrade is the reason for the problems with reserved tcp ports.
See https://github.com/docker/for-win/issues/3171#issuecomment-554587817
With this the problem should be fixed:
netsh int ipv4 set dynamic tcp start=49152 num=16384
That would explain why a clean install fixes these type of problems.

If you face this issue on Win10 2004 that's because of of an issue in this update do the following
netsh int ipv[46] set dynamic tcp start=49152 num=16384
reg add HKLM\SYSTEM\CurrentControlSet\Services\hns\State /v EnableExcludedPortRange /d 0 /f
I face this on opening Jetbrains IDEs and and many other program that use sockets

In case someone lands up here after facing similar issues post updating Windows to May 2020 update which contains WSL2. In my case I also switched docker to WSL2 post which I started facing similar issues with some of my containers.
Note: Run the following commands in powershell (Run as administrator)
Disable Hyper-V.
dism.exe /Online /Disable-Feature:Microsoft-Hyper-V
Remove all hyper-v related network adapters.
Get-HNSNetwork | Remove-HNSNetwork
Reboot.

I am having a similar issue right now(August 2021) and this GitHub issue has some insight into the problem. It seems
net stop winnat
net start winnat
Should fix the issue, you might need to start your application before restarting winnat

Expanding on the selected answer: if you have Hyper-V enabled, you can temporarily disable it during boot by using bcdedit:
bcdedit /Set {current} hypervisorlaunchtype off
Restart and exclude the necessary ports via:
netsh int ipv4 add excludedportrange protocol=tcp startport=<your port> numberofports=1
Then simply re-enable Hyper-V, then restart:
bcdedit /Set {current} hypervisorlaunchtype auto
This prevents having to completely remove and re-add the Hyper-V feature, which requires several restarts.

When I looked for it, the explanation was
An attempt was made to access a socket in a way forbidden by its access permissions. An example is using a broadcast address for sendto without broadcast permission being set using setsockopt(SO_BROADCAST).
Another possible reason for the WSAEACCES error is that when the bind function is called (on Windows NT 4.0 with SP4 and later), another application, service, or kernel mode driver is bound to the same address with exclusive access. Such exclusive access is a new feature of Windows NT 4.0 with SP4 and later, and is implemented by using the SO_EXCLUSIVEADDRUSE option.

Cloudera: Installation failed. Failed to receive heartbeat on CDH 5.14 cluster installation on singleNode from agents

I am trying to install cloudera CDH 5.14.0. and facing below error during Cluster Installation.
Installation failed. Failed to receive heartbeat from agent.
Ensure that the host's hostname is configured properly.
Ensure that port 7182 is accessible on the Cloudera Manager Server (check firewall rules).
Ensure that ports 9000 and 9001 are not in use on the host being added.
Check agent logs in /var/log/cloudera-scm-agent/ on the host being added. (Some of the logs can be found in the installation details).
If Use TLS Encryption for Agents is enabled in Cloudera Manager (Administration -> Settings -> Security), ensure that /etc/cloudera-scm-agent/config.ini has use_tls=1 on the host being added. Restart the corresponding agent and click the Retry link here.
Detail Log:
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO Neither verify_cert_file nor verify_cert_dir are configured. Not performing validation of server certificates in HTTPS communication. These options can be configured in this agent's config.ini file to enable certificate validation.
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO Agent Logging Level: INFO
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO No command line vars
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO Found database jar: /usr/share/java/mysql-connector-java.jar
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO Missing database jar: /usr/share/java/oracle-connector-java.jar (normal, if you're not using this database type)
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO Found database jar: /usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO Agent starting as pid 12771 user root(0) group root(0).
END (0)
end of agent logs.
scm agent restarted
Local ubuntu host details:
chaithu#chaithu:~$ hostname
chaithu
chaithu#chaithu:~$ hostname -f
chaithu
chaithu#chaithu:~$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 chaithu
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
chaithu#chaithu:~$ sudo netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:7180 0.0.0.0:* LISTEN 1353/java
tcp 0 0 0.0.0.0:7182 0.0.0.0:* LISTEN 1353/java
tcp 0 0 0.0.0.0:4433 0.0.0.0:* LISTEN 12930/python2.7
tcp 0 0 127.0.1.1:53 0.0.0.0:* LISTEN 1162/dnsmasq
tcp 0 0 127.0.0.1:7190 0.0.0.0:* LISTEN 12930/python2.7
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1031/sshd
tcp 0 0 0.0.0.0:7191 0.0.0.0:* LISTEN 12930/python2.7
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 5112/cupsd
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN 1077/postgres
tcp 0 0 127.0.0.1:19001 0.0.0.0:* LISTEN 1632/python
tcp 0 0 127.0.1.1:9000 0.0.0.0:* LISTEN 12771/python2.7
tcp 0 0 0.0.0.0:7432 0.0.0.0:* LISTEN 1434/postgres
tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 1045/mysqld
tcp6 0 0 :::80 :::* LISTEN 1441/apache2
tcp6 0 0 :::4434 :::* LISTEN 12930/python2.7
tcp6 0 0 :::22 :::* LISTEN 1031/sshd
tcp6 0 0 :::7191 :::* LISTEN 12930/python2.7
tcp6 0 0 ::1:631 :::* LISTEN 5112/cupsd
tcp6 0 0 :::7432 :::* LISTEN 1434/postgres
udp 0 0 0.0.0.0:57656 0.0.0.0:* 902/avahi-daemon: r
udp 0 0 0.0.0.0:631 0.0.0.0:* 5113/cups-browsed
udp 0 0 0.0.0.0:5353 0.0.0.0:* 2703/chrome
udp 0 0 0.0.0.0:5353 0.0.0.0:* 902/avahi-daemon: r
udp 0 0 0.0.0.0:39330 0.0.0.0:* 1162/dnsmasq
udp 0 0 0.0.0.0:7191 0.0.0.0:* 12930/python2.7
udp 0 0 127.0.1.1:53 0.0.0.0:* 1162/dnsmasq
udp 0 0 0.0.0.0:68 0.0.0.0:* 1117/dhclient
udp6 0 0 :::5353 :::* 2703/chrome
udp6 0 0 :::5353 :::* 902/avahi-daemon: r
udp6 0 0 :::46382 :::* 902/avahi-daemon: r
udp6 0 0 :::7191 :::* 12930/python2.7
I am trying to install CDH only on single node local machine. Do let me know if any more info required.

Why hdfs namenode and datanode both always listen on 0.0.0.0 at a random port?

I installed a new hadoop-2.2.0 today, I found after hdfs is started (using /sbin/start-dfs.sh), namenode and datanode both always listen on 0.0.0.0 at a random port? I cannot find relate configuration on http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml.
The port is not 50070, 50470, 50090, 50010, 50020, 50075, 50475...etc. which are listed on hdfs-default.xml, it is just a random port.
8369 Jps
8109 DataNode
7936 NameNode
Namenode listens on the followings:
tcp 0 0 0.0.0.0:46628 0.0.0.0:* LISTEN 7936/java <==
tcp 0 0 10.173.130.119:9000 0.0.0.0:* LISTEN 7936/java
tcp 0 0 10.173.130.119:50070 0.0.0.0:* LISTEN 7936/java
Datanode listens on the followings:
tcp 0 0 10.173.130.119:50020 0.0.0.0:* LISTEN 8109/java
tcp 0 0 0.0.0.0:35114 0.0.0.0:* LISTEN 8109/java <==
tcp 0 0 10.173.130.119:50010 0.0.0.0:* LISTEN 8109/java
tcp 0 0 10.173.130.119:50075 0.0.0.0:* LISTEN 8109/java
Thanks for any advise.

Yes It assigns random port everytime a namenode or datanode restarted. But if you observer all the namenode listeners running on same process id in this case (7936) and Datanode listeners run on same process id ie 8109. So internally process is same.

Hadoop2.2.0 can't visit the web http://<ip>:8088

I setup a hadoop cluster with two nodes hadoop01(master- 10.0.0.151) and hadoop02(slaves- 10.0.0.152)
when a type start-dfs.sh and then visit website
my_ip(just 10.0.0.151 above):50070. It's successful.
But when i type start-yarn.sh then visit website my_ip:8088. It's failed.
my yarn-site.xml:
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>hadoop01:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>hadoop01:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>hadoop01:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>hadoop01:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>hadoop01:8088</value>
</property>
core-site.xml:
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop01:8020</value>
</property>
</configuration>
the same set as hadoop02(slave)
my hadoop01 hadoop-2.2.0/etc/hadoop/slaves set
hadoop01
hadoop02
After typing start-dfs.sh & start-yarn.sh then i type jps
hadoop01:
21594 NameNode
22345 NodeManager
22007 SecondaryNameNode
22171 ResourceManager
23147 Jps
21762 DataNode
hadoop02:
29861 NodeManager
30358 Jps
29665 DataNode
my /etc/hosts in hadoop01:
localhost hadoop01
10.0.0.151 hadoop01
10.0.0.152 hadoop02
my /etc/hosts in hadoop02:
localhost hadoop02
10.0.0.151 hadoop01
10.0.0.152 hadoop02
This bellow link is my yarn-nodemanager.log I upload to google dirve
https://drive.google.com/file/d/0B7nCJ_XJWSrQN1BZVTVyOEgxd1E/edit?usp=sharing
but yarn-nodemanager.log doesn't appear ERROR if i didn't miss some information..
Please help me improve the problem why I can't visit the website http://10.0.0.151:8088
**if need other information (such as hdfs-site.xml...etc) just tell me. I'll update..
netstat -tunalp | grep LISTEN
tcp 0 0 0.0.0.0:50010 0.0.0.0:* LISTEN 17442/java
tcp 0 0 0.0.0.0:50075 0.0.0.0:* LISTEN 17442/java
tcp 0 0 0.0.0.0:50020 0.0.0.0:* LISTEN 17442/java
tcp 0 0 0.0.0.0:50090 0.0.0.0:* LISTEN 17693/java
tcp 0 0 10.0.0.151:8020 0.0.0.0:* LISTEN 17267/java
tcp 0 0 0.0.0.0:50070 0.0.0.0:* LISTEN 17267/java
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp6 0 0 :::13562 :::* LISTEN 21061/java
tcp6 0 0 10.0.0.151:8030 :::* LISTEN 20881/java
tcp6 0 0 10.0.0.151:8031 :::* LISTEN 20881/java
tcp6 0 0 10.0.0.151:8032 :::* LISTEN 20881/java
tcp6 0 0 10.0.0.151:8033 :::* LISTEN 20881/java
tcp6 0 0 :::33762 :::* LISTEN 21061/java
tcp6 0 0 :::8040 :::* LISTEN 21061/java
tcp6 0 0 :::8042 :::* LISTEN 21061/java
tcp6 0 0 :::22 :::* LISTEN -
tcp6 0 0 10.0.0.151:8088 :::* LISTEN 20881/java
After disable ipv6 , then i type netstat -tunalp | grep LISTEN
tcp 0 0 0.0.0.0:13562 0.0.0.0:* LISTEN 30608/java
tcp 0 0 0.0.0.0:50010 0.0.0.0:* LISTEN 29967/java
tcp 0 0 0.0.0.0:50075 0.0.0.0:* LISTEN 29967/java
tcp 0 0 10.0.0.151:8030 0.0.0.0:* LISTEN 30424/java
tcp 0 0 10.0.0.151:8031 0.0.0.0:* LISTEN 30424/java
tcp 0 0 0.0.0.0:52992 0.0.0.0:* LISTEN 30608/java
tcp 0 0 10.0.0.151:8032 0.0.0.0:* LISTEN 30424/java
tcp 0 0 10.0.0.151:8033 0.0.0.0:* LISTEN 30424/java
tcp 0 0 0.0.0.0:50020 0.0.0.0:* LISTEN 29967/java
tcp 0 0 0.0.0.0:8040 0.0.0.0:* LISTEN 30608/java
tcp 0 0 0.0.0.0:8042 0.0.0.0:* LISTEN 30608/java
tcp 0 0 0.0.0.0:50090 0.0.0.0:* LISTEN 30222/java
tcp 0 0 10.0.0.151:8020 0.0.0.0:* LISTEN 29790/java
tcp 0 0 0.0.0.0:50070 0.0.0.0:* LISTEN 29790/java
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp 0 0 10.0.0.151:8088 0.0.0.0:* LISTEN 30424/java
tcp6 0 0 :::22 :::* LISTEN -

the reason is you should change 10.0.0.151:8088 to 0.0.0.0:8088. the method is open the yarn-site.xml, add:
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>0.0.0.0:8088</value>
</property>

Here the problem is resouce manager is running, but the port(8030,8031,8032,8033,8088) occupied by resource manager uses tcp6 instead of tcp(see the left portion). You have two options either you can disable ipv6 in the linux system then restart yarn services.
or
Try modify your yarn-site.xml only on master node as follows. Don't modify the yarn-site.xml in slave nodes:
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<configuration>
Specifying hostname causes the ports started as tcp6, even with out specifying those ports it would take the default value. Have a look at the following default ports
http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
After modiying yarn-site.xml restart your yarn-service

For me this from Hadoop: binding multiple IP addresses to a cluster NameNode:
In hdfs-site.xml, set the value of dfs.namenode.rpc-bind-host to
0.0.0.0 and Hadoop will listen on both the private and public network interfaces allowing remote access and datanode access.
and opening 8088 in the firewall settings worked.

This is related to https://issues.apache.org/jira/browse/HADOOP-605 ; Hadoop configuration scripts have added the -Djava.net.preferIPv4Stack=true flag to force IPv4 binding, but this is missing from the Yarn configuration script. You can fix this by adding at the end of bin/yarn (before the exec):
YARN_OPTS="$YARN_OPTS -Djava.net.preferIPv4Stack=true"

Hadoop dfs error : INFO ipc.Client: Retrying connect to server: localhost

I have successfully setup a Hadoop cluster with 6 nodes (master, salve<1-5>)
Formatted the namenode -> done
Starting up and shutting down cluster -> works fine
Executing "hadoop dfs -ls /" gives this error -> Error: INFO ipc.Client: Retrying connect to server: localhost
I tried to see the services running using:
sudo netstat -plten | grep java
hduser#ubuntu:~$ sudo netstat -plten | grep java
tcp 0 0 0.0.0.0:50070 0.0.0.0:* LISTEN 1000 93307 11384/java
tcp 0 0 0.0.0.0:44440 0.0.0.0:* LISTEN 1000 92491 11571/java
tcp 0 0 0.0.0.0:40633 0.0.0.0:* LISTEN 1000 92909 11758/java
tcp 0 0 0.0.0.0:50010 0.0.0.0:* LISTEN 1000 93449 11571/java
tcp 0 0 0.0.0.0:50075 0.0.0.0:* LISTEN 1000 93673 11571/java
tcp 0 0 0.0.0.0:50020 0.0.0.0:* LISTEN 1000 93692 11571/java
tcp 0 0 127.0.0.1:40485 0.0.0.0:* LISTEN 1000 93666 12039/java
tcp 0 0 0.0.0.0:44582 0.0.0.0:* LISTEN 1000 93013 11852/java
tcp 0 0 10.42.43.1:54310 0.0.0.0:* LISTEN 1000 92471 11384/java
tcp 0 0 10.42.43.1:54311 0.0.0.0:* LISTEN 1000 93290 11852/java
tcp 0 0 0.0.0.0:50090 0.0.0.0:* LISTEN 1000 93460 11758/java
tcp 0 0 0.0.0.0:34154 0.0.0.0:* LISTEN 1000 92179 11384/java
tcp 0 0 0.0.0.0:50060 0.0.0.0:* LISTEN 1000 94200 12039/java
tcp 0 0 0.0.0.0:50030 0.0.0.0:* LISTEN 1000 93550 11852/java
Its the master IP binded to port 54310 and 54311 and not the localhost(loopback).
The conf-site.xml has been properly configured:
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hduser/hadoop/tmp</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://master:54310</value>
</property>
</configuration>
Why is it expecting localhost to be binded to 54310 rather than the master hich I have configured here. Help appreciated. How do I resolve this ??
Cheers

Apparently, someone added the older hadoop(1.0.3) bin directory into the path variable before I had added the new hadoop(1.0.4) bin directory. And thus whenever I ran "hadoop" from the CLI, it executed the binaries of the older hadoop rather than the new one.
Solution:
Remove the entire bin path of older hadoop
Shutdown cluster - Exit terminal
Login in new terminal session
Startup node
Tried hadoop dfs -ls / -> Works fine !!!! Good lesson learnt.

Looks like many people ran into this problem.
There might be no need to change /etc/hosts, and make sure you can access master and slave from each other, and your core-site.xml are the same pointing to the right master node and port number.
Then run $HADOOP/bin/stop-all.sh, $HADOOP/bin/start-all.sh on master node ONLY. (If run on slave might lead to problems). Use JPS to check whether all services are there as follows.
On master node:
4353 DataNode
4640 JobTracker
4498 SecondaryNameNode
4788 TaskTracker
4989 Jps
4216 NameNode
On slave node:
3143 Jps
2827 DataNode
2960 TaskTracker

In addition, check your firewall rules between namenode and datanode

Resources