DNS configuration for accessing consul remotely - amazon-ec2

Question

I have installed consul on AWS EC2, with 3 servers and 1 client.
server IPs = 11.XX.XX.1,11.XX.XX.2,11.XX.XX.3.
client IP = 11.XX.XX.4
consul config: /etc/consul.d/server/config.json
{
"bootstrap": false,
"server": true,
"datacenter": "abc",
"advertise_addr": "11.XX.XX.1",
"data_dir": "/var/consul",
"log_level": "INFO",
"enable_syslog": true,
"addresses": {
"http": "0.0.0.0"
},
"start_join": ["11.XX.XX.2", "11.XX.XX.3"]
}
netstat output on server:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:8400 0.0.0.0:* LISTEN 29720/consul
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1006/sshd
tcp 0 0 127.0.0.1:8600 0.0.0.0:* LISTEN 29720/consul
tcp6 0 0 :::8301 :::* LISTEN 29720/consul
tcp6 0 0 :::8302 :::* LISTEN 29720/consul
tcp6 0 0 :::8500 :::* LISTEN 29720/consul
tcp6 0 0 :::22 :::* LISTEN 1006/sshd
tcp6 0 0 :::8300 :::* LISTEN 29720/consul
curl is working fine from remote machine but dig is only working on the local machine.
; <<>> DiG 9.9.5-3ubuntu0.6-Ubuntu <<>> #127.0.0.1 -p 8600 web.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40873
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;web.service.consul. IN A
;; ANSWER SECTION:
web.service.consul. 0 IN A 11.XX.XX.4
;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Fri Dec 30 08:21:41 UTC 2016
;; MSG SIZE rcvd: 52
but dig is not working from remote machine:
dig #11.XX.XX.1 -p 8600 web.service.consul
; <<>> DiG 9.9.5-3ubuntu0.6-Ubuntu <<>> #11.XX.XX.1 -p 8600 web.service.consul
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
-----------------------------
How to make it work?

Answer 1

By default consul only listens for DNS connections on the instance loopback device. Best practices asks you to install the client on any remote machine looking to consume consul DNS. This is not always practical.
I have seen people expose DNS (consul port 8600) on all interfaces via the Consul configuration JSON like so:
{
"server": true,
"addresses": {
"dns": "0.0.0.0"
}
}
You can also expose all ports listening on loopback with the client_addr field in JSON or pass it via the command line with:
consul agent -client 0.0.0.0
There are more controls and knobs available to tweak (see docs):
https://www.consul.io/docs/agent/options.html