Configure Outlier detection for consul service mesh using nomad job - consul

Question

I am trying to configure Outlier Detection for a consul connect service mesh based on this documentation.
https://learn.hashicorp.com/tutorials/consul/service-mesh-circuit-breaking?in=consul/developer-mesh
The documentation shows that Outlier Detection and Circuit breaking can be configured using the config stanza inside proxy.upstreams. But the following job file throws error - Blocks of type "config" are not expected here.
job "docs" {
datacenters = ["dc1"]
group "docs" {
network {
mode = "bridge"
}
service {
name = "docs"
port = "5678"
connect {
sidecar_service {
proxy {
upstreams {
destination_name = "demo"
local_bind_port = 10082
config {
connect_timeout_ms = 3000
}
}
}
}
}
}
task "server" {
driver = "docker"
config {
image = "hashicorp/http-echo"
args = [
"-listen",
":5678",
"-text",
"hello world",
]
}
}
}
}
Am I doing anything wrong? Is this not the right way to configure circuit breaking in nomad job file?

Answer 1

sidecar Proxy, Circuit breaking, ingress, egress must be implemented with consul directly and not from nomad. Also, In your job you didn't map the port inside docker and outside port. consul work a specific version of envoy load balacner.
First launch your job without connect stanza and do port mapping
install envoy and do proxy connect connection manually to test
once test work make a service proxy to launch your sidecar your circuit breaking
1- Launching job: (by exemple your port inside docker is 8080 )
job "docs" {
datacenters = ["dc1"]
group "docs" {
network {
mode = "bridge"
}
task "server" {
driver = "docker"
config {
image = "hashicorp/http-echo"
args = [
"-listen",
":5678",
"-text",
"hello world",
]
port_map {
docs = 8080
}
}
resources {
network {
mbits = 10
port "docs" { static = 5678 }
}
}
service {
name = "docs"
port = "docs"
check {
name = "docs port alive"
type = "http"
path = "/"
interval = "10s"
timeout = "2s"
}
}
}
}
}
2-check your consul version and install supported envoy version here. i use consul 1.11 so i will install supported envoy 1.18.4
yum -y -q install tar
curl https://func-e.io/install.sh | bash -s -- -b /usr/local/bin
func-e use 1.18.4
make the envoy bin available
cp /root/.func-e/versions/1.18.4/bin/envoy /usr/local/bin/
Proxy integration
insert at your end of consul config .for me my config are stored in
/etc/consul.d/config.hcl
config_entries {
bootstrap = [
{
kind = "proxy-defaults"
name = "global"
config {
protocol = "http"
}
}
]
}
**restart your consul service to check if envoy proxy integration worked**
systemctl restart consul
Overwrite your service registration in consul with consul file :
cat > /etc/consul.d/docs.hcl <<- EOF
service {
name = "docs"
port = 5678
#token = "" # put api service token here
check {
id = "docs"
name = "HTTP API on Port 5678"
http = "http://localhost:5678"
interval = "30s"
}
connect {
sidecar_service {
port = 20000
check {
name = "Connect Envoy Sidecar"
tcp = "127.0.0.1:20000"
interval = "10s"
}
}
}
}
EOF
restart service consul or reload it
systemctl restart consul
Test proxy side car working
consul connect envoy -sidecar-for=docs
create docs service proxy Create at /etc/systemd/system/consul-envoy-docs.service and input the following:
cat > /etc/systemd/system/consul-envoy.service <<- EOF
[Unit]
Description=Consul Envoy
After=syslog.target network.target
[Service]
ExecStart=/usr/local/bin/consul connect envoy -sidecar-for=docs
ExecStop=/bin/sleep 5
Restart=always
[Install]
WantedBy=multi-user.target
EOF
Restart consul and start consul-envoy:
systemctl daemon-reload
systemctl restart consul
systemctl start consul-envoy-docs
In the event that consul-envoy fails, restart it with:
systemctl restart consul-envoy
3. Well if all work correctly , adapt conf file in /etc/systemd/system/consul-envoy-docs.service as described here to make circuit breaking
If someone have issue with nomad , consul , vault , envoy or hashistack tag me