I am trying to setup a Service Mesh PoC, I have three micro-services each running with their sidecar-proxy (through envoy). I ran the following command to launch the proxies :
consul connect envoy -sidecar-for <CONSUL_SERVICE_ID> -admin-bind 127.0.0.1:19000 -http-addr http://127.0.0.1:8500 -grpc-addr 127.0.0.1:8502
The problem is that my sidecar-proxy starts a public listener on port 21002 (don't know this port name comes from, the configuration files for envoy are nowhere to be found) and it is unreachable. This causes my sidecar health check to fail and therefore my service redirection to fail.
[2020-06-16 15:02:30.672][24383][debug][config] [external/envoy/source/server/filter_chain_manager_impl.cc:214] new fc_contexts has 1 filter chains, including 1 newly built
[2020-06-16 15:02:30.672][24383][debug][init] [external/envoy/source/common/init/target_impl.cc:15] init manager Server initializing target Listener-init-target public_listener:10.26.57.59:21000
[2020-06-16 15:02:30.672][24383][debug][init] [external/envoy/source/common/init/manager_impl.cc:45] init manager Listener-local-init-manager public_listener:10.26.57.59:21000 5712408582249607733 contains no targets
[2020-06-16 15:02:30.672][24383][debug][init] [external/envoy/source/common/init/watcher_impl.cc:14] init manager Listener-local-init-manager public_listener:10.26.57.59:21000 5712408582249607733 initialized, notifying Listener-local-init-watcher public_listener:10.26.57.59:21000
[2020-06-16 15:02:30.672][24383][debug][init] [external/envoy/source/common/init/watcher_impl.cc:14] target Listener-init-target public_listener:10.26.57.59:21000 initialized, notifying init manager Server
[2020-06-16 15:02:30.672][24383][debug][config] [external/envoy/source/server/listener_impl.cc:80] Create listen socket for listener public_listener:10.26.57.59:21000 on address 10.26.57.59:21000
[2020-06-16 15:02:30.672][24383][debug][config] [external/envoy/source/server/listener_impl.cc:70] Set listener public_listener:10.26.57.59:21000 socket factory local address to 10.26.57.59:21000
[2020-06-16 15:02:30.672][24383][debug][config] [external/envoy/source/server/listener_impl.cc:508] add active listener: name=public_listener:10.26.57.59:21000, hash=5712408582249607733, address=10.26.57.59:21000
[2020-06-16 15:02:30.672][24383][info][upstream] [external/envoy/source/server/lds_api.cc:76] lds: add/update listener 'public_listener:10.26.57.59:21000'
[2020-06-16 15:02:30.672][24383][warning][misc] [external/envoy/source/common/protobuf/utility.cc:198] Using deprecated option 'envoy.api.v2.listener.Filter.config' from file listener_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
[2020-06-16 15:02:30.672][24383][debug][config] [external/envoy/source/server/listener_manager_impl.cc:386] begin add/update listener: name=javatestrs-microc-cicdev:127.0.0.1:6610 hash=14335360969741422718
Do you have any idea on accessing this envoy public listener ?
sidecar log :
[2020-06-16 15:02:30.672][24383][debug][config] [external/envoy/source/server/filter_chain_manager_impl.cc:214] new fc_contexts has 1 filter chains, including 1 newly built
[2020-06-16 15:02:30.672][24383][debug][init] [external/envoy/source/common/init/target_impl.cc:15] init manager Server initializing target Listener-init-target public_listener:10.26.57.59:21000
[2020-06-16 15:02:30.672][24383][debug][init] [external/envoy/source/common/init/manager_impl.cc:45] init manager Listener-local-init-manager public_listener:10.26.57.59:21000 5712408582249607733 contains no targets
[2020-06-16 15:02:30.672][24383][debug][init] [external/envoy/source/common/init/watcher_impl.cc:14] init manager Listener-local-init-manager public_listener:10.26.57.59:21000 5712408582249607733 initialized, notifying Listener-local-init-watcher public_listener:10.26.57.59:21000
[2020-06-16 15:02:30.672][24383][debug][init] [external/envoy/source/common/init/watcher_impl.cc:14] target Listener-init-target public_listener:10.26.57.59:21000 initialized, notifying init manager Server
[2020-06-16 15:02:30.672][24383][debug][config] [external/envoy/source/server/listener_impl.cc:80] Create listen socket for listener public_listener:10.26.57.59:21000 on address 10.26.57.59:21000
[2020-06-16 15:02:30.672][24383][debug][config] [external/envoy/source/server/listener_impl.cc:70] Set listener public_listener:10.26.57.59:21000 socket factory local address to 10.26.57.59:21000
[2020-06-16 15:02:30.672][24383][debug][config] [external/envoy/source/server/listener_impl.cc:508] add active listener: name=public_listener:10.26.57.59:21000, hash=5712408582249607733, address=10.26.57.59:21000
[2020-06-16 15:02:30.672][24383][info][upstream] [external/envoy/source/server/lds_api.cc:76] lds: add/update listener 'public_listener:10.26.57.59:21000'
[2020-06-16 15:02:30.672][24383][warning][misc] [external/envoy/source/common/protobuf/utility.cc:198] Using deprecated option 'envoy.api.v2.listener.Filter.config' from file listener_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
[2020-06-16 15:02:30.672][24383][debug][config] [external/envoy/source/server/listener_manager_impl.cc:386] begin add/update listener: name=javatestrs-microc-cicdev:127.0.0.1:6610 hash=14335360969741422718
One thing that's very helpful to debug your Envoy proxies is the admin port. You can get the active listeners, clusters, and even a full config dump from that. Without using consul-connect, i can only guess a bit from your output, but I'd start with port 19000.
Exec into whatever host is running the Proxy, and try curl localhost:19000/listeners and see if it responds (or any other admin path). If it responds you can get the full /config_dump which will describe the entire setup for that sidecar: what ports it's listening on, whether they're TLS enabled, what paths they match on, etc etc.
The public listener port is auto-allocated by Consul to the sidecar from a default range (21000 - 21255). It is used to receive mTLS connections from other proxies in the mesh. The range can be defined in the Consul agent's configuration under the ports {} stanza.
ports {
sidecar_min_port = 30000
sidecar_max_port = 31000
}
See https://www.consul.io/docs/agent/options#sidecar_min_port for the specific documentation.
You can select a specific port using the port parameter in the sidecar service definition.
{
"service": {
"name": "web",
"port": 8080,
"connect": {
"sidecar_service": {
"port": 31000
}
}
}
}
Related
I am working on a project where I want to configure envoy using consul as the xds server. But I don't want to use consul connect. I only want to use consul for service discovery and the xds server.
I have started the consul agent with -dev mode enabling the grpc endpoints and I have registered a service hello-service which i can see in the consul ui.
My barebone envoy config
# admin web panel
admin:
access_log_path: ./admin_logs
address:
socket_address:
address: 0.0.0.0
port_value: 9902
dynamic_resources:
cds_config:
ads: {}
ads_config:
api_type: grpc
transport_api_version: v3
grpc_services:
- google_grpc:
target_uri: http://localhost:8502
stat_prefix: grpc-xds-service
I am trying to fetch some configuration like clusters/endpoints from consul for the registered hello-service, But when i start the envoy process I am getting this warning.
[2021-07-11 19:16:05.782][11825][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:93] StreamAggregatedResources gRPC config stream closed: 13,
What I could make of this error is that the connection to the management server is failing. But In the consul logs i can see the grpc port is up.
Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
Consul's xDS server is not designed to be used outside of Connect, and as such has not been tested or validated to be able to provide CDS/EDS data outside of that context.
You may want to take a look at consul-envoy-xds or envoy-control, both of which may provide the xDS functionality you're looking for without requiring you to deploy a full-blown service mesh.
I had came across this scenario's for all the below use cases.
Except Case 3 which is normal & general and rest of the cases needs to be resolved
These below Services were running without any errors:
Net.Msmq Listener Adapter
Net.Pipe Listener Adapter
Net.Tcp Listener Adapter
Net.Tcp Port Sharing Service
Windows Process Activation Service
also Enabled Protocols: net.tcp,http
From the command line or via application when calling WCF written services:
C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6.1 Tools>svcutil net.tcp://localhost/TNA/TAServices/AuthenticationManager/mex
Case 1:
Could not connect to
net.tcp://localhost/TNA/TAServices/AuthenticationManager/mex. The
connection attempt lasted for a time span of 00:00:04.0935290. TCP
error code 10061: No connection could be made because the target
machine actively refused it 127.0.0.1:808.
No connection could be made because the target machine actively refused it 127.0.0.1:808
Case 2:
There was no endpoint listening at
net.tcp://localhost/TNA/TAServices/AuthenticationManager/mex that
could accept the message. This is often caused by an incorrect address
or SOAP action. See InnerException, if present, for more details.
Case 3:
The socket connection was aborted. This could be caused by an error
processing your message or a receive timeout being exceeded by the
remote host, or an underlying network resource issue. Local socket
timeout was '00:04:59.9843875'.
An existing connection was forcibly closed by the remote host
Event Log for Case 2:
An error occurred in the Activation Service 'NetTcpActivator' of the protocol 'net.tcp' while trying to listen for the site '1', thus the protocol is disabled for the site temporarily. See the exception message for more details.
URL: WeakWildcard:net.tcp://username.domainname.com/
Status: FailedToListen
Exception: System.ServiceModel.AddressAlreadyInUseException: There is already a listener on IP endpoint 0.0.0.0:808. This could happen if there is another application already listening on this endpoint or if you have multiple service endpoints in your service host with the same IP endpoint but with incompatible binding configurations. ---> System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted
at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.Bind(EndPoint localEP)
at System.ServiceModel.Channels.SocketConnectionListener.Listen()
--- End of inner exception stack trace ---
at System.ServiceModel.Channels.SocketConnectionListener.Listen()
at System.ServiceModel.Activation.TransportListener.Go(IConnectionListener connectionListener)
at System.ServiceModel.Activation.TransportListener..ctor(IPEndPoint endPoint)
at System.ServiceModel.Activation.TransportListener.Listen(IPEndPoint endPoint)
at System.ServiceModel.Activation.RoutingTable.TcpStart(MessageQueue messageQueue, BaseUriWithWildcard path)
at System.ServiceModel.Activation.MessageQueue.Register(BaseUriWithWildcard path)
at System.ServiceModel.Activation.ListenerAdapter.RegisterBindings(IActivatedMessageQueue queue, Int32 siteId, String[] bindings, String path)
Process Name: SMSvcHost
Process ID: 4608
How should we need to achieve to go to case 3 by default, whenever, at times such as Booting and rebooting the system?
Case 1 and Case 2 require a Mex service endpoint in the WCF service configuration. Please add a Mex endpoint to exchange the service metadata.
<system.serviceModel>
<services>
<service name="WcfService1.Service1">
<endpoint address="service1" binding="basicHttpBinding" contract="WcfService1.IService1" ></endpoint>
<endpoint address="service2" binding="netTcpBinding" contract="WcfService1.IService1"></endpoint>
<!--for exchanging service metadata.-->
<endpoint address="mex" binding="mexTcpBinding" contract="IMetadataExchange"></endpoint>
</service>
</services>
Feel free to let me know if there is anything I can help with.
I have a Windows Service which recently was set to Automatic(Delayed Start) from Automatic because sometimes it was failing to start and that would have required manual intervention to start it.
But on one of the machines we are testing with the service failed to open a socket on TCP port 5000.
The error is the following:
Initialize(): System.Net.Sockets.SocketException (0x80004005): An attempt was made to access a socket in a way forbidden by its access permissions
at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.Bind(EndPoint localEP)
There is a rule on the firewall to allow the service to listen on TCP
port 5000
Restarting the service results in the same socket error
netstat -o does not show any other process listening on TCP port 5000. Only on System start up I had one instance of the following: TCP [::1]:5000 PrecisionLab01:49844 TIME_WAIT 0
Returning the service to Automatic startup resolves the issue. No other system of our 4 test machines has that problem when the service is set for Automatic (Delayed Start)
Is there a way to diagnose which app is responsible for making the System kernel process to claim TCP port 5000?
Thank you in advance.
I'm trying to configure a vsftp server on one of my ec2 instances but when I try to connect via a ftp client in passive mode I receive the error message "Server sent passive reply with unroutable address. Passive mode failed."
I have added to my vsftpd.conf file the following lines
pasv_enable=YES
pasv_min_port=1024
pasv_max_port=1048
port_enable=YES
pasv_address=xxx.xxx.xxx.xxx
and I have configured ports 20,21 and 1024-1048 in my security group.
What am I missing?
I find the solution to the problem...
If the instance has and ipv6 address associated vsftpd(bug? desired behaviour? who knows...) will always return 0.0.0.0 as remote address.
You can avoid this behaviour setting this parameters:
listen=YES
listen_ipv6=NO
We have two ports are exposed 8081 and 8080 in our application. This creates two contexts in the same applicaiton which causes problems such as when the 8080 port is not ready, 8081 port can respond to requests. I want to know whether is there some smart ways to sync those ports so I can rely that application has successfully started whether 8080 or 8081 port responds? In my some situation I want to respond to ping request OK if my cache is loaded correctly.
So we solved this problem by listening to ApplicationReadyEvent at each of the endpoints we wanted it to not respond with 200, instead we respond with 4xx status code until the event is received.