I have been struggling with what I believe to be a networking issue and haven't made much progress. I wanted to see if someone had some ideas to a solution or possible steps that could provide me more information to arrive at a solution.
Environment
I have a mobile web application (angular.js) that is running on a html 5 browser (proprietary browser) on a barcode scanning gun (intermec ck71) connecting to a spring boot backend (non clustered single instance dev box). I wrote the mobile web app, but have assumed ownership of the spring boot back end (first time working with springboot). I am in a corporate office where I do not own the networks so I don't have a complete understanding of how things are laid out.
Problem
I cannot get a mobile web app to connect to our dev box on the office wireless with https
Steps already taken
We recently switched our team dev box to https now that development is finished. Before https the mobile application loaded properly but now just shows a spinning wheel on the mobile browser and then times out after about a minute.
The first step I did was to load the server running https locally and checked to see if I could get the scan gun to connect (which it did).
I then tried my desktop computer loading the mobile web app from the dev box (which also worked).
So it seems like the problem is isolated to the mobile gun on the wireless network at the office with https. I had the networking team make sure port 443 and port 8443 were open (which they saw were blocked and opened for me).
I retried connecting and still no luck. I ran a tcpdump on the server and see the following logs on the calls that fail
From this log it looks like the server (172.18.222.121) is closing the connection by sending packet 12 (FIN, ACK)
I added some logging to the tomcat instance of the server and see the following
[pid 22505] accept(41, {sa_family=AF_INET, sin_port=htons(1037), sin_addr=inet_addr("172.31.12.157")}, [16]) = 46
[pid 22505] setsockopt(46, SOL_SOCKET, SO_LINGER, {onoff=0, linger=0}, 8) = 0
[pid 22505] setsockopt(46, SOL_TCP, TCP_NODELAY, [1], 4) = 0
but I still dont see anything about actively telling the conncetion to close. My next thought was that the box might have ip ranges whitelisted/blacklisted but according the the team that maintains the server there are no ip rules, firewalls, or anything that would interfere with network traffic on the box.
So I took this to the network team to show that traffic and they informed me that it is not a networking problem because the initial traffic is getting through and that the real problem is the FIN packet. I took that information to the server team who informed me that they had nothing on the box that would cause a FIN packet to be sent and that it was most probably a networking issue. I decided that it very well could be a server issue since that is what I have access to and everyone else believes there portion is functioning properly. I decided to create a new SSL cert as I am getting the untrusted message on the desktop browser when I connect. I created this new cert and then added it to the trusted root on the mobile scanner gun. I tried to connect again on the gun to no effect.
Next I reached out to the mobile gun vendor to see if there was any information available to me on the gun to troubleshoot it. After some troubleshooting with them they said they cannot reproduce on their side, and that because it works on my side except for connecting on that one network that it is probably a networking issue.
I went back to the server and added logging on the tomcat valve to see if I could see if I could get any additional information. When I look at the logs that the valve produces I get entries for any successful connection, but when I am unable to connect nothing shows up. My connector looks like this just for completeness.
Connector code:
<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
maxThreads="150" scheme="https" secure="true"
clientAuth="false" sslProtocol="TLS"
keystoreType="PKCS12"
keystoreFile="/home/devadmin/.ssh/keystore.p12"
keystorePass="password" />
And the valve:
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="localhost_access_log." suffix=".txt"
pattern="%a %H %p %U %h %l %u %t "%r" %s %b %{xxx}i" />
Conclusion
So basically I can get the application to load everywhere except this one network which leads me to believe that it is networking related. Yet the network team says that it can't be networking related because the initial packets are being transmitted and that the problem is with the FIN packet being sent by the server. I have exhausted what I know about the environments... I am a mobile developer after all:) I still feel it is networking related, but don't have anything to show why the FIN packet is being sent. Any ideas on things to check or what could be causing the problem?
If you can connect with your browser or with curl (for instance) then I suspect the issue is in the client. You said that you had to ignore the certificate violations in the browser so that's a smell, and then you said you "create a new SSL cert". Since it isn't really possible for mortals to create a trusted certificate, that means you installed a self signed or otherwise incomplete certificate, and I would hazard a guess that the client will not allow access over a connection with an untrusted cert (for good reasons)?
This turned out to be related to a redhat/java bug. When the scanner gun reached out to this instance of the server which had the right combinations of versions to cause a bug the server was throwing a SSL exception caused by the library that had the problem. I didn't see these exceptions because of the way the logging was configured.
Here is a link to the bug for people who might be reading this after me.
https://bugzilla.redhat.com/show_bug.cgi?id=1022017
Related
I have a Windows application (APP) and Audio Processing Object (APO) loaded by AudioDG.exe that communicate via gRPC:
APP part that is written in C# creates server via Grpc.Core.
APO part creates client via grpc++.
Server is on 127.0.0.1:20000 (I can see it's up and listening with netstat -ano).
I can confirm that APO is loaded into audio device graph by inspecting it with process explorer.
Everything worked like a charm on Windows 8 and 10, but on 11 it cannot communicate at all - I get either Error Code 14, Unavailable, failed to connect to all addresses or 4, Deadline Exceeded.
After enabling debug traces, I now see "socket is null" description for "connect failed" error:
I0207 16:20:59.916447 0 ..\..\..\src\core\ext\filters\client_channel\subchannel.cc:950: subchannel 000001D8B9B01E20 {address=ipv4:127.0.0.1:10000, args=grpc.client_channel_factory=0x1d8bb660460, grpc.default_authority=127.0.0.1:10000, grpc.internal.subchannel_pool=0x1d8b8c291b0, grpc.primary_user_agent=grpc-csharp/2.43.0 (.NET Framework 4.8.4470.0; CLR 4.0.30319.42000; net45; x64), grpc.resource_quota=0x1d8b8c28d90, grpc.server_uri=dns:///127.0.0.1:10000}: connect failed: {"created":"#1644240059.916000000","description":"socket is null","file":"..\..\..\src\core\lib\iomgr\tcp_client_windows.cc","file_line":112}
What I've tried so far:
Updating both parts to the latest grpc versions.
Using "no proxy", "Http2UnencryptedSupport" and other env variables.
Using "localhost" or "0.0.0.0" instead of "127.0.0.1".
Updating connection to use self signed SSL certificates (root CA, server cert + key, client cert + key).
Adding inbound / outbound rules for my port, and then disabling firewall completely.
Creating server on APO side and trying to connect with the client in APP.
Everything works (both insecure and SSL creds) if I create both client and server in C# part, but as soon as it's APP-APO communication it feels blocked or sandboxed.
What has been changed in Windows 11 that can "block" gRPC?
Thanks in advance!
In your input you write:
Server is at 127.0.0.1:20000
Further looking at the logs, you can see that:
The server is located at
grpc.server_uri=dns:///127.0.0.1:10000
Based on the question posed and the amount of data provided, I would check which port the server is really using and which port the client is looking for a connection on.
The easiest way to do this is to use the built-in Resource Monitor application. On the Network tab, in the TCP Connections list, you can find the application and the port it uses.
You can also use the PowerShell command
Test-NetConnection -Port 10000 -InformationLevel "Detailed"
Test-NetConnection -Port 20000 -InformationLevel "Detailed"
At least this is the first thing I would check based on what you described.
Regarding your question about the changes in Windows 11, I do not think that this is something that's causing problems for you. However, Windows 11 has additional security features compared to Windows 10, try disabling the security features completely as a test. Perhaps this will help solve the problem.
As for ASP.NET Core 6.0 itself (if I understood the version correctly), then there is a possibility that the server part, working not in the sandbox of the programming environment, still does not accept the client certificate. At the program level, you can try to fix this by adding the following exception to the code:
// This switch must be set before creating the GrpcChannel/HttpClient.
AppContext.SetSwitch(
"System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true);
// The port number(5000) must match the port of the gRPC server.
var channel = GrpcChannel.ForAddress("http://localhost:5000");
var client = new Greet.GreeterClient(channel);
More troubleshooting issues with ASP.NET Core 6.0 Microsoft described in detail here.
https://learn.microsoft.com/en-us/aspnet/core/grpc/troubleshoot?view=aspnetcore-6.0
I hope it was useful and at least one of the solutions I suggested will help solve your problem. In any case, if I had more information, I think I could help you more accurately.
Hosting a rails web app on heroku; no recent code changes but about 2 days ago the application started reporting constant H31 misdirect errors. After looking into the documentation, since the application was still accessible I moved on.
The next day users from some of the subdomains could not access the application while others could. The application uses a wildcard certificate and we are behind a WAF. The first thing I tried was checking firewall logs and nothing was showing as blocked. I turned off the firewall and connected the domain directly to the application and still received the H31 stream, after 15 minutes of no change i re-enabled the firewall.
Reached out to the firewall and they said that their logs only showed upstream prematurely closed connection while reading response header
At this point they suggested the issue was most likely on the Heroku DNS servers, so I'm waiting for a response at this point.
Is there anything I can do in the heroku environment that would change this or am I stuck waiting for a response?
I have a web page to control a thermostat on a raspberry pi, and I'm running into difficulties when trying to get websockets to work from a remote client. It seems to work fine when on LAN however. I'm obviously missing something (and likely something basic), but I can't seem to figure out what it is.
The pi's local ip is 192.168.1.134. The web page (served from apache server) has the URL http://192.168.1.134:8010/thermostat.html. The page starts up some javascript, which then tries to connect to the pi's main program using websockets via ws://192.168.1.134:9000. (the server on the pi is running libwebsockets). The websocket comes up, and it seems to work fine. I then tried to connect via a remote client (a cell phone, where wifi was turned off) from http:\\23.239.99.99:8010\thermostat.html. The html/js files load fine, but the web socket attempts to connect to uri ws:\\23.239.99.99:9000, and this fials.
As far as I can tell, the NAT seems to be configured properly:
name ext ext protocol int int ip addr interface
port port port port
start end start end
Thermostat3 8010 8010 TCP 8010 8010 192.168.1.134 eth3.1
Thermostat5 8000 8000 TCP/UDP 80 80 192.168.1.134 eth3.1
Thermostat_ws 9000 9000 TCP/UDP 9000 9000 192.168.1.134 eth3.1
I checked, and the router does not have any firewalls set up, neither does my modem. I didn't install a firewall on the pi (I checked, and there's no odd iptables rule). Does anyone know what I'm missing?
--- EDIT ---
I'm still stuck on this. I called my ISP and they assure me there are no firewalls on their servers. Is there any way to tell if port 9000 is being blocked, and by who?
Bind your apache server to 0.0.0.0 address to make it accessible from remote machines
Try this tool to determine if the port is inaccessible (use the custom port): http://www.whatsmyip.org/port-scanner/
Everything else looks fine. As a sanity check I would try putting the ws port to 8010 to see if that works. I would also recommend using a tool like Advanced Web Client to isolate networking issues.
This is interesting. I once had a similar problem. I set up a WebSocket (I was using a nodejs ws) and once I tried to access it from remote client I was not able to reach it with ws://yourip:port but instead I had to use http://yourip:port. I don't know if you have the same problem, mine was due to a proxy I was using.
I still have an advice for you how you might be able to solve your problem. I don't know how concerned you are about security but as far as I understood your idea you basically connect to your raspberry pi through a WebSocket and tell it to change the temperature.
Back when did a similar project I found it rather hard to secure my WebSocket connection. I was basically sending a password plus command through the WebSocket to my server which then checks wether the password is correct. Otherwise everyone on the internet could heat your house. Not cool...
But therefore, I had to tunnel the connection through https to prevent a middleware attack.
I quickly threw the towel and decided to go with a completely different solution. Basically I set up a nodejs express server (can easily be configured with a self signed certificate to use https or used behind a nginx/apache https server) and authenticated with username and password. When someone made a POST request to /api/thermostats?id=0 with a temperature request, the server checks if the user is authenticated and then executes a terminal command from within node.
Maybe this idea also fits your demands.
I'm working to create a video calling web app using WebRTC.
The communication is working fine on same network. But when communicating in different network I'm getting ICE failed error.
Error: ICE failed, see about:webrtc for more details
In about:webrtc I' able to get local and remote SDP's, but ICE State is failed. http://imgur.com/a/nPPDr
Here is the code of my main.js file
Here is the my log file from about:webrtc
P.S: Before posting the question I've checked several posts in SO and in other sites but no one did the trick.
Looking at the log file you provided it looks like you provided a TURN server, but the communication with that TURN server simply times out. So either something like a local firewall is blocking the communication with your TURN server or your TURN server is not working.
In case your local firewall blocks UDP traffic it might help to configure and use TURN TCP additionally to get through the firewall.
your about:webrtc does not show any relay candidates gathered from a TURN server. At the risk of sounding like a broken record: you need a TURN server for the majority of connections between different networks.
stupid problem. I get those from a client connecting to a server. Sadly, the setup is complicated making debugging complex - and we run out of options.
The environment:
*Client/Server system, both running on the same machine. The client is actually a service doing some database manipulation at specific times.
* The cnonection comes from C# going through OleDb to an EasySoft JDBC driver to a custom written JDBC server that then hosts logic in C++. Yeah, compelx - but the third party supplier decided to expose the extension mechanisms for their server through a JDBC interface. Not a lot can be done here ;)
The Symptom:
At (ir)regular intervals we get a "Address already in use: connect" told from the JDBC driver. They seem to come from one particular service we run.
Now, I did read all the stuff about port exhaustion. This is why we have a little tool running now that counts ports and their states every minute. Last time this happened, we had an astonishing 370 ports in use, with the count rising to about 900 AFTER the error. We aleady patched the registry (it is a windows machine) to allow more than the 5000 client ports standard, but even then, we are far far from that limit to start with.
Which is why I am asking here. Ayneone an ide what ELSE could cause this?
It is a Windows 2003 Server machine, 64 bit. The only other thing I can see that may cause it (but this functionality is supposedly disabled) is Symantec Endpoint Protection that is installed on the server - and being capable of actinc as a firewall, it could possibly intercept network traffic. I dont want to open a can of worms by pointing to Symantec prematurely (if pointing to Symantec can ever be seen as such). So, anyone an idea what else may be the cause?
Thanks
"Address already in use", aka WSAEADDRINUSE (10048), means that when the client socket prepared to connect to the server socket, it first tried to bind itself to a specific local IP/Port pair that was already in use by another socket, either an active one or one that has been closed but is still in the FD_WAIT state. This has nothing to do with the number of ports that are available.
I'm having the same issue on a Windows 2000 Server with a .Net application connecting to a SQL Server 7.0. There's like 10 servers with the same configuration and only one is showing this error several times a day. With a small test program I'm able to reproduce the error by just establishing a TCP connection on the SQL Server listening port. Running CurrPorts (http://www.nirsoft.net/utils/cports.html) shows there's still plenty of available ports in range 1024-5000.
I'm out of ideas and would like to know if you've found a solution since you've posted your question.
Edit : I finally found the solution : a worm was present on the server (WORM_DOWNAD.A) and exhausted local ports without being noticed.