I am working on a dropbox like system and I am wondering how the client gets notified when the files change on the server side. It is my impression that both dropbox and ubuntu one operate over HTTP ports and work as follows:
1. if files change on client machine, inotify detects it and preforms a push from the client to the server. (I get this part)
2. if files change on the server a simple unsolicited notification (just a message saying "time to sync") is sent from the server to the client. Then the client initiates a sync to the server.
I dont really care which language I do this in. I am just wondering how the client gets contacted. Specifically, what if a client is behind a firewall with its own local IP addresses. How does the server locate it?
Also, what kind of messaging protocols would be used to do something like this? I was planning on doing this over HTTP or SSH, but I have no attachment do that.
I'm not sure what Dropbox is using, but it could be websockets (unlikely, it's a pretty new and not widely deployed thing) or more likely a pending Ajax request from the client to the server -- to which the server only responds when it has new stuff for the client. The latter is the common way to implement (well, OK -- "hack";-) some form of "server push" with HTTP.
It took a little research into networking to see how this would work, but it is far more trivial then I expected. I am now using standard Java sockets for this. Start up the server process which listens for a socket connection. Then start up the client which connects to the server.
Once the connection is made, messages can be sent back and fourth. This works through NAT (network address translation) which is standard method for routing packets on private networks behind a firewall.
Related
I was experimenting hashicorp/yamux over gorilla/websocket, and got stuck.
I started with vanilla WebSocket using the echo example from Gorilla WebSocket project. It was very a straight forward client-server setup. Then image that the server is now behind a firewall, thus the client cannot make a direct connection to it. So I introduced a hub and an agent. The hub is supposed to be publicly visible and connectable for the client. The agent would run alongside the server, who would first make a WebSocket connection to the hub and then multiplex the connection using Yamux so that the hub can then initiate requests to the server. In this way, I effectively "exposed" the server beyond the firewall.
For normal HTTP endpoints, things are good. The client can make requests to the hub, who would proxy these requests to the agent using the WebSocket connection initiated by the hub, and then the hub would further proxy these requests to the server.
However, this trick failed to work with WebSocket endpoints. For the echo example, the client can access the HTML on / through the hub-agent-server chain, but would fail on the /echo path, which is a WebSocket endpoint.
My question is, is this WebSocker over Yamux over WebSocket fundamentally impossible, or do I just need some extra lines to get things work? Here's the code I've been experimenting with. Really appreciate your helps!
I am trying to build a SOCKS solution for forward proxy. I am using dante SOCKS proxy as I have heard that big companies like google uses it as forward proxy solution.
on the SOCKS server, I am allowing based on FQDN's like google.com:443
Now the problem is, when the client constructs the packet, it tries to resolve google.com and gets X.X.X.X and sends connect request to SOCKS server. Now when the server receives the packets, it tries to reconstruct the packet to send out to internet, the server again does DNS resolution and if the server gets response as Y.Y.Y.Y, then it doesn't allow client's request as the destination IP in the client's request is different then the server's resolved IP address.
There was a solution in dante client which tells client to put a dummy destination address 0.0.0.1 and sends request to server and server processes it properly then. However that is creating a problem with internal domains as after using that dns resolution method, every requests goes through dante server :(
Please let me know
If there is any solution through which would help me in maintaining a DNS record expiry DC wide for e.g. google.com resolves to X.X.X.X and I should be able to resolve to this same IP address on 100's of DNS client and in case if the record changes, then it should immediately change/expire on client.
Any other proxy/socks solution which should be transparent to applications for forward proxy
I went ahead with this solution in case anyone is curious to see the solution.
I used PowerDNS Auth Server with Pipe backend. The requests would land to PowerDNS server for resolution, it will pass on all the data to Pipe backend script with ABI, the script analysis the requests, sees if it is present under cached variable/memory map, if it is cache hit, it will respond using cached DNS records else it will use a DNS resolver to resolve that query like a resolver resolves normally.
PowerDNS version lower than 4.1 supports Pipe backend + resolver. This way, the request would first land to pipe backend script, if the script doesn't have any entries cached, it will not respond or will respond blank and then PowerDNS would resolve it with the mentioned resolver server in the configuration. However with version 4.1 and above, the resolver part is removed from PowerDNS Auth server hence you need to handle that behaviour via Pipe backend script.
It depends on your client. Firefox, for example, sends hostname to SOCKS proxy without resolving it. You can confirm that by Wireshark.
PS. assume you are using a SOCKS5/4a proxy. SOCKS4 does not support hostname. Ref: https://en.wikipedia.org/wiki/SOCKS#SOCKS4a
Preamble.
I have some specific application (called LinkBit PacketCraft) for network signaling testing. Scripts of this App have a specific procedure for opening a socket for requests receiving (in my case it is SIP over UDP and HTTP requests) that consist of two blocks: "TCP/IP Control.Open Request" and "TCP/IP Control.Open Confirm" with such parameters as IP(v4/v6), port and protocol (TCP/UDP). I don't know what they exactly do, but as a result after this procedure I can receive requests on specified ip/port.
The Problem.
It worked well until our IT engineers re-installed OS (Windows Server 2008 R2). Don't ask me why, just should be. After re-installation I have one server where it works as well and one server where it doesn't work. My script show me that socket opened successfully, I do see incoming requests (SIP over UDP and HTTP) in the WireShark on this machine, but application doesn't receive them.
I have completely the same script and the same version of application on another server where it works.
Our IT-service can't find any difference between servers configuration, but I don't believe them.
Who may knows which setting or configuration may be responsible for requests delivery to application?
P.S. Just one remark. If I send SIP HTTP request in my script, the application can receive requests and responses to the same socket was used for sending.
The problem was in the Firewall. I received all requests when I disabled it.
I am new to Websockets. While reading about websockets, I am not been able to find answers to some of my doubts. I would like if someone clarifies it.
Does websocket only broadcasts the data to all clients connected instead of sending to a particular client? Whatever example (mainly chat apps) I tried they sends data to all the clients. Is it possible to alter this?
How it works on clients located on NAT (behind router).
Since client server connection will always remain open, how will it affect server performance for large number of connections?
Since I want all my clients to get real time updates, it is required to connect all my clients to server, so how should I handele the client connection limit?
NOTE:- My client is not a Web browser but a desktop application.
No, websocket is not only for broadcasting. You send messages to specific clients, when you broadcast you just send the same message to all connected clients, but you can send different messages to different clients, for example a game session.
The clients connect to the server and initialise the connections, so NAT is not a problem.
It's good to use a scalable server, e.g. an event driven server (e.g. Node.js) that doesn't use a seperate thread for each connection, or an erlang server with lightweight processes (a good choice for a game server).
This should not be a problem if you use a good server OS (e.g. Linux), but may be a limitation if your server uses a desktop version of Windows (e.g. may be limited to 200 connections).
Is it possible to create a website that makes possible this scenario:
User A logs into the website, uploads a file making a direct TCP connection to user B that it is within the same site at same time downloading the file. Without passing the file trough the server.
How to make user B to listen through a browser?
Would this violate "Same origin policy"?
Point is to use browser and no other software like P2P clients.
Is this crazy idea possible?
I doubt webRTC covers exactly what you need.
You have two issues:
B, if running from a web browser, cannot open a port to receive an inbound connection
Even if B were able to do that, you will likely have to face NAT traversal issues.
The solution is to use/implement a relay server:
A opens an outbound HTTP/Websocket connection to the relay server
B opens an outbound HTTP/Websocket connection to the relay server
A sends data to the relay server on the outbound channel (HTTP POST for instance)
B reads data from the relay server on the response (to an HTTP GET for instance)
Easier said than done.... (and yes it is a feature of advanced P2P networks like JXTA, XMPP, Skype..., and yes you unfortunately need an intermediate server)
Check ICE for a specification of 'how to do NAT traversal'
Sure, it's called P2P. You don't even need Ajax.
What I was looking is: WebRTC.
Others have mentioned WebRTC, but here is a live example:
http://sharefest.me