Is there a way to let haproxy or squid to run a (bash)script (or another http request) before performing the proxying of the incoming requests?
I want to host a userX specific http server(and service) at userX.mydomain.com, but these kind of services can be running or not, depending on the load of the machine that hosts them.
So the first time in the day, the userX access to the url userX.mydomain.com, the http server hosting the serviceX has to be started.
I already managed, thanks to haproxy, xinetd, some bash script, and the "HTTP Refresh header directive" to perform a refresh after the http server/service start..
but now I would like to make it even better, to let the "http service starting" to be transparent to the client asking for a GET, a PUT or a POST, and to immediately reply correctly, with the correct service response even at the first http request.
So I will need to start the service and then immediately proxying the request to the service just started.
I already try the "http-check" and "check" options in haproxy but I don't think they can be useful to me, because the healt checks are asynchrnous to the request handling of haproxy. Instead, I will need to perform this script execution for each request and before that haproxy proxies the request..
If squid allows to perform this kind of action, I can even let haproxy to proxy the request to squid, that then, can start the service and proxy the request
Does someone have an idea to achieve it?
Thanks in advance.
This can be done using proxymachine - https://github.com/mojombo/proxymachine
Basically proxymachine can intercept the HTTP request, parse the headers, run arbitrary Ruby code, and then forward the connection.
You would need to terminate the SSL prior to proxymachine getting the connection - e.g. using haproxy (with the new SSL capability).
Related
Is there a way for Squid Proxy to intercept and filter HTTPS payload from request made before it is being sent? Just like the way mitmproxy does.
I found out it's possible to log the payload with ssl-bumping, I had some success also altering response data with sample ecap. Only if I knew some C programming.
Also, iptables has a function to filter payload by string but only with http traffic. I don't know if it's possible to strip the SSL traffic first before feeding it to iptables and send the request outside before sending back (mitm proxy basically).
The only drawback with mitmproxy right now is unable for daemonization (you need to hack it), and I have my doubt using it in a large load.
Any one way or another?
I have a use case where I have to put a middle server or relay or tunnel to do network communication with the following points:
I have a web server running, let say when I hit an API /request hosted my web server, it creates a post request to https://www.google.com and gives me a response through the endpoint.
I want a middle server (proxy etc.) which I will call while creating this post request instead of communicating through my webserver,
the call goes to the middle server and gives me the same response as I was getting directly.
For this, the SQUID proxy worked for me.
I came across NGINX, but we can not use NGINX as a forward proxy, also there are some observations that might be useful with this regard.
SQUID proxy also uses the conf file as similar to NGINX,
HTTPS traffic is encrypted, the proxy server need to do some more work to get something with Https requests,
For intercepting, and creating ACL rules, someone will need to have a dummy certificate to be used by the server to act as the owner of the requested content through the proxy,
a list of rules can be incorporated within SQUID.conf to achieve the filtering.
I hope this could be useful to achieve something like this.
I am trying to build a SOCKS solution for forward proxy. I am using dante SOCKS proxy as I have heard that big companies like google uses it as forward proxy solution.
on the SOCKS server, I am allowing based on FQDN's like google.com:443
Now the problem is, when the client constructs the packet, it tries to resolve google.com and gets X.X.X.X and sends connect request to SOCKS server. Now when the server receives the packets, it tries to reconstruct the packet to send out to internet, the server again does DNS resolution and if the server gets response as Y.Y.Y.Y, then it doesn't allow client's request as the destination IP in the client's request is different then the server's resolved IP address.
There was a solution in dante client which tells client to put a dummy destination address 0.0.0.1 and sends request to server and server processes it properly then. However that is creating a problem with internal domains as after using that dns resolution method, every requests goes through dante server :(
Please let me know
If there is any solution through which would help me in maintaining a DNS record expiry DC wide for e.g. google.com resolves to X.X.X.X and I should be able to resolve to this same IP address on 100's of DNS client and in case if the record changes, then it should immediately change/expire on client.
Any other proxy/socks solution which should be transparent to applications for forward proxy
I went ahead with this solution in case anyone is curious to see the solution.
I used PowerDNS Auth Server with Pipe backend. The requests would land to PowerDNS server for resolution, it will pass on all the data to Pipe backend script with ABI, the script analysis the requests, sees if it is present under cached variable/memory map, if it is cache hit, it will respond using cached DNS records else it will use a DNS resolver to resolve that query like a resolver resolves normally.
PowerDNS version lower than 4.1 supports Pipe backend + resolver. This way, the request would first land to pipe backend script, if the script doesn't have any entries cached, it will not respond or will respond blank and then PowerDNS would resolve it with the mentioned resolver server in the configuration. However with version 4.1 and above, the resolver part is removed from PowerDNS Auth server hence you need to handle that behaviour via Pipe backend script.
It depends on your client. Firefox, for example, sends hostname to SOCKS proxy without resolving it. You can confirm that by Wireshark.
PS. assume you are using a SOCKS5/4a proxy. SOCKS4 does not support hostname. Ref: https://en.wikipedia.org/wiki/SOCKS#SOCKS4a
I have a proxy servlet that is implemented using Jetty's AsyncProxyServlet.Transparent (Jetty 9). Proxied requests are occasionally failing with EarlyEOF exceptions because of the way the remote server sometimes closes connections. In these cases, I would like the proxy to retry the request on behalf of the client instead of returning a 502 status response. What is the correct way to do this?
I assume I need to override AbstractProxyServlet's onProxyResponseFailure method and implement my own error handling, but I'm not sure how to create and send a new proxy request and associate it with the original request from the client.
Proxy retry with AsyncProxyServlet isn't feasible.
The Async nature of both the browser HTTP exchange and the proxied HTTP exchange means they are tied at the hip to each other.
If one fails, both fail, automatically.
Its very difficult to retry, as the browser HTTP exchange is already committed and partially completed as well.
In essence, the browser HTTP exchange would need to be suspended, and then the proxy HTTP exchange would need to be restarted, from scratch, then you'll need to "catch up" the exchange on the proxy side to the point where you are on the browser side. Once you are caught up, you'll have to adapt the proxy response to match the techniques for the browser response (things like known content-length, gzip state, chunking, etc..)
This is further complicated if the proxy response changes between requests, even in minor ways (response headers, sizes, compression, content, etc..)
The only way you can accomplish retry is to NOT use async, but use full caching of the proxy response BEFORE you send the response to the client (but this is actually more difficult to implement than the Async proxy techniques, as you have to deal with complex memory, http caching, and timeout concerns)
Our web application has a button that is supposed to send data to a server on the local network that in turn prints something on a printer.
So far it was easy: The button triggered an AJAX POST request to http://printerserver/print.php with a token, that page connected to the web application to verify the token and get the data to print and then printed.
However, we are now delivering our web application via HTTPs (and I would rather not go back to HTTP for this) and newer versions of Chrome and Firefox don't make the request to the HTTP address anymore, they don't even send the request to check CORS headers.
Now, what is a modern alternative to the cross-protocol XHR? Do Websockets suffer from the same problem? (A Google search did not make clear what is the current state here.) Can I use TCP Sockets already? I would rather not switch to GET requests either, because the action is not idempotent and it might have practical implications with preloading and caching.
I can change the application on the printerserver in any way (so I could replace it with NodeJS or something) but I cannot change the users' browsers (to trust a self-signed certificate for printerserver for example).
You could store the print requests on the webserver in a queue and make the printserver periodically poll for requests to print.
If that isn't possible I would setup a tunnel or VPN between the webserver and printserver networks. That way you can make the print request from the webserver on the server-side instead of the client. If you use curl, there are flags to ignore invalid SSL certificates etc. (I still suspect it's nicer to introduce a queue anyway, so the print requests aren't blocking).
If the webserver can make an ssh connection to something on the network where the printserver is on, you could do something like: ssh params user#host some curl command here.
Third option I can think of, if printserver can bind to for example a subdomain of the webserver domain, like: print.somedomain.com, you may be able to make it trusted by the somedomain.com certificate, IIRC you have to create a CSR (Certificate Signing Request) from the printserver certificate, and sign it with the somedomain.com certificate. Perhaps it doesn't even need to be a subdomain for this per se, but maybe that's a requirement for the browser to do it client-side.
The easiest way is to add a route to the webapp that does nothing more than relay the request to the print server. So make your AJAX POST request to https://myapp.com/print, and the server-side code powering that makes a request to http://printerserver/print.php, with the exact same POST content it received itself. As #dnozay said, this is commonly called a reverse proxy. Yes, to do that you'll have to reconfigure your printserver to accept (authenticated) requests from the webserver.
Alternatively, you could switch the printserver to https and directly call it from the client.
Note that an insecure (http) web-socket connection on a secure (https) page probably won't work either. And for good reason: generally it's a bad idea to mislead people by making insecure connections from what appears to them to be a secure page.
The server hosting the https webapp can reverse proxy the print server,
but since the printer is local to the user, this may not work.
The print server should have the correct CORS headers
Access-Control-Allow-Origin: *
or:
Access-Control-Allow-Origin: https://www.example.com
However there are pitfalls with using the wildcard.
From what I understand from the question, printserver is not accessible from the web application so the reverse proxy solution won't work here.
You are restricted from making requests from the browser to the printserver by cross-origin-policy.
If wish to communicate with the printserver from an HTTPS page you will need the printserver to expose print.php as HTTPS too.
You could create a DNS A record as a subdomain of your web application that resolves to the internal address of your printserver.
With those steps in place you should be able to update your printserver page to respond with permissive CORS headers which the browser should then respect. I don't think the browser will even issue CORS requests across different protocol schemes (HTTPS vs HTTP) or to internal domains, without a TLD.