I'm going nuts, I have a very simple WEBrickProxy (WEBrick 1.3.1, with Ruby 1.9.3), and I want to try it out with curl. Here is the proxy:
require 'webrick'
require 'webrick/httpproxy'
server = WEBrick::HTTPProxyServer.new(:BindAddress => "localhost", :Port => 8888)
trap('INT') { server.shutdown }
server.start
And here is the cURL command
curl --proxy localhost:8888 http://www.google.de -v
But the curl command always returns a bad request:
* About to connect() to proxy localhost port 8888 (#0)
* Trying ::1...
* connected
* Connected to localhost (::1) port 8888 (#0)
> GET http://www.google.de HTTP/1.1
> User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
> Host: www.google.de
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 400 Bad Request
< Content-Type: text/html; charset=ISO-8859-1
< Server: WEBrick/1.3.1 (Ruby/1.9.3/2012-10-12)
< Date: Mon, 18 Mar 2013 13:44:27 GMT
< Content-Length: 295
< Connection: close
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN">
<HTML>
<HEAD><TITLE>Bad Request</TITLE></HEAD>
<BODY>
<H1>Bad Request</H1>
bad URI `http://www.google.de'.
<HR>
<ADDRESS>
WEBrick/1.3.1 (Ruby/1.9.3/2012-10-12) at
23tuxmb.local:8888
</ADDRESS>
</BODY>
</HTML>
* Closing connection #0
curl --version returns on my Mac OS X 10.8
curl 7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IPv6 Largefile NTLM NTLM_WB SSL libz
I can't find out where the error is, and I tried that piece of code a couple of weeks ago, and I can remember that it worked.
The weird thing is, that when I configure my Mac to globally use that proxy (inside the System Settings -> Network -> Advanced -> Proxies -> Web Proxy), everything works. A request with Chrome or any other application is routed through the proxy (and I see this inside the Terminal log of the proxy).
So, can anyone reproduce this issue? Is it a curl bug? Or a Webrick related issue?
EDIT
More information: The output of the ruby script itself when curl tries to connect is
[2013-03-18 17:16:32] ERROR bad URI `http://www.amazon.de'.
localhost - - [18/Mar/2013:17:16:32 CET] "GET http://www.amazon.de HTTP/1.1" 400 286
- -> http://www.amazon.de
thx!
If you add a trailing slash to the URL you request, does it work then? Ie a command line like this:
curl --proxy localhost:8888 http://www.google.de/ -v
(this is a curl bug reported in bug #1206, and fixed in git and next release...)
Related
I am running a Linux (Debian based) docker container on a Windows host.
I have a device with an HTTP based API that I want to interface with from the container.
When I run the curl command to the devices API under WSL from my host machine, it responds correctly.
<username>#<hostname>:/mnt/c/Users/$ curl -H "Authorization: Basic YWRtaW46aGVybW9zYTA=" http://172.16.255.33/Set.cmd?CMD=GetPower -v
* Trying 172.16.255.33:80...
* TCP_NODELAY set
* Connected to 172.16.255.33 (172.16.255.33) port 80 (#0)
> GET /Set.cmd?CMD=GetPower HTTP/1.1
> Host: 172.16.255.33
> User-Agent: curl/7.68.0
> Accept: */*
> Authorization: Basic YWRtaW46aGVybW9zYTA=
>
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Connection: close
<
<html>p61=1,p62=1,p63=0,p64=0</html>
* Closing connection 0
However, when I do the same curl command from my container, I get the error (52) Empty reply from server
root#9af1adf97ba7:/# curl -H "Authorization: Basic YWRtaW46aGVybW9zYTA=" http://172.16.255.33/Set.cmd?CMD=GetPower -v
* Trying 172.16.255.33:80...
* Connected to 172.16.255.33 (172.16.255.33) port 80 (#0)
> GET /Set.cmd?CMD=GetPower HTTP/1.1
> Host: 172.16.255.33
> User-Agent: curl/7.74.0
> Accept: */*
> Authorization: Basic YWRtaW46aGVybW9zYTA=
>
* Empty reply from server
* Connection #0 to host 172.16.255.33 left intact
curl: (52) Empty reply from server
In Wireshark, I see the device is setting the RST flag in its response when queried from the container. No such reset in the response when queried from my host machine.
I am starting my container using these arguments:
docker run -it -p 80:80 <image-name> bash
When I start to run the container on a Linux host using the same args, the curl request succeeds (as a test, I have to run this app on a Windows host).
I have tried both WSL2 based and Hyper-V based versions of Docker on Windows.
Under WSL2 container, I get no response from the device.
Under Hyper-V based container, I get the issue described above.
I've spent about a day Googling and trying various things, and I am out of ideas at this point.
Thanks.
I'm trying to connect to internet services (specifically Facebook API) from an internal application, and for that I have to go through one of two proxies provided by the security team. If I use one of these proxies with CURL then it works OK:
export http_proxy=http://<user>:<pass>#<proxy_ip>:<port>/
export https_proxy=http://<user>:<pass>#<proxy_ip>:<port>/
curl -v -I https://graph.facebook.com/v7.0/me/messages
* About to connect() to proxy <proxy_ip> port <port> (#0)
* Trying <proxy_ip>...
* Connected to <proxy_ip> (<proxy_ip>) port <port> (#0)
* Establish HTTP proxy tunnel to graph.facebook.com:443
* Proxy auth using Basic with user '<user>'
> CONNECT graph.facebook.com:443 HTTP/1.1
> Host: graph.facebook.com:443
> Proxy-Authorization: Basic <base64>
> User-Agent: curl/7.29.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=*.facebook.com,O="Facebook, Inc.",L=Menlo Park,ST=California,C=US
* start date: May 14 00:00:00 2020 GMT
* expire date: Aug 05 12:00:00 2020 GMT
* common name: *.facebook.com
* issuer: CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US
> HEAD /v7.0/me/messages HTTP/1.1
> User-Agent: curl/7.29.0
> Host: graph.facebook.com
> Accept: */*
>
< HTTP/1.1 400 Bad Request
HTTP/1.1 400 Bad Request
< WWW-Authenticate: OAuth "Facebook Platform" "invalid_request" "An active access token must be used to query information about the current user."
Facebook replied with HTTP 400 Bad Request but it doesn't matter now, I'm just trying to see that it reaches Facebook correctly, which it does.
Now, thing is, security team provided two proxies, and I should use them in failover mode, meaning after a (preferably configurable) number of timeouts from proxy 1, I should switch to proxy 2. Before I implement this feature in my application, I wanted to see if I can configure HAProxy to handle all this for me (as I'm already using HAProxy on this same server for inbound direction from Facebook, through another proxy).
So, this is my HAProxy configuration:
global
log 127.0.0.1 local2 debug
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
stats socket /etc/haproxy/haproxysock level admin
defaults
log global
option httplog
mode http
timeout connect 5000
timeout client 50000
timeout server 50000
# Incoming messages from Facebook
frontend chatbot_front
bind *:8773 ssl crt /etc/haproxy/haproxy.cer
use_backend chatbot_back
backend chatbot_back
balance roundrobin
server chatbot1 <chatbot_ip1>:<chatbot_port1> check ssl verify none
server chatbot2 <chatbot_ip2>:<chatbot_port2> check ssl verify none
# Outbound messages to Facebook through another proxy
frontend serverproxy_front
bind *:8775 ssl crt /etc/haproxy/haproxy.cer
use_backend serverproxy_back
backend serverproxy_back
balance roundrobin
option httpclose
option forwardfor header X-Client
cookie SERVERID insert indirect nocache
http-request set-header Proxy-Authorization "Basic <base64>"
server serverproxy1 <proxy_ip1>:<port1> check
server serverproxy2 <proxy_ip2>:<port2> check
listen stats
bind *:8774 ssl crt /etc/haproxy/haproxy.cer
mode http
maxconn 5
stats enable
stats refresh 10s
stats realm Haproxy\ Statistics
stats uri /stats
stats auth <user>:<password>
The relevant config is the frontend serverproxy_front and the backend serverproxy_back settings above. I tried to look into tutorials on how to set up HAProxy as forward proxy, and this is what I got. But when I try to use it, it doesn't work:
export http_proxy=http://127.0.0.1:8775
export https_proxy=http://127.0.0.1:8775
curl -v -I https://graph.facebook.com/v7.0/me/messages
* About to connect() to proxy <local_ip> port 8775 (#0)
* Trying <local_ip>...
* Connected to <local_ip> (<local_ip>) port 8775 (#0)
* Establish HTTP proxy tunnel to graph.facebook.com:443
> CONNECT graph.facebook.com:443 HTTP/1.1
> Host: graph.facebook.com:443
> User-Agent: curl/7.29.0
> Proxy-Connection: Keep-Alive
>
* Proxy CONNECT aborted
* Connection #0 to host <local_ip> left intact
curl: (56) Proxy CONNECT aborted
What am I missing here?
EDIT: fixed typo
I found it. It is an embarrassingly basic issue: the 8775 port on the haproxy was configured as "https", but I was using it as simple "http" for proxy URL...
Changing config to:
frontend serverproxy_front
# bind *:8775 ssl crt /etc/haproxy/haproxy.cer
bind *:8775
And restarting haproxy, it works successfully.
Why does Tomcat version 7.0.88 gives "400 Bad Request" error code if the hostname ends with xyx.com-abc.
For testing purpose let's assume we have the following entry in the hosts file
127.0.0.1 hello.hello.hello-erq
And we try to access this url from curl
curl -v hello.hello.hello-er:8080
We get the following output
* Rebuilt URL to: hello.hello.hello-er:8080/
* Trying 127.0.0.1...
* Connected to hello.hello.hello-er (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> Host: hello.hello.hello-er:8080
> User-Agent: curl/7.49.0
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< Server: Apache-Coyote/1.1
< Transfer-Encoding: chunked
< Date: Thu, 20 Dec 2018 19:53:09 GMT
< Connection: close
<
* Closing connection 0
While using the localhost in the url we get
C:\playground\apache-tomcat-7.0.88-windows-x64\apache-tomcat-7.0.88\bin>curl -v localhost:8080
* Rebuilt URL to: localhost:8080/
* Trying ::1...
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.49.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: Apache-Coyote/1.1
< Content-Type: text/html;charset=ISO-8859-1
< Transfer-Encoding: chunked
< Date: Thu, 20 Dec 2018 20:00:07 GMT
<
<!DOCTYPE html>
All the tomcat configurations are the same for both the tests and the same issue is replicated on a vanilla out of the box tomcat server too.
I tried to replicate the same issue on tomcat-8 but both the url's worked fine there. How can i dig deeper and find out the root cause of this issue in 7.0.88 ?
Is there some additional logging which i can enable to get more on this issue ?
Or the only thing i have left is to pull my hairs and upgrade ?
Tomcat was attempting to enforce the domain name specification by refusing your hostname with a hyphen in the TLD. This was deemed a bug in Tomcat and fixed in 7.0.89 (and versions of Tomcat 8.0.x, 8.5.x and 9.0.x released around the same time).
So it seems that all you need is a small version bump.
This image is ProxyDroid application and I saw some proxy soft wares like these one.
I find some free servers for http method (http proxy the famous one) and find servers for socks 4 and 5 but I cant find any server that support https and http tunnel and in other word I cant understand what are exactly these protocols.
Proxying HTTPS is done with a HTTP proxy by using the CONNECT request. Using this request the HTTP proxy is instructed to create a tunnel to the target server. Inside this tunnel the client can the do the TLS handshake needed for HTTPS:
> CONNECT example.org:443 HTTP/1.0
>
... proxy established TCP connection to example.org:443 ...
< HTTP/1.0 200 Connection established
<
... tunnel to target established
... proxy forwards data between client and target unchanged
<-> TLS handshake
<-> application data protected by TLS
HTTP tunnel is similar. Normally a HTTP request gets proxied by sending a HTTP proxy request:
> GET http://example.org/index.html HTTP/1.0
> Host: example.org
>
... proxy connects to target example.org and forwards request
... then sends response from target server back
< HTTP/1.0 200 ok
< Content-length: ...
< ...
With HTTP tunnel the client instead uses the CONNECT method described above to create a tunnel to the target server and send the request:
> CONNECT example.org:80 HTTP/1.0
>
< HTTP/1.0 200 Connection established
<
... tunnel established, send HTTP request over tunnel and get reply back
> GET /index.html HTTP/1.0
> Host: example.org
> ...
< HTTP/1.0 200 ok
< ...
I am testing an API I have made using Springboot from my laptop (192.168.1.217:8080) and I am trying to get a cURL request via SSH from my Raspberry Pi.
Here is the error I am receiving when I try to send the request via port 8080 which it seems to not like:
pi#raspberrypi:~ $ curl -v 192.168.1.217:8080/api
* Hostname was NOT found in DNS cache
* Trying 192.168.1.217...
However cURL does work for the same IP but with port 80:
pi#raspberrypi:~ $ curl -v 192.168.1.217
* Rebuilt URL to: 192.168.1.217/
* Hostname was NOT found in DNS cache
* Trying 192.168.1.217...
* Connected to 192.168.1.217 (192.168.1.217) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.38.0
> Host: 192.168.1.217
> Accept: */*
>
< HTTP/1.1 302 Found
< Date: Thu, 30 Mar 2017 17:20:43 GMT
* Server Apache/2.4.23 (Win32) OpenSSL/1.0.2h PHP/5.5.38 is not blacklisted
< Server: Apache/2.4.23 (Win32) OpenSSL/1.0.2h PHP/5.5.38
< X-Powered-By: PHP/5.5.38
< Location: http://192.168.1.217/dashboard/
< Content-Length: 0
< Content-Type: text/html
<
* Connection #0 to host 192.168.1.217 left intact
pi#raspberrypi:~ $
I've tried looking around but to no avail... anybody have any suggestions as to why I cannot find my own hostname in the DNS cache?
Cheers
No, Hostname was NOT found in DNS cache is not the problem. You can clearly see on the next line, in both examples, that after saying that curl is trying to connect to 192.168.1.217. Your problem is that nothing is answering on port 8080 on that IP address (while an Apache server is answering on port 80 there).
If you're getting a long pause and then a timeout rather than a quick "Connection refused", you almost certainly need to open port 8080 in the local firewall on your server machine.
curl without a protocol prefix presumes HTTP port 80.
To use another port, where it does not make that assumption, all you need to do is change your command's URL to be like this:
curl -v http://192.168.1.217:8080/api
Here is a decent article on the subject: Using CURL For Testing Web Applications