I am debugging my website. When it has an error, the full text form of the HTTP request that caused the error is logged. I want to be able to replay these HTTP requests to help debugging the error.
For instance, I have this in my log now:
POST /ipn/handler.ashx?inst=272&msgType=result HTTP/1.0
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: mysite.com
Content-Length: 28
User-Agent: AGENT/1.0 (UserAgent)
region=website&compName=ACTL
I want a way to make this exact request again on my local test machine (with changed Host attribute). What is the best way to do this?
You could use telnet to talk to your web server and type the exact requests.
You could also use libcurl (& curl) to make a program which is an HTTP client.
And many scripting languages (Python, Ruby, Perl, Ocaml, ...) also have HTTP client libraries (sometimes above Curl).
Related
I am working on a RESTful web application using Go + gorilla/mux v1.4 framework. Some basic security testing after a release revealed an Open URL Redirection vulnerability in the app that allows user to submit a specially crafted request with an external URL that causes server to response with a 301 redirect.
I tested this using Burp Suite and found that any request that redirects to an external URL in the app seems to be responding with a 301 Moved Permanently. I've been looking at all possible ways to intercept these requests before the 301 is sent but this behavior seems to be baked into the net/http server implementation.
Here is the raw request sent to the server (myapp.mycompany.com:8000):
GET http://evilwebsite.com HTTP/1.1
Accept: */*
Cache-Control: no-cache
Host: myapp.mycompany.com:8000
Content-Length: 0
And the response any time is:
HTTP/1.1 301 Moved Permanently
Location: http://evilwebsite.com/
Date: Fri, 13 Mar 2020 08:55:24 GMT
Content-Length: 0
Despite putting in checks for the request.URL to prevent this type of redirect in the http.handler, I haven't had any luck getting the request to reach the handler. It appears that the base http webserver is performing the redirect without allowing it to reach my custom handler code as defined in the PathPrefix("/").Handler code.
My goal is to ensure the application returns a 404-Not Found or 400-Bad Request for such requests. Has anybody else faced this scenario with gorilla/mux. I tried the same with a Jetty web app and found it returned a perfectly valid 404. I've been at this for a couple of days now and could really use some ideas.
This is not the claimed Open URL redirect security issue. This request is invalid in that the path contains an absolute URL with a different domain than the Host header. No sane client (i.e. browser) can be lured into issuing such an invalid request in the first place and thus there is no actual attack vector.
Sure, a custom client could be created to submit such a request. But a custom client could also be made to interpret the servers response in a non-standard way or visit a malicious URL directly without even contacting your server. This means in this case the client itself would be the problem and not the servers response.
I am trying to write a simple function which would easily extract the contact information from a classified listing.
Background
The URL I'm looking at is
https://www.idealista.pt/imovel/27542922/
Looking through the developer tools in Chrome, I see that it makes a GET request to this URL. https://www.idealista.pt/pt/ajax/listingController/adContactInfoForListing.ajax?adId=27542922
If I make a GET request in Postman or just copy the second URL into Chrome I get a JSON containing various details.
My code
(Ruby)
uri = URI('https://www.idealista.pt/pt/ajax/listingController/adContactInfoForListing.ajax?adId=27542922')
foo = Net::HTTP.get(uri)
JSON.parse(foo)
The problem
The response is a 403 with a body saying that the system has detected that many requests have been made in a short period of time.
I can replicate this in Postman by doing seven or eight consecutive requests, but then if I wait a minute or two before trying again I get back to seeing the JSON.
Through Ruby it happens straight away.
What I've tried
I've tried copying some or all of the temporary headers created by Postman into my request in Ruby but I still get the same error or 404
User-Agent - PostmanRuntime/7.22.0
Accept - */*
Cache-Control - no-cache
Postman-Token - 6c68a9eb-83d5-4724-9f41-3fc51971db9f
Host - www.idealista.pt
Accept-Encoding - gzip, deflate, br
Cookie - userUUID=c017919a-6115-4905-95b3-5d949c6fb447; _pxhd=34ed938caca242bf6050147e1514cda07b704cc7681245a4beec5a64e0a5cf66:d4f21381-522a-11ea-a954-6f59910ff05b; SESSION=887b6dbc-78a4-4abd-9600-7ce401507331; WID=15a353ca7aab3446|XlEN6|XlEN4
Connection - keep-alive
you have to use a proxy, and chanfe the ip
I'm writing a client library for HTTP.
With HttpResponse, how can I get Reason-Phrase value?
I suggest anyone came with the same curiosity to take a look at
https://evertpot.com/http-2-finalized/
HTTP/2 does away with the ‘reason phrase’. In HTTP/1.1 it was possible for servers to submit a human readable reason along with the status code, such as HTTP/1.1 404 Can't find it anywhere!. Or HTTP/1.1 200 I like pebbles, but in HTTP/2 this is removed.
I am using ESP8266-Websocket, aws-sdk-arduino(cleaned branch) and pubsubclient to try to comunicate with aws iot mqtt service using websockets.
My question is about the first connection request. I am using this browser app as reference https://github.com/awslabs/aws-iot-examples and the sign code from aws-sdk-arduino (that works fine calling the aws iot restful api)
My request was this (after connect to the endpoint at 443 port):
GET wss://ENDPOINT.iot.us-west-2.amazonaws.com/mqtt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AWSKEY%2F20160318%2Fus-west-2%2Fiotdevicegateway%2Faws4_request&X-Amz-Date=20160318T183246Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=a1f0d7b58983f9dff7e3bf6cab062db3243ebafc990803a018c6a23433891404 HTTP/1.1
host: ENDPOINT.iot.us-west-2.amazonaws.com
Connection: Upgrade
Upgrade: websocket
Origin: file://
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: D2alJOyUkBlR+8yhv2UBLg==
Sec-WebSocket-Protocol: mqtt
but I keep getting
HTTP/1.1 403 Forbidden
content-type: application/json
content-length: 241
date: Fri, 18 Mar 2016 18:34:57 GMT
x-amzn-RequestId: f2edfe83-1bbc-4481-97e0-39ccfc4d1c2f
connection: Keep-Alive
x-amzn-ErrorType: ForbiddenException:
am i missing some request header parameter? is there anyway to get a better feedback from x-amzn-ErrorType: ForbiddenException? am i messing up in the sign process? (even though it works for rest call)
Yeah, I've finally got response status 101 switching protocols \o/
when you are building the request, it must be:
GET /mqtt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AWSKEY%2F20160318%2Fus-west-2%2Fiotdevicegateway%2Faws4_request&X-Amz-Date=20160318T183246Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=a1f0d7b58983f9dff7e3bf6cab062db3243ebafc990803a018c6a23433891404 HTTP/1.1
instead of
GET wss://ENDPOINT.iot.us-west-2.amazonaws.com/mqtt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AWSKEY%2F20160318%2Fus-west-2%2Fiotdevicegateway%2Faws4_request&X-Amz-Date=20160318T183246Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=a1f0d7b58983f9dff7e3bf6cab062db3243ebafc990803a018c6a23433891404 HTTP/1.1
the js example that I was following was using the full path. When I got the request built by chrome (throught developer tools) the path was full as well. Just after use firebug I got the real request that browser was sending to the server (with short path).
now I can continue to try the solution (mqtt over websockets at esp8266) :-) if it works, I will share the code ;-)
I found the apache xfire has add one head parameter in its post header:
POST /testservice/services/TestService1.1 HTTP/1.1
SOAPAction: "testAPI" Content-Type: text/xml; charset=UTF-8
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; XFire Client +http://xfire.codehaus.org)
Host: 192.168.10.111:9082
Expect: 100-continue
Will this Expect: 100-continue make the roundtrip call between the xfire client and its endpoint server a little bit waste because it will use one more handshake for the origin server to return the "willing to accept request"?
This just my guess.
Vance
I know this is old question but as I was just researching the subject, here is my answer. You don't really need to use "Expect: 100-continue" and it indeed does introduce extra roundtrip. The purpose of this header is to indicate to the server that you want your request to be validated before posting the data. This also means that if it is set, you are committed to waiting (within your own timeout period - not indefinitely!) for server response (either 100 or HTTP failure) before sending your form or data. Although it seems like extra expense, it is meant to improve performance in failure cases, by allowing the server to make you know not to send the data (since the request has failed).
If the header is not set by the client, this means you are not awaiting for 100 code from the server and should send your data in the request body. Here is relevant standard: http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html (jump to section 8.2.3).
Hint for .NET 4 users: this header can be disabled using static property "Expect100Continue"
Hint for libcurl users: there was a bug in old version 7.15 when disabling this header wasn't working; fixed in newer versions (more here: http://curl.haxx.se/mail/lib-2006-08/0061.html)