I am trying to write a simple function which would easily extract the contact information from a classified listing.
Background
The URL I'm looking at is
https://www.idealista.pt/imovel/27542922/
Looking through the developer tools in Chrome, I see that it makes a GET request to this URL. https://www.idealista.pt/pt/ajax/listingController/adContactInfoForListing.ajax?adId=27542922
If I make a GET request in Postman or just copy the second URL into Chrome I get a JSON containing various details.
My code
(Ruby)
uri = URI('https://www.idealista.pt/pt/ajax/listingController/adContactInfoForListing.ajax?adId=27542922')
foo = Net::HTTP.get(uri)
JSON.parse(foo)
The problem
The response is a 403 with a body saying that the system has detected that many requests have been made in a short period of time.
I can replicate this in Postman by doing seven or eight consecutive requests, but then if I wait a minute or two before trying again I get back to seeing the JSON.
Through Ruby it happens straight away.
What I've tried
I've tried copying some or all of the temporary headers created by Postman into my request in Ruby but I still get the same error or 404
User-Agent - PostmanRuntime/7.22.0
Accept - */*
Cache-Control - no-cache
Postman-Token - 6c68a9eb-83d5-4724-9f41-3fc51971db9f
Host - www.idealista.pt
Accept-Encoding - gzip, deflate, br
Cookie - userUUID=c017919a-6115-4905-95b3-5d949c6fb447; _pxhd=34ed938caca242bf6050147e1514cda07b704cc7681245a4beec5a64e0a5cf66:d4f21381-522a-11ea-a954-6f59910ff05b; SESSION=887b6dbc-78a4-4abd-9600-7ce401507331; WID=15a353ca7aab3446|XlEN6|XlEN4
Connection - keep-alive
you have to use a proxy, and chanfe the ip
Related
I am working on a RESTful web application using Go + gorilla/mux v1.4 framework. Some basic security testing after a release revealed an Open URL Redirection vulnerability in the app that allows user to submit a specially crafted request with an external URL that causes server to response with a 301 redirect.
I tested this using Burp Suite and found that any request that redirects to an external URL in the app seems to be responding with a 301 Moved Permanently. I've been looking at all possible ways to intercept these requests before the 301 is sent but this behavior seems to be baked into the net/http server implementation.
Here is the raw request sent to the server (myapp.mycompany.com:8000):
GET http://evilwebsite.com HTTP/1.1
Accept: */*
Cache-Control: no-cache
Host: myapp.mycompany.com:8000
Content-Length: 0
And the response any time is:
HTTP/1.1 301 Moved Permanently
Location: http://evilwebsite.com/
Date: Fri, 13 Mar 2020 08:55:24 GMT
Content-Length: 0
Despite putting in checks for the request.URL to prevent this type of redirect in the http.handler, I haven't had any luck getting the request to reach the handler. It appears that the base http webserver is performing the redirect without allowing it to reach my custom handler code as defined in the PathPrefix("/").Handler code.
My goal is to ensure the application returns a 404-Not Found or 400-Bad Request for such requests. Has anybody else faced this scenario with gorilla/mux. I tried the same with a Jetty web app and found it returned a perfectly valid 404. I've been at this for a couple of days now and could really use some ideas.
This is not the claimed Open URL redirect security issue. This request is invalid in that the path contains an absolute URL with a different domain than the Host header. No sane client (i.e. browser) can be lured into issuing such an invalid request in the first place and thus there is no actual attack vector.
Sure, a custom client could be created to submit such a request. But a custom client could also be made to interpret the servers response in a non-standard way or visit a malicious URL directly without even contacting your server. This means in this case the client itself would be the problem and not the servers response.
I am trying to test a call to my local Django development server. This is an AJAX call that returns a JSON object. The Django view does this:
return JsonResponse(response, safe=False)
I test this by launching the Django server (which runs the back end) and then launching my browser and opening the HTML page I have modified. These pages are not served by Django.
In Firefox, I can make the ajax call on the page. It returns the correct data, with the correct MIME type.
In Chrome, I can not. I get the following error:
Cross-Origin Read Blocking (CORB) blocked cross-origin response http://localhost:8000/dataload/alljson with MIME type application/json. See https://www.chromestatus.com/feature/5629709824032768 for more details.
I have tried to search for the solution to this problem. I've checked this question, but the suggestion to clear my cache doesn't do anything. I've also found suggestions that involve using django-cors-headers, but this is CORB (not CORS). Nonetheless, I've installed corsheaders and set CORS_ORIGIN_ALLOW_ALL = True to eliminate that as a possibility.
My MIDDLEWARE setting has this
MIDDLEWARE = [
...
"corsheaders.middleware.CorsMiddleware",
"django.middleware.common.CommonMiddleware",
...
]
The response headers I get in Firefox are:
Content-Length 12827
Content-Type application/json
Date Wed, 06 Feb 2019 18:38:03 GMT
Server WSGIServer/0.2 CPython/3.6.7
Vary Origin
X-Frame-Options SAMEORIGIN
I am calling http://localhost:8000/dataload/alljson. If I call http://localhost:8000/dataload/alljson/ (with the trailing slash), I get a 404.
Still no dice.
How can I get Chrome to allow the response from my development Django server for this ajax call?
All,
Every HTTP Request I make to my test REST Service is sent with the method set to GET. Tomcat rejects with a 405 - Unsupported Method. Doesn't matter what I change it to (POST, PUT, etc) Jmeter always sends a GET.
I set up the simplest possible test case by creating a Threadgroup with an HTTP Request Sampler and a View Results Tree. I send a JSON body to the REST Services which just echos back the request along with an ID. Works great with Google's REST Client UI.
Here is the result from the View Results Tree:
Response code: 405
Response message: Method Not Allowed
Response headers:
HTTP/1.1 405 Method Not Allowed
Server: Apache-Coyote/1.1
Allow: POST
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 1045
Date: Fri, 18 Jul 2014 21:39:27 GMT
Here is the RequestMapping from my REST Service
#RequestMapping(method = RequestMethod.POST, consumes = "application/json", produces = "application/json")
Here are some screenshots from my results. I wonder why there are two URI's below the HTTP Request in the tree? Notice the POST request looks correct.
Since the right answer is not provided yet: It's the "Follow Redirects" Option that causes this behavoir under certain circumstances.
see http://www.sqaforums.com/showflat.php?Cat=0&Number=687068&Main=675937
Try to end the 'Path' value of HTTP Request with '/'. It has to remove the GET result in View Results Tree.
I had the same problem. I tried everything also I read this question and all answers before find the thing that worked for me.
Content-Type should be application/json. It can not be text/html.
Set that in HTTP Header Manager. I assume you have set authentication details correctly.
We need to have three things properly set.
Content type which will be application/json
set the endpoint correctly in the path ,which you can see in soup ui
Check the port number on which the api wil get run on [All this u can first check on soupui and then try running the same in jmeter
I have a strange problem with JMeter.
I've made recording of some sort of web application without any problems. Problem appears during playback of test.
For some reason I receive different results during playback than during recording.
When I compare Http Request made during recording and playback I don't see a single difference (except for some security token which I'm extracting from earlier requests and passing as parameter).
To be more exact during recording I receive a response with a big body (>5kB), and during playback body of response is empty. Response code is 200 (OK).
This body contains crucial data from database, so I'm afraid that measurement made by this JMeter script will not reflect actual behavior of application, simply I will not measure what I really need.
Now my questions:
is there some tool or JMeter plug-in which will allow more effectively see contents of HTTP requests and its responses? It would be great If I could compare of requests made during recording and playback. So far I used two listeners: "View Results Tree". I've sandwiched between them to compare request from recording and playback.
is there some known bug in JMeter which could explain the difference? For example something related to recording process?
Here is example of request:
POST http://10.133.27.81:8080/c/portal/render_portlet
POST data:
p_l_id=69210&p_p_id=blank_WAR_Blank_INSTANCE_iNM3&p_p_action=0&p_p_state=normal&p_p_mode=view&p_p_col_id=column-2&p_p_col_pos=1&p_p_col_count=2
[no cookies]
Request Headers:
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Accept-Language: pl
Accept: */*
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
csrf_token: 1GXK-0QD7-GFPJ-JLDG-JP2G-J390-BFLG-7LL7
Pragma: no-cache
Method: POST /c/portal/render_portlet HTTP/1.1
X-Requested-With: OWASP CSRFGuard Project
Referer: http://10.133.27.81:8080/group/bou
Accept-Encoding: gzip, deflate
Content-Length: 143
Host: 10.133.27.81:8080
Update: to make sure which headers or parameters are constant I made 4 recordings of same test case during different sessions and compared them, so I'm quite sure that only csrf_token has to be field with value fetched from other request. I've added debug sampler to verify that this value is fetched properly.
Update 2: Problem found.
There where two problems:
There is a bug in JMeter when you do a search (Ctrl-F) it searches the whole project except for HTTP Header Menagers and my request contained csrf_token inside of header (I detected that before posting this question). Making a search in xml using text editor was good workaround for that.
when I try to find source of problems, before I've found problem number one, I've added a new problem by removing a HTTP Cookie Manager (I'm blaming myself and IE for this).
Generally changing Internet Explorer to FireFox with HttpFox add-on help to spot the problem.
Thanks everyone for support.
Marek
Response code 200 doesn't have to mean that everything went well at the application level.
To find out more details you can use Debug sampler and Debug PostProcessor.
Example here.
Your issue comes certainly from a missing dynamic request parameter that you didn't compute.
Look for example at csrf_token header Did you make it variable? Or do you transmit its initial recorded valur, but also at any parameter that contains some hashdata or numeric data referencing some content that does not exist in your page or request.
For example col pos I see p_p_col_id and related parameters, are you sure they reference something in your replay.
There is really very little chance of a JMeter bug in this case.
I am debugging my website. When it has an error, the full text form of the HTTP request that caused the error is logged. I want to be able to replay these HTTP requests to help debugging the error.
For instance, I have this in my log now:
POST /ipn/handler.ashx?inst=272&msgType=result HTTP/1.0
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: mysite.com
Content-Length: 28
User-Agent: AGENT/1.0 (UserAgent)
region=website&compName=ACTL
I want a way to make this exact request again on my local test machine (with changed Host attribute). What is the best way to do this?
You could use telnet to talk to your web server and type the exact requests.
You could also use libcurl (& curl) to make a program which is an HTTP client.
And many scripting languages (Python, Ruby, Perl, Ocaml, ...) also have HTTP client libraries (sometimes above Curl).