Why does requests library fail on this URL? - debugging

I have a url. When I try to access it programmatically, the backend server fails (I don't run the server):
import requests
r = requests.get('http://www.courts.wa.gov/index.cfm?fa=controller.managefiles&filePath=Opinions&fileName=875146.pdf')
r.status_code # 200
print r.content
When I look at the content, it's an error page, though the status code is 200. If you click the link, it'll work in your browser -- you'll get a PDF -- which is what I expect in r.content. So it works in my browser, but fails in Requests.
To diagnose, I'm trying to eliminate differences between my browser and Requests library. So far I've:
Disabled Javascript
Disabled (and deleted) cookies
Set the User-Agent to be the same in each
But I can't get the thing to work properly in Requests or fail in my browser due to disabling something. Can somebody with a better idea of browser-magic help me diagnose and solve this?

Does the request work in Chrome? If so, you can open the web inspector and right-click the request to copy it as a curl command. Then you'll have access to all the headers, params, and request body, which you can play around with to see which are triggering the failure you're seeing with the requests library.

You're probably running into a server that discriminates based on User-Agent. This works:
import requests
S = requests.Session()
S.headers.update({'User-Agent': 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)'})
r = S.get('http://www.courts.wa.gov/index.cfm?fa=controller.managefiles&filePath=Opinions&fileName=875146.pdf')
with open('dl.pdf', 'wb') as f:
f.write(r.content)

Related

How to simulate POST request?

I'm testing on Windows, trying to simulate POST requests (with different form variables) for load testing. I have tried all kinds of load testing software but failed to get it working.
For GET requests, I know I can just put parameters behind the url
http://www.example.com?id=yyy&t=zzz
But how do I simulate a POST request?
I have a chrome REST Client but I do not know what to put in the headers and data.
Here's what I've tried so far:
class Program
{
static void Main(string[] args)
{
string viewstateid = "/wEPDwUKLTY3NjEyMzE4NWRkK4DxZpjTmZg/RGCS2s13vkEWmwWiEE6v+XrYoWVuxeg=";
string eventid ="/wEdAAoSjOGPZYAAeKGjkZOhQ+aKHfOfr91+YI2XVhP1c/pGR96FYSfo5JULYVvfQ61/Uw4pNGL67qcLo0vAZTfi8zd7jfuWZzOhk6V/gFA/hhJU2fx7PQKw+iST15SoB1LqJ4UpaL7786dp6laCBt9ubQNrfzeO+rrTK8MaO2KNxeFaDhrQ0hxxv9lBZnM1SHtoODXsNUYlOeO/kawcn9fX0BpWN7Brh7U3BIQTZwMNkOzIy+rv+Sj8XkEEA9HaBwlaEjg=";
string username = "user1";
string password = "ttee";
string loginbutton = "Log In";
string URLAuth = "http://localhost/login.aspx";
string postString = string.Format("VIEWSTATE={0}&EVENTVALIDATION={1}&LoginUser_UserName={2}&LoginUser_Password={3}&LoginUser_LoginButton={4}",viewstateid,eventid, username, password,realm,otp,loginbutton);
const string contentType = "application/x-www-form-urlencoded";
System.Net.ServicePointManager.Expect100Continue = false;
CookieContainer cookies = new CookieContainer();
HttpWebRequest webRequest = WebRequest.Create(URLAuth) as HttpWebRequest;
webRequest.Method = "POST";
webRequest.ContentType = contentType;
webRequest.CookieContainer = cookies;
webRequest.ContentLength = postString.Length;
webRequest.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1";
webRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
webRequest.Referer = "http://localhost/login.aspx";
StreamWriter requestWriter = new StreamWriter(webRequest.GetRequestStream());
requestWriter.Write(postString);
requestWriter.Close();
StreamReader responseReader = new StreamReader(webRequest.GetResponse().GetResponseStream());
string responseData = responseReader.ReadToEnd();
Console.WriteLine(responseData);
responseReader.Close();
webRequest.GetResponse().Close();
}
}
It would be helpful if you provided more information - e.g. what OS your using, what you want to accomplish, etc. But, generally speaking cURL is a very powerful command-line tool I frequently use (in linux) for imitating HTML requests:
For example:
curl --data "post1=value1&post2=value2&etc=valetc" http://host/resource
OR, for a RESTful API:
curl -X POST -d #file http://host/resource
You can check out more information here-> http://curl.haxx.se/
EDITs:
OK. So basically you're looking to stress test your REST server? Then cURL really isn't helpful unless you want to write your own load-testing program, even then sockets would be the way to go. I would suggest you check out Gatling. The Gatling documentation explains how to set up the tool, and from there your can run all kinds of GET, POST, PUT and DELETE requests.
Unfortunately, short of writing your own program - i.e. spawning a whole bunch of threads and inundating your REST server with different types of requests - you really have to rely on a stress/load-testing toolkit. Just using a REST client to send requests isn't going to put much stress on your server.
More EDITs
So in order to simulate a post request on a socket, you basically have to build the initial socket connection with the server. I am not a C# guy, so I can't tell you exactly how to do that; I'm sure there are 1001 C# socket tutorials on the web. With most RESTful APIs you usually need to provide a URI to tell the server what to do. For example, let's say your API manages a library, and you are using a POST request to tell the server to update information about a book with an id of '34'. Your URI might be
http://localhost/library/book/34
Therefore, you should open a connection to localhost on port 80 (or 8080, or whatever port your server is on), and pass along an HTML request header. Going with the library example above, your request header might look as follows:
POST library/book/34 HTTP/1.0\r\n
X-Requested-With: XMLHttpRequest\r\n
Content-Type: text/html\r\n
Referer: localhost\r\n
Content-length: 36\r\n\r\n
title=Learning+REST&author=Some+Name
From here, the server should shoot back a response header, followed by whatever the API is programed to tell the client - usually something to say the POST succeeded or failed. To stress test your API, you should essentially do this over and over again by creating a threaded process.
Also, if you are posting JSON data, you will have to alter your header and content accordingly. Frankly, if you are looking to do this quick and clean, I would suggest using python (or perl) which has several libraries for creating POST, PUT, GET and DELETE request, as well as POSTing and PUTing JSON data. Otherwise, you might end up doing more programming than stress testing. Hope this helps!
Postman is the best application to test your APIs !
You can import or export your routes and let him remember all your body requests ! :)
EDIT : This comment is 5 yea's old and deprecated :D
Here's the new Postman App :
https://www.postman.com/
Simple way is to use curl from command-line, for example:
DATA="foo=bar&baz=qux"
curl --data "$DATA" --request POST --header "Content-Type:application/x-www-form-urlencoded" http://example.com/api/callback | python -m json.tool
or here is example how to send raw POST request using Bash shell (JSON request):
exec 3<> /dev/tcp/example.com/80
DATA='{"email": "foo#example.com"}'
LEN=$(printf "$DATA" | wc -c)
cat >&3 << EOF
POST /api/retrieveInfo HTTP/1.1
Host: example.com
User-Agent: Bash
Accept: */*
Content-Type:application/json
Content-Length: $LEN
Connection: close
$DATA
EOF
# Read response.
while read line <&3; do
echo $line
done
This should help if you need a publicly exposed website but you're on a dev pc. Also to answer (I can't comment yet): "How do I post to an internal only running development server with this? – stryba "
NGROK creates a secure public URL to a local webserver on your development machine (Permanent URLs available for a fee, temporary for free).
1) Run ngrok.exe to open command line (on desktop)
2) Type ngrok.exe http 80 to start a tunnel,
3) test by browsing to the displayed web address which will forward and display the local default 80 page on your dev pc
Then use some of the tools recommended above to POST to your ngrok site ('https://xxxxxx.ngrok.io') to test your local code.
https://ngrok.com/ ngrok
Dont forget to add user agent since some server will block request if there's no server agent..(you would get Forbidden resource response) example :
curl -X POST -A 'Mozilla/5.0 (X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0' -d "field=acaca&name=afadxx" https://example.com

Making requests from userscript to the localhost

So, I'm trying to make cross-site AJAX request from my own script to the localhost. In the userscript (running on Firefox's Scriptish engine) I'm loading my script like this
myscript_include.setAttribute('src', 'http://localhost/myscript.js?' + Math.random());
head.appendChild(myscript_include);
It works indeed. Then, in myscript.js, I try to read data from localhost (finally, I would like to make get-post requests to scripts on my localhost to add any needed functionality to the web-page without writing actual Firefox extension).
Following instructions on making cross-site AJAX requests I add to myscript.js:
$.getJSON('http://localhost/ajaxdata.json', function(json) {
alert(json.message);
});
Firefox JS console shows that GET request was actually made, and status is 200 OK. It even shows Content-Length 39, which is true indeed, but Response field remains empty and alert isn't shown!
What's wrong with that construction (except of it's horrible itself)? Is there some way to do what I want?
Not sure, but maybe adding Access-Control-Allow-Origin headers to localhost will solve this?
ref: https://developer.mozilla.org/En/HTTP_access_control

How to manually send HTTP POST requests from Firefox or Chrome browser

I want to test some URLs in a web application I'm working on. For that I would like to manually create HTTP POST requests (meaning I can add whatever parameters I like).
Is there any functionality in Chrome and/or Firefox that I'm missing?
I have been making a Chrome app called Postman for this type of stuff. All the other extensions seemed a bit dated so made my own. It also has a bunch of other features which have been helpful for documenting our own API here.
Postman now also has native apps (i.e. standalone) for Windows, Mac and Linux! It is more preferable now to use native apps, read more here.
CURL is awesome to do what you want! It's a simple, but effective, command line tool.
REST implementation test commands:
curl -i -X GET http://rest-api.io/items
curl -i -X GET http://rest-api.io/items/5069b47aa892630aae059584
curl -i -X DELETE http://rest-api.io/items/5069b47aa892630aae059584
curl -i -X POST -H 'Content-Type: application/json' -d '{"name": "New item", "year": "2009"}' http://rest-api.io/items
curl -i -X PUT -H 'Content-Type: application/json' -d '{"name": "Updated item", "year": "2010"}' http://rest-api.io/items/5069b47aa892630aae059584
Firefox
Open Network panel in Developer Tools by pressing Ctrl+Shift+E or by going Menubar -> Tools -> Web Developer -> Network. Select a row corresponding to a request.
Newer versions
Look for a resend button in the far right. Then a new editing form would open in the left. Edit it.
Older versions
Then Click on small door icon on top-right (in expanded form in the screenshot, you'll find it just left of the highlighted Headers), second row (if you don't see it then reload the page) -> Edit and resend whatever request you want
Forget the browser and try CLI. HTTPie is a great tool!
CLI HTTP clients:
HTTPie
Curlie
HTTP Prompt
Curl
wget
If you insist on a browser extension then:
Chrome:
Postman - REST Client (deprecated, now has a desktop program)
Advanced REST client
Talend API Tester - Free Edition
Firefox:
RESTClient
Having been greatly inspired by Postman for Chrome, I decided to write something similar for Firefox.
REST Easy* is a restartless Firefox add-on that aims to provide as much control as possible over requests. The add-on is still in an experimental state (it hasn't even been reviewed by Mozilla yet) but development is progressing nicely.
The project is open source, so if anyone feels compelled to help with development, that would be awesome: https://github.com/nathan-osman/Rest-Easy
* the add-on available from http://addons.mozilla.org will always be slightly behind the code available on GitHub
You specifically asked for "extension or functionality in Chrome and/or Firefox", which the answers you have already received provide, but I do like the simplicity of oezi's answer to the closed question "How can I send a POST request with a web browser?" for simple parameters. oezi says:
With a form, just set method to "post"
<form action="blah.php" method="post">
<input type="text" name="data" value="mydata" />
<input type="submit" />
</form>
I.e., build yourself a very simple page to test the POST actions.
I think that Benny Neugebauer's comment on the OP question about the Fetch API should be presented here as an answer since the OP was looking for a functionality in Chrome to manually create HTTP POST requests and that is exactly what the fetch command does.
There is a nice simple example of the Fetch API here:
// Make sure you run it from the domain 'https://jsonplaceholder.typicode.com/'. (cross-origin-policy)
fetch('https://jsonplaceholder.typicode.com/posts',{method: 'POST', headers: {'test': 'TestPost'} })
.then(response => response.json())
.then(json => console.log(json))
Some of the advantages of the fetch command are really precious:
It's simple, short, fast, available and even as a console command it stored on your chrome console and can be used later.
The simplicity of pressing F12, write the command in the console tab (or press the up key if you used it before) then press Enter, see it pending and returning the response is what making it really useful for simple POST requests tests.
Of course, the main disadvantage here is that, unlike Postman, this won't pass the cross-origin-policy, but still I find it very useful for testing in local environment or other environments where I can enable CORS manually.
Here's the Advanced REST Client extension for Chrome.
It works great for me -- do remember that you can still use the debugger with it. The Network pane is particularly useful; it'll give you rendered JSON objects and error pages.
For Firefox there is also an extension called RESTClient which is quite nice:
RESTClient, a debugger for RESTful web services
It may not be directly related to browsers, but Fiddler is another good software.
You could also use Watir or WatiN to automate browsers. Watir is written for Ruby and Watin is for .NET languages. I am not sure if it's what you are looking for, though.
http://watin.sourceforge.net/
http://watir.com/
There have been some other clients born since the rise of Postman that is worth mentioning here:
Insomnia: with both desktop application and Chrome plugin
Hoppscotch: previously known as Postwoman, and with a Chrome plugin available as well. You can also make it work locally with docker if you want to get funny
Paw: if you are on Mac
Advanced Rest Client: already mentioned as a Chrome plugin, but it is worth pointing out that it also has a desktop application
soapUI: written in Java and with lots of testing functionality
Boomerang: yet another way to test APIs. It comes with SOAP integration and it also has a Chrome plugin available
Thunder Client: if you use VS Code as your text editor then you should go and check out this awesome extension
Try Runscope. A free tool sampling their service is provided at https://www.hurl.it/.
You can set the method, authentication, headers, parameters, and body. The response shows status code, headers, and body. The response body can be formatted from JSON with a collapsable hierarchy.
Paid accounts can automate test API calls and use return data to build new test calls.
COI disclosure: I have no relationship to Runscope.
Check out http-tool for Firefox...
Aimed at web developers who need to debug HTTP requests and responses.
Can be extremely useful while developing REST based API.
Features:
GET
HEAD
POST
PUT
DELETE
Add header(s) to request.
Add body content to request.
View header(s) in response.
View body content in response.
View status code of response.
View status text of response.
So it occurs to me that you can use the console, create a function, and just easily send requests from the console, which will have the correct cookies, etc.
so I just grabbed this from here: https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch#supplying_request_options
// Example POST method implementation:
async function postData(url = '', data = {}, options = {}) {
// Default options are marked with *
let defaultOptions = {
method: 'POST', // *GET, POST, PUT, DELETE, etc.
mode: 'cors', // no-cors, *cors, same-origin
cache: 'no-cache', // *default, no-cache, reload, force-cache, only-if-cached
credentials: 'same-origin', // include, *same-origin, omit
headers: {
'Content-Type': 'application/json'
// 'Content-Type': 'application/x-www-form-urlencoded',
},
redirect: 'follow', // manual, *follow, error
referrerPolicy: 'no-referrer', // no-referrer, *no-referrer-when-downgrade, origin, origin-when-cross-origin, same-origin, strict-origin, strict-origin-when-cross-origin, unsafe-url
body: JSON.stringify(data) // body data type must match "Content-Type" header
}
// update the default options with specific options (e.g. { "method": "GET" } )
const requestParams = Object.assign(defaultOptions, options);
const response = await fetch(url, requestParams);
return response.text(); // displays the simplest form of the output in the console. Maybe changed to response.json() if you wish
}
IF YOU WANT TO MAKE GET REQUESTS, you can just put them in your browser address bar!
if you paste that into your console, then you can make POST requests by repeatedly calling your function like this:
postData('https://example.com/answer', { answer: 42 })
.then(data => {
console.log(data); // you might want to use JSON.parse on this
});
and the server output will be printed in the console (as well as all the data available in the network tab)
This function assumes you are sending JSON data. If you are not, you will need to change it to suite your needs
You can post requests directly from the browser with ReqBin.
No plugin or desktop application is required.
I tried to use postman app, had some auth issues.
If you have to do it exclusively using browser, go to network tab, right click on the call, say edit and send response. There is a similar ans on here about Firefox, this right click worked for me on edge and pretty sure it would work for chrome too
Windows CLI solution
In PowerShell you can use Invoke-WebRequest. Example syntax:
Invoke-WebRequest -Uri http://localhost:3000 -Method POST -Body #{ username='clever_name', password='hunter2' } -UseBasicParsing
On systems without Internet Explorer, you need the -UseBasicParsing flag.
The question being 12 years old now, it is easy to understand why the author asked a solution for Firefox or Chrome back then. After 12 years though, there are also other browsers and the best one which does not involve any add-ons or additional tools is Microsoft Edge.
Just open devtools (F12) and then Network Console tab (not the Network or Console tab. Click on + sign and open it, if it is not visible.).
And here is the official guide:
https://learn.microsoft.com/en-us/microsoft-edge/devtools-guide-chromium/network-console/network-console-tool
Have fun!

HTTP Referrer and IE7 and IE8

Hi i've tried the following to find the referrer in MSIE / IE7 and IE8 but its returning blank each time;
PHP:
<?
echo $_SERVER['HTTP_REFERER'];
?>
JAVASCRIPT:
document.write('Thanks for visiting from ' + document.referrer);
Does any know what the issue could be I'm referering using document.location from a page on another domain and work fine with all other browsers minus MSIE.
Any help would be great!
The HTTP Referer header is not required by the HTTP Protocol :
It is only sent as an information
The browser can, or cannot, send it (Which would explain why you're getting it with some browsers, and not getting it with some others)
Some firewall / security software could remove it, I suppose, in some situations (I've seen that, some years ago, if I remember correctly)
It can be forged easily by the user
Which means that you cannot rely on the Referer for your application : you can use it to provide some additionnal functionnality, but your application must work even if it's not there, or not correct.
HTTP_REFERRER does't work in IE browser it works fine for all browsers like mozilla, safari, opera etc... Referrer method doesn't recognize in IE it will return null when we apply it for IE. Actually using HTTP_REFERRER itself is not a right criteria because we can't expect it will work or not as it is not a standard HTTP HEADER.
Here's a hack you might try:
var referLink = document.createElement('');
document.body.appendChild(referLink);
referLink.click();

Why does ie8's user agent return 'opera'?

My code at: http://www.mgxvideo.com/mgxcopy-dev/get_browser.php, returns Opera when I run IE8. My source is:
<?php
$browser = get_browser(null, true);
echo $browser['browser'];
?>
It doesn't. The get_browser() function is making educated (but ill-informed) guesses about which browser the user-agent is running. Your browser capabilities file is likely outdated, probably because it was made before IE8 was released. Update it here.
The real IE8 user-agent string looks something like this:
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0)
You should use a lower-level tool, like a packet trace or server logging or a header dump to see what is being sent.

Resources