Expect: 100-Continue header with XmlHTTPRequest - ajax

How do I force XmlHttpRequest to add Expect: 100-continue header? How can I make use of this feature in desktop browsers world?
var xmlhttp = new XMLHttpRequest();
var dataToSend = new FormData();
dataToSend.append('some', 'data');
dataToSend.append('token', 'secret-token');
xmlhttp.open("POST", "/post", true);
xmlhttp.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
xmlhttp.setRequestHeader("Expect", "100-continue");
xmlhttp.setRequestHeader("Custom-Header", "This is custom data");
xmlhttp.send(dataToSend);
Here is the TCP Dump output piece
POST /post HTTP/1.1
Host: 127.0.0.1:3000
Connection: keep-alive
Content-Length: 243
Origin: http://127.0.0.1:3000
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36
Custom-Header: This is custom data
Content-type: application/x-www-form-urlencoded
Accept: */*
Referer: http://127.0.0.1:3000/
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Cookie: connect.sid=s%3AAKgYIit0sVHMcI7na85UR_Na.o7uSDBEidLEyQ3cTaGyXlMrPiF3vADrwpnCrkCrJBD0
------WebKitFormBoundary9agFn5mlxh7SUBf7
Content-Disposition: form-data; name="some"
data
------WebKitFormBoundary9agFn5mlxh7SUBf7
Content-Disposition: form-data; name="token"
secret-token
------WebKitFormBoundary9agFn5mlxh7SUBf7--

You cannot force the XMLHttpRequest.setRequestHeader() method to add the Expect header for security reasons, as you can read in the W3C XMLHttpRequest specification:
The setRequestHeader(header, value) method must run these steps:
If the state is not OPENED, throw an "InvalidStateError" exception and terminate these steps.
If the send() flag is set, throw an "InvalidStateError" exception and terminate these steps.
If header does not match the field-name production, throw a "SyntaxError" exception and terminate these steps.
If value does not match the field-value production, throw a "SyntaxError" exception and terminate these steps (note: The empty string is legal and represents the empty header value).
Terminate these steps if header is a case-insensitive match for one of the following headers:
Accept-Charset
Accept-Encoding
Access-Control-Request-Headers
Access-Control-Request-Method
Connection Content-Length
Cookie
Cookie2
Blockquote
Date
DNT
Expect
Host
Keep-Alive Origin
Referer
TE
Trailer
Transfer-Encoding
Upgrade
User-Agent
Via
...or if the start of header is a case-insensitive match for Proxy- or Sec- (including when header is just Proxy- or Sec-).
The above headers are controlled by the user agent to let it control those aspects of transport. This guarantees data integrity to some extent. Header names starting with Sec- are not allowed to be set to allow new headers to be minted that are guaranteed not to come from XMLHttpRequest.
As a further reference:
Webkit Tests "set-dangerous-headers.html"
Some browsers (Chrome, for example) will also display an error in their "JavaScript Console":

Related

No response for XHR request in python with requests.get()

I want to scrape german poll data from a server. Here, I search for an examplary street, straße "Judengasse".
I have been trying to reproduce this. Unfortunately, the link from the reference is not intact anymore, so I couldn't directly compare it to my problem. Since I am fairly inexperienced, I do not know what is exactly needed to reproduce the request that is submitted via the web interface.
I don't now which attributes of the header are needed for my request to work and what of it might be redundant. In Chrome's inspect mode I see that in my case there are more header attributes than in the referenced example.
My code so far (which does not work) from trying to reproduce the SE post:
import requests
url = 'https://online-service2.nuernberg.de/Finder/action/getItems'
data = {
"finder":"Wahlraumfinder",
"strasse":"Judengasse",
"hausnummer":"0"
}
headers = {
'Host': 'online-service2.nuernberg.de',
'Referer': 'https://online-service2.nuernberg.de/Finder/?Wahlraumfinder',
'Accept': '*/*',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7',
'Connection': 'keep-alive',
'Content-Length': '312',
'Content-Type': 'multipart/form-data; boundary=----WebKitFormBoundaryeJZfrnZATOw6B5By',
'DNT': '1',
'Host': 'online-service2.nuernberg.de',
'Referer': 'https://online-service2.nuernberg.de/Finder/?Wahlraumfinder',
'Sec-Fetch-Mode': 'cors',
'Sec-Fetch-Site': 'same-origin',
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36',
'X-Requested-With': 'XMLHttpRequest'
}
response = requests.get(url, data=data, headers=headers)
I don't get a respone. I added all request headers to headers.
Not sure, if more headers are needed.
Further, I am not sure if the url is correct.
I am looking to generate output of the following form, for this specific request "Judengasse":
Nr 0652
Wahllokal Willstätt.-Gym., Innerer Laufer Platz 11
This corresponds to putting in "Judengasse" into the search bar and hitting go on
the search "Suche" and extracting parts of the first output box "Wahl-/Stimmbezirk"
When I look at the XHR in Chrome's dev mode:
General
Request URL: https://online-service2.nuernberg.de/Finder/action/getItems
Request Method: POST
Status Code: 200 OK
Remote Address: 193.22.166.102:443
Referrer Policy: no-referrer-when-downgrade
Response Header
Connection: Keep-Alive
Content-Length: 1149
Content-Type: application/json;charset=UTF-8
Date: Wed, 04 Dec 2019 00:21:30 GMT
Keep-Alive: timeout=5, max=100
Server: Apache
Request Header
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
Connection: keep-alive
Content-Length: 312
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryx2jHYJHo3ejnKw0l
DNT: 1
Host: online-service2.nuernberg.de
Origin: https://online-service2.nuernberg.de
Referer: https://online-service2.nuernberg.de/Finder/?Wahlraumfinder
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36
X-Requested-With: XMLHttpRequest
From Data
------WebKitFormBoundaryx2jHYJHo3ejnKw0l
Content-Disposition: form-data; name="action"
"action/getItems"
------WebKitFormBoundaryx2jHYJHo3ejnKw0l
Content-Disposition: form-data; name="data"
{"finder":"Wahlraumfinder","strasse":"Judengasse","hausnummer":"0"}
------WebKitFormBoundaryx2jHYJHo3ejnKw0l--
Thank you for reading.
After some research I finally managed to get a 200 response from this server.
Firstly, requests.get in this case should be replace by requests.post, since you want to replicate an HTTP POST request, according to the info you got from Chrome's dev mode, "General" section.
Secondly, from the headers we can see that the data is sent as being of type "multipart/form-data" request. As far as I could understand, this is a type of request that is used to send files instead of regular data (more about this type of request here).
So, I converted the string sent through the POST request to binary (this is achieved by prepending b) and passed it to the files parameter of the request. For some reason, this parameter requires a tuple (a, b) inside a set {c}, hence the {(None, data)}.
I also passed the street name as a parameter to data, so it's easier to manipulate it.
I got this working code (I'm using my browser's request):
import requests
url = 'https://online-service2.nuernberg.de/Finder/action/getItems'
street = b'Judengasse'
data = b'-----------------------------15242581323522\r\n' \
b'Content-Disposition: form-data; name=\"action\"\r\n\r\n' \
b'\"action/getItems\"\r\n-----------------------------15242581323522\r\n' \
b'Content-Disposition: form-data; name="data"\r\n\r\n' \
b'{\"finder\":\"Wahlraumfinder\",\"strasse\":\"%s\",\"hausnummer\":\"0\"}\r\n' \
b'-----------------------------15242581323522--' % street
headers = {"Host": "online-service2.nuernberg.de",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0",
"Accept": "*/*",
"Accept-Language": "en-US,en;q=0.5",
"Accept-Encoding": "gzip, deflate, br",
"X-Requested-With": "XMLHttpRequest",
"Content-Type": "multipart/form-data; boundary=---------------------------15242581323522",
"Content-Length": "321",
"Origin": "https://online-service2.nuernberg.de",
"DNT": "1",
"Connection": "keep-alive",
"Referer": "https://online-service2.nuernberg.de/Finder/?Wahlraumfinder",
}
multipart_data = {(None, data,)}
response = requests.post(url, files=multipart_data, headers=headers)
print(response.text)
I got this raw response:
{"id":"8c4f7a57-1bd6-423a-8ab8-e1e40e1e3852","items":[{"zeilenbeschriftung":"Wahl-/Stimmbezirk","linkAdr":null,"mapUrl":"http://online-service.nuernberg.de/Themenstadtplan/sta_gebietsgli
ederungen.aspx?p_urlvislayer=Stimmbezirke&XKoord=4433503.05&YKoord=5480253.301&Zaehler=1&Textzusatz=Judengasse+0&z_XKoord=4433670.0&z_YKoord=5480347.0&z_Zaehler=1&z_Textzusatz=Wahllokal%
20Willst%E4tt.-Gym.%2C+Innerer+Laufer+Platz+11","items":["0652","Judengasse, Neue Gasse","Willstätt.-Gym., Innerer Laufer Platz 11","Zi. 101 ,1. OG",null]},{"zeilenbeschriftung":"Stimmkr
eis Landtagswahl","linkAdr":null,"mapUrl":"http://online-service.nuernberg.de/Themenstadtplan/sta_gebietsgliederungen.aspx?p_urlvislayer=Stimmkreis_LTW&XKoord=4433503.05&YKoord=5480253.3
01&Zaehler=1&Textzusatz=Judengasse+0&p_scale=100000","items":["501","Nürnberg-Nord"]},{"zeilenbeschriftung":"Wahlkreis Bundestagswahl","linkAdr":null,"mapUrl":"http://online-service.nuer
nberg.de/Themenstadtplan/sta_gebietsgliederungen.aspx?p_urlvislayer=Wahlkreis_BTW&XKoord=4433503.05&YKoord=5480253.301&Zaehler=1&Textzusatz=Judengasse+0&p_scale=150000","items":["244","N
ürnberg-Nord"]}],"status":200}
which you can easily parse to get the result you expect:
print(response.json()["items"][0]["items"])
yilding...
['0652', 'Judengasse, Neue Gasse', 'Willstätt.-Gym., Innerer Laufer Platz 11', 'Zi. 101 ,1. OG', None]
Hope it helps.
Regards

GetMapping "produces" works even though doesn't match accept header

Intro
There is a #GetMapping attribute, as the following in one of our projects:
#GetMapping(path = "/", produces = SaConstants.SA_MEDIA_TYPE)
public HttpEntity<Resource<Home>> get(HttpServletResponse response) {
In the SaConstants class:
public static final String SA_MEDIA_TYPE="application/sa+json";
When I access the page from any internet browser, I am getting the proper response that I want - and my breakpoint in the controller is being triggered.
The browser is sending the following headers:
Host: 127.0.0.1:8001
Connection: keep-alive
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.104 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
DNT: 1
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8
Cookie: io=Qt74kp5V5ziUNIxlAAAG
When I make a request to the page, without an Accept header, the page is not working.
If I add to postman the following Accept header, everything works:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Question
My question are:
why does it work even if the Accept: header of the request doesn't match the produces attribute of the Rest Controller?
Why does it fail if no Accept header is provided (given the first question).
"*/*" means all types, this header is by default provided by most of the popular browsers

Angular's $http.post doesn't work, nor does its' $http... but jQuerys ajax does. Why?

For some reason this:
return jquery.ajax('my url', {
crossDomain : true
, data : JSON.stringify({"brand": self.current})
, type : 'POST'
}).success(function(data){
scope.results = data;
});
and/or this:
curl -X POST -H "Content-Type: application/json" -d '{"brand":"target"}' myUrl
work fine, but this:
var req = {
method: "POST"
, url : "my url"
, data : JSON.stringify({"brand": self.current})
};
return $http(req).
success(function(data){
scope.results = data;
});
fails miserably with
"OPTIONS my url (anonymous function) # angular.js:9866sendReq # angular.js:9667$get.serverRequest # angular.js:9383processQueue # angular.js:13248(anonymous function) # angular.js:13264$get.Scope.$eval # angular.js:14466$get.Scope.$digest # angular.js:14282$get.Scope.$apply # angular.js:14571(anonymous function) # angular.js:21571jQuery.event.dispatch # jquery.js:4430jQuery.event.add.elemData.handle # jquery.js:4116
(index):1 XMLHttpRequest cannot load my url. No 'Access-Control-Allow-Origin' header is present on the requested resource.
Origin 'http://localhost:5000' is therefore not allowed access. The response had HTTP status code 404."
They're the same url. Wtf.
I have a sneaking suspicion that the "crossDomain : true" option in jquery is why the jquery one works, but if that's the case, then the question is:
how do I do that with angular?
-- When using jquery's default ajax method, the scope isn't updating with the results, but i know the data is being assigned because i'm logging it out, and if i submit the request again, the scope does update with the second value.
Second question- why isn't my view updating with the results?
update:
The reason this is failing has nothing to do with the response I'm getting back from the server, the problem is that Angular is transforming this POST request into an OPTIONS request:
(taken from google chromes' xhr tool:)
Remote Address: the remote address
Request URL:the request endpoint
Request Method:OPTIONS <-------------
Status Code:404 Not Found
Further inspection reveals:
OPTIONS /my url HTTP/1.1 <--------------
Host: my urls host
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Access-Control-Request-Method: POST
Origin: http://localhost:5000
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36
Access-Control-Request-Headers: accept, charset, content-type
Accept: */*
Referer: http://localhost:5000/
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
which is not what it should be doing because I'm specifically saying in the req object i'm passing to $http that this is a POST request.
...
So how do I make angular... NOT do that?
also- why is it doing that?
When you do a cross-origin request from your browser, all browsers hit the URL (provided in AJAX call) to confirm if the cross-origin request is available or not which is known as preflight request. https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS
So, your server's endpoint must allow the preflight request in order to make this call work by setting some response headers like (an example in Groovy):
response.setHeader "Access-Control-Allow-Headers", "Content-Type"
response.setHeader "Access-Control-Allow-Methods", "POST,DELETE,PUT"
response.setHeader "Access-Control-Allow-Origin", "*"

Redirect as response to Ajax request ends up returning empty

We are using Primefaces 3M4 and one of our pages has a p:dataTable which uses ajax calls for events:
<p:ajax event="rowSelect" update=":newsForm:newsDlg" oncomplete="newsDlg.show();"/>
When the session times out the page gets redirected to /login.xhtml which works fine for non-ajax actions (menu items, etc) but when I select a row in the datatable after the session has expired the page doesn't change to the login page and in Firebug I see the following:
Under dashboard.xhtml Headers section of Firebug
Response Headers
Server Apache-Coyote/1.1
X-Powered-By JSF/2.0
Location http://localhost:8080/RetailerPortal/faces/login.xhtml
Content-Length 0
Date Fri, 11 Nov 2011 18:32:42 GMT
Request Headers
Host localhost:8080
User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0
Accept application/xml, text/xml, */*; q=0.01
Accept-Language en-us,en;q=0.5
Accept-Encoding gzip, deflate
Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection keep-alive
Content-Type application/x-www-form-urlencoded; charset=UTF-8
Faces-Request partial/ajax
X-Requested-With XMLHttpRequest
Referer http://localhost:8080/RetailerPortal/faces/dashboard.xhtml
Content-Length 389
Cookie csfcfc=_30Xsr; JSESSIONID=fg1bV1sZkzKIgNtkH0bz0N0f; JSESSIONID=C65BF4EED70299ABFE4B73614118295E
Under dashboard.xhtml Response
<?xml version='1.0' encoding='ISO-8859-1'?>
<partial-response><changes><update id="javax.faces.ViewState"><![CDATA[-3728406524126180805:2441995557020829808]]></update></changes></partial-response>
Under dashbaoard.xhtml Post
Parametersapplication/x-www-form-urlencoded
javax.faces.ViewState 7521050094575005695:7928145831130537413
javax.faces.behavior.even... rowSelect
javax.faces.partial.ajax true
javax.faces.partial.event rowSelect
javax.faces.partial.execu... newsForm:newsTable
javax.faces.partial.rende... newsForm:newsDlg
javax.faces.source newsForm:newsTable
newsForm newsForm
newsForm:newsTable_instan... 3
newsForm:newsTable_select... 3
Source
newsForm=newsForm&newsForm%3AnewsTable_selection=3&javax.faces.ViewState=7521050094575005695%3A7928145831130537413&javax.faces.partial.ajax=true&javax.faces.source=newsForm:newsTable&javax.faces.partial.execute=newsForm:newsTable&javax.faces.partial.render=newsForm:newsDlg&javax.faces.behavior.event=rowSelect&javax.faces.partial.event=rowSelect&newsForm:newsTable_instantSelectedRowKey=3
Under login.xhtml's headers
Response Headers
Server Apache-Coyote/1.1
X-Powered-By JSF/2.0
Cache-Control no-cache
Set-Cookie JSESSIONID=MdhyizD+8IkuFvLZD+6jWlUz; Path=/RetailerPortal
Content-Type text/xml;charset=UTF-8
Content-Length 196
Date Fri, 11 Nov 2011 18:32:42 GMT
Request Headers
Host localhost:8080
User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0
Accept application/xml, text/xml, */*; q=0.01
Accept-Language en-us,en;q=0.5
Accept-Encoding gzip, deflate
Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection keep-alive
Referer http://localhost:8080/RetailerPortal/faces/dashboard.xhtml
X-Requested-With XMLHttpRequest
Faces-Request partial/ajax
Content-Type application/x-www-form-urlencoded
Cookie csfcfc=_30Xsr; JSESSIONID=fg1bV1sZkzKIgNtkH0bz0N0f; JSESSIONID=C65BF4EED70299ABFE4B73614118295E
Under login.xhtml's XML section
XML Parsing Error: no element found Location: moz-nullprincipal:{6ccf85cf-5c69-438c-a9bb-e66423a36a48} Line Number 1, Column 1:
^
Response code
HttpServletResponse servResponse = (HttpServletResponse) response;
servResponse.sendRedirect("login.xhtml");
servResponse.setHeader("Access-Control-Allow-Origin", "*");
Just a guess--
If you're trying to redirect from an ajax response, you can't do that 301/302 style-- you have to send a message back to the browser and have the browser redirect via javascript.
Probably the non-ajax ones are working because they're using 301/302s.
I found the answer to this question in this blog
with the relevant code for at the bottom of the blog post in the doRedirect method.

Guice servlet 405 err: seems to not set Content-type set to "text/json" for MediaType.APPLICATION_JSON

I'm making a call from jQGrid to a Guice servlet that has the following binding:
#Produces({MediaType.APPLICATION_JSON})
#Path("/{param}")
public String getJson(#PathParam("param") String param) {
...
return return json.toString();
}
Requesting the url directly, I can see the JSON. When jqgrid executes the request, I get 405 method not allowed response. I've seen this happen before when the returning page doesn't have the Content-type set to "text/json" (jqgrid is not very flexible here).
HERE IS THE REQUEST:
Key Value
Request POST /myapp/json/jqgrid/json ... HTTP/1.1
x-requested-with XMLHttpRequest
Accept-Language en-us
Referer http://localhost:8080/myapp/myPage...
Accept application/json, text/javascript, /
Content-Type application/x-www-form-urlencoded
Accept-Encoding gzip, deflate
User-Agent Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
Host localhost:8080
Content-Length 63
Connection Keep-Alive
Cache-Control no-cache
HERE IS THE RESPONSE:
Key Value
Response HTTP/1.1 405 Method Not Allowed
Server Apache-Coyote/1.1
Allow GET,OPTIONS,HEAD
Content-Type text/html;charset=utf-8
Content-Length 1034
Any thoughts on how to get the guice servlet to set the Content-type to "text/json" and allow the response?
This one is solved. I was using a #GET annotation and jQGrid was issuing a post. I changed the #POST and it started working. This may solve the problem for others with related 405 errors.

Resources