Can PHP/or other server side software tell who sent an ajax get request, things like the user agent or ip geo location if nothing like that is sent in the ajax? I mean, can server technology detect that I have sent using say firefox and that i am in italy without needing me to have sent that info from the client side, or does it need to get that info from being fed through the ajax?
$.get(
'request',
{
foo : 'no user info here',
action : 'click'
});
Would the above request also transmit my ip (or anything other than foo or action) in the headers? When I inspect the headers of a request to say google images in the console > Network > Headers
I see from what I can tell the header sent to me for an image (when I make the request in ajax would something simular be sent to the server)...
Remote Address:173.194.40.31:443
Request URL:https://www.google.it/search?q=cat&safe=off&espv=2&biw=1280&bih=439&source=lnms&tbm=isch&sa=X&ei=hTLiVMS1KtbtavaugqAB&ved=0CAYQ_AUoAQ&dpr=1
Request Method:GET
Status Code:200 OK
Request Headers
:authority:www.google.it
:method:GET
:path:/search?q=cat&safe=off&espv=2&biw=1280&bih=439&source=lnms&tbm=isch&sa=X&ei=hTLiVMS1KtbtavaugqAB&ved=0CAYQ_AUoAQ&dpr=1
:scheme:https
accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
accept-encoding:gzip, deflate, sdch
accept-language:it-IT,it;q=0.8,en-US;q=0.6,en;q=0.4
avail-dictionary:e9KXljul
cookie:SID=DQAAABABAAB_FyDDYGPkx5neGQPWarf87-orPHLggOvof1m7ANKPwpxKRc0LBxJpvQqWpvF6_qk2G8rfG-ZIeNkXG6zGES07ZQpIKOxZNfNO_yX4o2VJaK2mWLUtR6alp6ANCVsJhhM9qe5CxTiS5iMsV0ehKtUQyhUe4qgV4ysI-0yNczd-Vf6i2ZwQpjXqP7OkepVDCdxowaWiGTN5oH8gazhEirTkwJHNeXvANPpdnH9agrh807WvnINDfHMabvNMX86yrhDztgJV8Gt93tMKz2mQIrt4U6kkWpVSJRNICz1l-mlqpAroR-0Gh1hkSSQjTljduf0Gw0brc5ghRzZjnUgRx8-R-Utw2a7jtFZZZ8r04sn9mA; HSID=AiyA-7nS0qOcJPXs4; SSID=AOqRsDMDbkeyP70To; APISID=G4pCmX4uO0A42AlZ/Aw1eRZuRDgbTcsS9u; SAPISID=Fyq-DT3qTKlXX0BV/AFeL6M-cBLSA80AMA; NID=67=qSXRNvV7cf8TvEKdQ-hXpivK8o5iB68hmamjpPgcab9LaF2gi8YcLEZ3a0CkZE_YtRA1-xEbcBy6OF5kQQWvsKkh7wfly_zGcT007kA3rtTybNZxAWyc3ZgQw5h5ZcKYB8uh4rMMkqmloRbVJqsGtLni46mjevXs8xQ6qELItYJzwEkuocVE8ZS3nGIkduBPwOnktmEjTQ; PREF=ID=da3065f670165ae8:U=d4b51905e77a1373:FF=4:LD=en:TM=1420582780:LM=1424110248:SG=1:S=_-WZG1_sLNN6wYUA
referer:https://www.google.it/
user-agent:Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.94 Safari/537.36
I see that the remote address is an ip but the remote url is not. What confuses me is the user agent is sent back to me or is the above what got sent to the server with my request?
Related
I am trying to open and download pdfs using python requests based on urls I get from an API. This works for many of the files, but for files stored at one specific site I get a 500 Internal Server error response. In the respone there is a simple html with only the text: Not Authenticated.
When I paste the same url in Chrome I get the pdf. However I can see a "503 - Failed to load resource" error in the console as it failed to load some icon. Can this be relevant somehow?
The url also works when I run it in Postman with no headers at all.
I have seemingly the same issue as described in this question:
python requests http response 500 (site can be reached in browser)
However the fix of adding User-Agent to the header of the request does not help. Can there be some other header data required, and is there any way of checking what request my Chrome browser sends?
Update: I logged what request Chrome is sending and copyed the header to my python request. Still the same error. I have tried with our without the same cookie.
Here is my code:
import requests
headers = {'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'nb,en-GB;q=0.9,en-US;q=0.8,en;q=0.7',
'Connection': 'keep-alive',
'Cookie': 'JSESSIONID=a95b392a6d468e2188e73d2c296b; NSC_FS-NL-CET-XFC-IUUQ-8081=ffffffff3d9c37c545525d5f4f58455e445a4a4229a1; JSESSIONID=7b1dd39854eee82b2db41225150e',
'Host': url.split('/')[2],
'Upgrade-Insecure-Requests': '1',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'}
response = requests.get(url, headers=headers, verify=True)
I use Python 3.6.3
I found that I only get the error when I run the GET through requests. So I changed to using: urllib.request.urlopen(url)
More info about this approach here: Download file from web in Python 3
I have started using HP Loadrunner as part of a performance testing project.
We are trying to hit an initial URL which redirects to an identity server and then redirects to the web portal login screen.
For now I am trying in parallel to see the message exchange in both loadrunner log and browser log (just to compare the two request response messages). For some reason even though the initial HTTP GET message seems to be the same as the browser one the response coming back from the web portal is not the same. So at the location property when i get the rediction url the path is missing the port number at the redirect_uri point (...../url:portnumber2/.....). that creates an issue with the second redirection as instead of getting me to the proper redirection url it takes me to the error page redirection.
so the message exchange in browser looks like below:
Request URL:https://urladdress:portnumber1/
Request Method:GET
Status Code:302
Remote Address:10.33.5.83:4020
Referrer Policy:no-referrer-when-downgrade
Response Headers
content-length:0
date:Fri, 25 Aug 2017 06:50:47 GMT
location:/connect/authorize?client_id=...&redirect_uri=https%3A%2F%2Furladdress%3Aportnumber%2F....sign........
server:.......
set-cookie:.AspNetCore.OpenIdConnect........
set-cookie:.AspNetCore.........
status:302
x-powered-by:ASP.NET
x-ua-compatible:IE=Edge,chrome=1
Request Headers
:authority:urladdress:portnumber1
:method:GET
:path:/
:scheme:https
accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
accept-encoding:gzip, deflate, br
accept-language:en-US,en;q=0.8
upgrade-insecure-requests:1
user-agent:.......
second redirection
Request URL:https://urladdress:portnumber2/connect/authorize?......
Request Method:GET
Status Code:302
Remote Address:ipaddress:portnumber2
Referrer Policy:no-referrer-when-downgrade
Response Headers
content-length:0
date:Fri, 25 Aug 2017 06:50:47 GMT
location:https://urladdress:portnumber2/account/login?.......
server:......
status:302
x-powered-by:ASP.NET
Request Headers
:authority:urladdress:portnumber2
:method:GET
:path:/connect/authorize?......
:scheme:https
accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
accept-encoding:gzip, deflate, br
accept-language:en-US,en;q=0.8
cookie:.AspNetCore.......
upgrade-insecure-requests:1
user-agent:..........
and the message exchange in loadrunner looks the same with the only difference of not including the portnumber2 after the end of the return_uri property.
That results in being redirected to an error page instead of the second redirection.
The script we are using is shown below:
web_set_sockets_option("SSL_VERSION", "TLS1.2");
web_cleanup_cookies();
web_cache_cleanup();
web_add_auto_header("Accept-Encoding","gzip, deflate, br");
web_add_auto_header("Accept-Language","en-US,en;q=0.8");
web_add_auto_header("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
web_add_auto_header("Upgrade-Insecure-Requests","1");
web_add_auto_header("User-Agent","Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36");
web_set_option("MaxRedirectionDepth","0",LAST);
web_set_max_html_param_len("10000000");
web_reg_save_param_ex("ParamName=url1",
"LB=Location: ",
"RB=\r\n",
LAST);
web_url("urladdress:portnumber1",
"URL=https://urladdress:portnumber1",
"Resource=0",
"RecContentType=text/html",
"Referer=",
"Mode=HTTP",
LAST);
lr_message("redirected address 1 = %s " ,lr_eval_string("{url1}"));
web_reg_save_param_ex("ParamName=url2",
"LB=Location: ",
"RB=\r\n",
LAST);
web_url("connect",
"URL={url1}",
"Resource=0",
"RecContentType=text/html",
"Referer=https://urladdress:portnumber1",
"Mode=HTTP",
LAST);
return 0;
I need to know why the port number on a url is not fetched by the http response message( perhaps there is a configuration I need to create).
Any other comments that might help would be much appreciated.
Have you considered that having a double redirection, particularly ones which are temporary, are going to extract a large performance hit. You have two double handshakes in a row. This is bad magic, independent of the testing tool being used. It is poor architecture on the part of your designers/developers. Cite the defect - Go for a max redirect level of one.
I am using oauth2 to handle a user login via facebook. The error occurs when I call the authorization server in the golang api.
Here is the network error.
Fetch API cannot load https://www.facebook.com/dialog/oauth?client_id=1543358959292867&redirect_u…=email+public_profile&state=mUi4IpdY8yF5TNVVptMNNSn8IbVSZxJXTSEFM8Zg8LM%3D. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'null' is therefore not allowed access. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
network info (chrome)
Request URL:https://www.facebook.com/dialog/oauth?client_id=1543358959292867&redirect_uri=http%3A%2F%2Flocalhost%3A7001%2FFBLogin%2FCallback&response_type=code&scope=email+public_profile&state=lkmenB4FjNOShUQzL0Gpymi1xsvauaL7TawmUjCyvI4%3D
Request Method:GET
Status Code:302
Remote Address:[2a03:2880:f003:c1f:face:b00c:0:25de]:443
request headers
:authority:www.facebook.com
:method:GET
:path:/dialog/oauth?client_id=1543358959292867&redirect_uri=http%3A%2F%2Flocalhost%3A7001%2FFBLogin%2FCallback&response_type=code&scope=email+public_profile&state=KPbwnGWQoI7PHvwhJ_JvZ7RowPthjqpaDTgKVI5NHrM%3D
:scheme:https
accept:*/*
accept-encoding:gzip, deflate, sdch
accept-language:en-US,en;q=0.8
origin:null
referer:http://localhost:3000/
user-agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.86 Safari/537.36
Javascript function that calls server api.
let loginUrl = "http://localhost:7001/FBLogin" //server url
let config = {
method: 'GET',
mode: 'cors',
}
fetch(loginUrl, config).then(response => {
...
}
Golang function that calls oauth authorization server (facebook).
func FBLogin(w http.ResponseWriter, r *http.Request) {
state, err := hashutil.GenerateRandomState()
logutil.Fatal(err)
url := config["fb"].AuthCodeURL(state)
logutil.OauthSessionLogger(state)
http.Redirect(w, r, url, 302)
}
I set the proper headers on the server side following this guide. Setting HTTP headers in Golang
I can't seem to find where the error comes from. (facebook API developer console, the callback headers in the server code, or the javascript initiating the call? I spent a lot of time changing the server headers but the error still persists as a CORS issue.
The origin in my request header is null, is that something to consider?
edit added no-cors mode result.
FBLogin 302 text/html SessionActions.js?524b:35 736 B 5 ms
oauth?client_id=154335&redirect_uri=http%3A%2F%2Flocalhost%3A7001%2FFBLogin%2FCallback&D
302 text/html
http://localhost:7001/FBLogin 611 B 57 ms
login.php?skip_api_login=1&api_key=154335 200
All three calls were successful, but was never redirected to the oauth page even when it said so. The response type was 'opaque' with a status 0.
I am developing an iOS application utilizing the Ionic framework with Cordova and have run into a bit of a snag in my dev cycle. While running on my local dev web server, any RESTful call happening from the $http service is failing with a 401 Not Authorized error even though I am passing the Authorization header. Interestingly enough, the call works fine once I build the app and deploy onto the iOS Emulator.
FWIW, the api calls are to an Atlassian Confluence api.
CORS doesn't seem to be the issue as that has been configured, tested and working on non-authenticated calls.
Here is the very basic call that is failing within the browser but working on the emulator when using the same headers:
$http.defaults.headers.common['Accept'] = 'application/json';
$http.defaults.headers.common.Authorization = 'Basic xxxxxxxxxxxxxx';
$http.get('https://www.example.com/rest/prototype/1/search/site?type=blogpost&spaceKey=TESTSPACE&os_authType=basic')
.success(function (data, status) {
this.serviceData = data;
})
.error(function (data, status) {
console.log(status, data.result);
});
I am sending two custom headers: Accept: application/json and Authorization: Basic *** If I run the application in Chrome with the above service configuration, I see the following Request Headers. There is an added Access-Control-Request-Headers that mentions the Accept and Authorization headers, but those headers are not there. I see the following Request Headers going across the wire:
Accept:*/*
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8,nl;q=0.6
Access-Control-Request-Headers:accept, authorization <!---- This is added but there is no Authorization Header
Access-Control-Request-Method:GET
Cache-Control:no-cache
Connection:keep-alive
Host:www.example.com
Origin:https://local.example-client.com
Pragma:no-cache
Referer:https://local.example-client.com/
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.94 Safari/537.36
If I make the same call with the Authorization Header using the Advanced Rest Client Chrome extension, it works correctly.
Accept:application/json <!----------- This one is */* when called from $http
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8,nl;q=0.6
Authorization:Basic *** REMOVED *** <!----------- This one is missing when called from $http
Cache-Control:no-cache
Connection:keep-alive
Cookie: *** REMOVED ***
Host:www.example.com
Pragma:no-cache
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.94 Safari/537.36
One thing to note is that if I remove the Authorization header line (below), then the Accept header now appears:
//$http.defaults.headers.common.Authorization = 'Basic ************************';
Can anyone provide me any insight on what I may be missing here?
It's probably a problem with your CORS configuration. Authenticated requests require very specific configuration to work properly with CORS.
Make sure you're
Returning the Access-Control-Allow-Headers header to allow those headers
Returning the exact origin in Access-Control-Allow-Origin (not "*")
Setting withCredentials to true on the XHR (in Angular this can be set in the defaults just like the headers)
(I'm not entirely sure if the "Authorization" header is considered a "credential" if you set it by hand so the last two may or may not be required.)
The reason it works under Cordova is because Cordova doesn't implement origin restrictions for XHRs (they would be a bit pointless since your app doesn't have anything "local" to communicate with)
I've got a problem with an Ajax request in my web project. Most of the time it works just fine but sometimes there is simply no request from the client (it does not show in wireshark), FireBug just shows the request marked red.
Project looks like the following:
Java Backend with JSPs, Client is using the Firefox Browser
The request is done via prototypejs using the Ajax Updater function. I can debug the request preparation via firebug until the Ajax Update function fires the request but then no request is listed in my wireshark output (although all other request are listed just fine).
So I have a (sometimes) vanished Ajax request with no error and no clue where to look :( Any help would be much appreciated.
Request:
[...]/ajax/gruppe/loadTreeList.do?ajax=true&tstamp=645&context=GRUPPE&level=0&itemid=0
RequestHeader:
Accept text/javascript, text/html, application/xml, text/xml, /
Accept-Encoding gzip, deflate
Accept-Language de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Connection keep-alive
Cookie JSESSIONID=326D8AEAAFD254760CEC8D050734807E
Host localhost:8080
Referer [...]/domain/druck/gruppe.do
User-Agent Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20100101 Firefox/17.0
X-Prototype-Version 1.7.1
X-Requested-With XMLHttpRequest
There is no response.