I'm trying to get 'https://translate.google.com' via luasocket.
Headers is based on HttpFox'es output;
My attempt to get content:
local r, c, h, fc = http.request { -- result (1 or nil on error), code (should be 200), headers, fancy code
url = 'http://translate.google.com';
method = 'GET';
sink = sink;
headers = {
['User-Agent'] = 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:55.0) Gecko/20100101 Firefox/55.0';
['Accept'] = 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8';
['Host'] = 'translate.google.com';
['Accept-Encoding'] = 'gzip, deflate, br';
['Accept-Language'] = 'ru,ru-RU;q=0.8,en;q=0.5,en-US;q=0.3';
['DNT'] = '1';
['Upgrade-Insecure-Requests'] = '1';
['Connection'] = 'close';
};
}
sink is valid ltn12 sink;
Result:
Code: HTTP/1.1 302 Found
Headers: {
content-type : "text/html; charset=UTF-8"
connection : "close"
content-length : "226"
x-xss-protection : "1; mode=block"
p3p : "CP="This is not a P3P policy! See https://www.google.com/support/accounts/answer/151657?hl=en for more info.""
server : "HTTP server (unknown)"
cache-control : "private, max-age=0"
content-language : "ru"
date : "Tue, 03 Oct 2017 13:50:21 GMT"
set-cookie : "NID=113=CO3BdznV6UYwcIZoIdF9F7dW1Cooi5ZVmNML0cQI6kA_TvfNig8xRQgS5E9dSKnIZxcfR3jxUbo3RA-7AEsqOCakciOo7Swtrcvz70Cmpm5M-_m1UQYTBBCg8VyNxzBW; expires=Wed, 04-Apr-2018 13:50:21 GMT; path=/; domain=.google.com; HttpOnly"
x-content-type-options : "nosniff"
expires : "Tue, 03 Oct 2017 13:50:21 GMT"
location : "https://translate.google.com/"
}
Body:
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
here.
</BODY></HTML>
What is wrong with my request? Why firefox got correct page and me not?
Code: HTTP/1.1 302 Found
location : "https://translate.google.com/"
It's a redirect to HTTPS location, but luasocket doesn't "know" how to deal with https, so it doesn't follow the redirect. If you install luasec and replace local http = require "socket.http" with local http = require "ssl.https", you should get back something like the following:
1 200 table: 0x000274e0 HTTP/1.1 200 OK
Related
i am working on an ESP32 project. one of my goals is to communicate with the ESP32 from a website using javascript fetch or XMLHttpRequest().
the ESP32 is connected to my local network and i am using the esp32_https_server library. it uses a self-signed certificate which the browser indicates as valid (but issues a warning, "Connection not protected" due to the self-signed certificate). the website has a CA certificate and is secure.
in testing, the esp32 is conected via USB to my computer, idealy i would like it to stand alone.
the problem i am experiencing is that i cannot seem to connect to the esp32. i keep getting status code 499 errors.
my questions are:
1) how do i successfully connect to the esp32 server from a secure website to get data frome the esp32?
2) how do i do this when the esp32 is not connected to my pc via the usb cable?
please see more info regarding the esp32 set up and responses below.
here's the esp32 code:
ResourceNode *nodeRoot = new ResourceNode("/", "GET", [](HTTPRequest *req, HTTPResponse *res) {
ResourceParameters *params = req->getParams();
std::string action = params->getRequestParameter("action");
String aksie = action.c_str();
Serial.println("Aksie: " + aksie);
if (aksie != "upload_data" && aksie != "upload_current_temp")
{
// this should be home page displayed
// Set the response status
res->setStatusCode(200);
res->setStatusText("success");
res->println("Secure Hello World!!!");
}
else
{
// either uploads..
processParams(aksie, res);
}
});
secureServer->registerNode(nodeRoot);
and here's the code that processes the "upload_current_temp" request:
if (action == "upload_current_temp")
{
// get random temperature
int currentTemp = random(0, 9);
String temp = String(currentTemp);
Serial.println("upload current temperature");
Serial.println("uploadCurrentTemp: " + temp);
std::string tem = temp.c_str();
// Set the response status
res->setStatusCode(200);
res->setStatusText("success current temperature");
StaticJsonDocument<200> doc;
doc["temperature"] = temp;
// Produce a minified JSON document
String output;
serializeJson(doc, output);
Serial.println("curent temp json output: " + output);
deserializeJson(doc, output);
// Set the content type of the response
res->setHeader("Content-Type", "application/json");
res->setHeader("Access-Control-Allow-Origin", "*");
res->setHeader("Access-Control-Allow-Methods", "GET,POST,OPTIONS");
// As HTTPResponse implements the Print interface, this works fine. Just remember
// to use *, as we only have a pointer to the HTTPResponse here:
serializeJson(doc, *res);
}
and also in setUp() i have this line:
secureServer->setDefaultHeader("Access-Control-Allow-Origin", "*"); //replace * with actual address
when using:
const xhr = new XMLHttpRequest();
const url = 'https://192.168.0.102/?action=upload_current_temp';
xhr.open('GET', url);
xhr.responseType = 'text';
xhr.onload = function () {
const data = xhr.response;
console.log(data);
if (this.readyState == 4 && this.status == 200) {
var obj = JSON.parse(this.responseText);
console.log("getCurTemp(), responseText: " + JSON.stringify(this.responseText, null, 2));
currentTemperature = obj.temperature;
console.log("current temperature: " + currentTemperature);
document.getElementById('currentTemp').innerHTML = currentTemperature;
}
};
xhr.send();
i get these errors (in opera):
499 (Request has been forbidden by antivirus)
has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
and in chrome:
has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
with these headers (opera):
Request URL: https://192.168.0.102/?action=upload_current_temp
Request Method: GET
Status Code: 499 Request has been forbidden by antivirus
Remote Address: 192.168.0.102:443
Referrer Policy: no-referrer-when-downgrade
Cache-Control: no-store, no-cache, must-revalidate, max-age=0
Connection: close
Content-Length: 52266
Content-Type: text/html; charset=utf-8
Expires: Mon, 04 Dec 1999 21:29:02 GMT
Pragma: no-cache
Accept: /
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Host: 192.168.0.102
Origin: https://istimuli.co.uk
Referer: https://istimuli.co.uk/?code=66b72f8e-400c-4adb-ad42-f4efec391d06
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36 OPR/67.0.3575.79
action: upload_current_temp
and when using :
var url = "https://192.168.0.102/?action=upload_current_temp";
var request = new Request(url, {
method: 'GET',
mode: 'cors', // no-cors, *cors, same-origin
headers: {
'Content-Type': 'application/json'
}
});
fetch(request).then(function (response) {
// Convert to JSON
return response.json();
}).then(function (data) {
console.log("temp: " + JSON.stringify(data));
return data;
}).catch(function (error) {
console.log('Request failed', error)
return 000;
});
i get these errors in opera:
499 (Request has been forbidden by antivirus)
has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
and in chrome:
has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
and these are the headers (opera):
1 requests
51.3 KB transferred
51.0 KB resources
Request URL: https://192.168.0.102/?action=upload_current_temp
Request Method: OPTIONS
Status Code: 499 Request has been forbidden by antivirus
Remote Address: 192.168.0.102:443
Referrer Policy: no-referrer-when-downgrade
Cache-Control: no-store, no-cache, must-revalidate, max-age=0
Connection: close
Content-Length: 52266
Content-Type: text/html; charset=utf-8
Expires: Mon, 04 Dec 1999 21:29:02 GMT
Pragma: no-cache
Accept: /
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Access-Control-Request-Headers: content-type
Access-Control-Request-Method: GET
Connection: keep-alive
Host: 192.168.0.102
Origin: https://istimuli.co.uk
Referer: https://istimuli.co.uk/?code=66b72f8e-400c-4adb-ad42-f4efec391d06
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36 OPR/67.0.3575.79
action: upload_current_temp
I'm using a react app running on localhost:3000 which makes ajax requests to our website. We recently switched our authentification system from using WordPress authentification to https://github.com/delight-im/PHP-Auth.
Since then, using the same settings inside ajax and on our web server, our authentification cookies are not sent cross domain. However, it's working when requesting them from the same domain.
Our request:
fetchLoginStatus = () => {
const ajax = new XMLHttpRequest();
ajax.withCredentials = true;
ajax.open("POST", "https://our-website.com/src/php/checkLoggedIn.php");
ajax.onload = () => {
const response = JSON.parse(ajax.responseText);
};
ajax.send();
};
Our request headers (from localhost:3000):
:authority: my-website.com
:method: POST
:path: /src/php/checkLoggedIn.php
:scheme: https
accept: */*
accept-encoding: gzip, deflate, br
accept-language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
content-length: 0
cookie: plesk-items-per-page; plesk-sort-field, phpMyAdmin; databases-active-list-state-collapsed; plesk-list-type; io=R_dL3fjUEYe64ykHAAAp; isAsyncProgressBarCollapsed=true; PLESKSESSID; plesk-sort-dir;
origin: https://localhost:3000
referer: https://localhost:3000/
Our response headers (we are running an nginx server):
access-control-allow-credentials: true
access-control-allow-headers: origin, x-requested-with, content-type
access-control-allow-methods: PUT, GET, POST, DELETE, OPTIONS access-`
control-allow-origin: https://localhost:3000
cache-control: no-store, no-cache, must-revalidate
content-encoding: br
content-type: text/html; charset=UTF-8
date: Sun, 10 Mar 2019 15:26:08 GMT
expires: Thu, 19 Nov 1981 08:52:00 GMT pragma:
no-cache server: nginx
set-cookie: PHPSESSID=someId;
path=/; SameSite=Lax status: 200
vary: Accept-Encoding
x-powered-by: PleskLin`
When I don't send the request cross-domain PHPSESSID is inside the cookies of my request headers. However when I send the request from localhost:3000 it's not there.
Does somebody know how I can send the PHPSESSID from localhost too?
Thanks for any help in advance!
Asked the same question inside the github repository and the owner solved it.
https://github.com/delight-im/PHP-Auth/issues/154
Solution:
vendor/delight-im/auth/src/UserManager.php
Replace Session::regenerate(true); with Session::regenerate(true, null);
vendor/delight-im/auth/src/Auth.php
Replace #Session::start(); with #Session::start(null);
Replace Session::regenerate(true); with Session::regenerate(true, null);
After $cookie->setSecureOnly($params['secure']); append $cookie-
>setSameSiteRestriction(null); in all three (!) occurrences
I try to log into Booking.com using Mechanize at this URL : https://admin.booking.com/hotel/hoteladmin/
So far, it's impossible for me to pass the login process. I am afraid that they use a javascript function during the send of the form to set the csrf_token. Here is the code I use :
login_url = "https://admin.booking.com/hotel/hoteladmin"
agent = Mechanize.new
agent.user_agent_alias = 'Mac Safari'
agent.verify_mode= OpenSSL::SSL::VERIFY_NONE
# Get the login page
page = agent.get(login_url)
form = page.form_with(:name => 'myform')
form.loginname = my_username
form.password = my_password
form.add_field!("csrf_token", "empty-token")
# Submit the form
page = form.submit( form.button_with(:name => "Login") )
When I load the page on my browser I get :
var token = '..................EXTRA-LONG-TOKEN..................' || 'empty-token',
But when I check it using Mechanize I get :
var token = '' || 'empty-token',
Please find the complete page body using Mechanize here.
So they use the javascript to set this variable in a new field created when we submit the form ?
if (
form &&
form.method &&
form.method.toLowerCase() === 'post' &&
typeof form.elements.csrf_token === 'undefined'
) {
input = doc.createElement( 'input' );
input.name = 'csrf_token';
input.type = 'hidden';
input.value = token;
form.appendChild( input );
}
I also try to take a look at Firebug in the Network tab without success. When we submit the form there is this sequence :
302 - POST - login.html
302 - GET - https://admin.booking.com/hotel/hoteladmin/index-hotel.html?page=&lang=xu&ses=89abb0da735818bc6252d69ece255276&t=1429195712.93074
302 - GET - https://admin.booking.com/hotel/hoteladmin/extranet_ng/manage/index.html?lang=xu&ses=89abb0da735818bc6252d69ece255276&hotel_id=XXXXXX&t=1429195713.11779
200 - GET - /home.html
When I check on the POST request I can see in "Data of the request" :
Content-Type: application/x-www-form-urlencoded
Content-Length: 95
ses=e7541870781128880d7c61aa1e4cc357&loginname=my_login&password=my_password&lang=xu&login=Login+
So, I don't know if the csrf_token from above is used or not and if it is I don't know where. And I don't know if it is the csrf_token that blocks me from logging in.
Here is the request/response headers from my browser for a success login :
---------- Request ----------
Host: admin.booking.com
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:37.0) Gecko/20100101 Firefox/37.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Referer: https://admin.booking.com/hotel/hoteladmin/login.html
Cookie: cwd-extranet=1; ecid=RtSy3w%2Fk5BG5Z67OY8E2rQZz; slan=xu; auth_token=569054884; ut=e; _ga=GA1.2.357900853.1429171802
Connection: keep-alive
---------- Response ----------
Connection: keep-alive
Content-Type: text/html; charset=UTF-8
Date: Thu, 16 Apr 2015 14:57:24 GMT
Location: /hotel/hoteladmin/index-hotel.html?page=&lang=xu&ses=8df70f6f7699cf5c5d63271fbbb47bb1&t=1429196244.67621
Server: nginx
Set-Cookie: cwd-extranet=1; path=/; expires=Tue, 14-Apr-2020 14:57:24 GMT
slan=xu; path=/; expires=Wed, 18-May-2033 03:33:20 GMT; HttpOnly
Strict-Transport-Security: max-age=2592000
Transfer-Encoding: chunked
And here is headers from Mechanize, login failed (no location on response header ?) :
form encoding: utf-8
query: "ses=e1520f97a6e9056940b4cf4e90684836&loginname=my_login&password=my_password&lang=xu&csrf_token=empty-token"
Net::HTTP::Post: /hotel/hoteladmin/login.html
request-header: accept-encoding => gzip,deflate,identity
request-header: accept => */*
request-header: user-agent => Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/534.51.22 (KHTML, like Gecko) Version/5.1.1 Safari/534.51.22
request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
request-header: accept-language => en-us,en;q=0.5
request-header: host => admin.booking.com
request-header: referer => https://admin.booking.com/hotel/hoteladmin/
request-header: content-type => application/x-www-form-urlencoded
request-header: content-length => 105
status: Net::HTTPOK 1.1 200 OK
response-header: server => nginx
response-header: date => Thu, 16 Apr 2015 14:39:22 GMT
response-header: content-type => text/html; charset=UTF-8
response-header: transfer-encoding => chunked
response-header: connection => keep-alive
response-header: vary => Accept-Encoding
response-header: x-powered-by => en105admapp-04
response-header: strict-transport-security => max-age=2592000
response-header: content-encoding => gzip
Thanks for your help
I managed to resolve the issue without taking care of the CSRF Token.
What I did is to follow the POST/GET sequence found with Firebug, only the SES token which can be found on the login form (hidden) is important.
So for the login POST we have :
uri = URI.parse("https://admin.booking.com/hotel/hoteladmin/login.html")
data = URI.encode("lang=en&login=Login&ses=#{token}&loginname=#{username}&password=#{password}")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
request = Net::HTTP::Post.new(uri.request_uri)
request.body = data
request['Cookie'] = cookie
response = http.request(request)
cookie = response.response['set-cookie']
location = response.response['location']
And then we follow the redirection with the previous cookie & location until we get a 200 response code with something like :
uri = URI.parse("https://admin.booking.com#{location}")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
request = Net::HTTP::Get.new(uri.request_uri)
request['Cookie'] = cookie
response = http.request(request)
cookie = response.response['set-cookie']
location = response.response['location']
Here's my problem, I'm trying to add the following Header to my GET ajax request :
Authorization: Basic XXXXXXX
So I wrote this :
function() {
var storage = window.localStorage;
var currentUsername = storage.getItem("username");
var currentPassword = storage.getItem("password");
var auth = makeBasicAuth(currentUsername, currentPassword);
$.ajaxSetup ({
beforeSend: function(request) {
request.setRequestHeader( "Authorization", auth );
}
});
I already have check the value of auth which is correct. And when I try to send my request, I get that :
OPTIONS <myurl> Error 401
Here are the headers of the request and of its answer :
Request :
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding gzip, deflate
Accept-Language fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3
Access-Control-Request-He... authorization
Access-Control-Request-Me... GET
Connection keep-alive
Host <myhost>
Origin null
User-Agent Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:22.0) Gecko/20100101 Firefox/22.0
Answer :
Access-Control-Allow-Head... Authorization
Access-Control-Allow-Meth... *
Access-Control-Allow-Orig... *
Access-Control-Max-Age 3628800
Connection Keep-Alive
Content-Encoding gzip
Content-Length 440
Content-Type text/html;charset=utf-8
Date Tue, 23 Jul 2013 13:00:58 GMT
Keep-Alive timeout=15, max=100
Server Apache-Coyote/1.1
Set-Cookie JSESSIONID=03A48D00EA04DE16396824E804E914AC; Path=/...
Vary Accept-Encoding
WWW-Authenticate Basic realm="Name Of Your Realm"
Note : I get this result whatever is the header I try to add.
Where is the problem ?
Thanks ^^.
I use the following code:
- (NSURLRequest *)connection:(NSURLConnection *)connection willSendRequest:(NSURLRequest *)request redirectResponse:(NSHTTPURLResponse *)response {
NSLog(#"Received redirect Response: %# %#", [response allHeaderFields], [NSHTTPURLResponse localizedStringForStatusCode:[response statusCode]]);
return request;
}
When I receive a 302 with the following header data:
< HTTP/1.1 302 Found
< Date: Wed, 03 Mar 2010 07:47:17 GMT
< Server: lighttpd/1.4.19
< Content-length: 0
< Content-type: text/html;charset=utf-8
< Location: `<new Location>`
< Vary: Accept-Encoding
this is the output in gdb console:
2010-03-03 08:42:03.265 MyProg[68106:207] Received redirect Response:
(null) server error 2010-03-03 08:42:14.414 MyProg[68106:207]
Received redirect Response: {
Connection = "Keep-Alive";
"Content-Encoding" = gzip;
"Content-Length" = 20;
"Content-Type" = "text/html;charset=utf-8";
Date = "Wed, 03 Mar 2010 07:42:10 GMT";
"Keep-Alive" = "timeout=15, max=100";
Location = "<new Location>";
Server = "lighttpd/1.4.19";
Vary = "Accept-Encoding"; } found
When using Curl I only get one response and tracedump tells the same, so I am sure that the server sends only one redirect.
Why is this selector called twice?
connection:willSendRequest:redirectResponse: gets called before every request, so it is called once on the original request, which was not a redirect so response is nil; then it gets called when loading the redirection target, where response is the 302 response to the initial request.