Websocket handshake: Incorrect 'Sec-WebSocket-Accept' header value - websocket

I am writing a websocket server in C++ and am not able to get the handshake to work. Chrome reports the error is due to a bad accept header, but I believe the value to be correct.
As one example exchange, the client sends the following key:
Sec-WebSocket-Key: ypX0m2zum/pt80mxlVo8PA==
and my server sends back:
Sec-WebSocket-Accept: Kl4mnqm5QA6bBmGf3EAN0nyGXws=
I have tested my server against the example in the RFC and it checks out. I don't know why its not being accepted. My theory is that I must be doing something else that generates the same error as a bad accept value.
Here is different request from a wireshark capture:
Hypertext Transfer Protocol
GET /websocket HTTP/1.1\r\n
Host: 127.0.0.1:8443\r\n
Connection: Upgrade\r\n
Pragma: no-cache\r\n
Cache-Control: no-cache\r\n
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36\r\n
Upgrade: websocket\r\n
Origin: chrome-extension://eajaahbjpnhghjcdaclbkeamlkepinbl\r\n
Sec-WebSocket-Version: 13\r\n
Accept-Encoding: gzip, deflate, br\r\n
Accept-Language: en-US,en;q=0.9\r\n
Sec-WebSocket-Key: +zJ3/KI/Zrumgh+AjxopRQ==\r\n
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits\r\n
\r\n
[Full request URI: http://127.0.0.1:8443/websocket]
[HTTP request 1/1]
[Response in frame: 6]
And here is the response:
Hypertext Transfer Protocol
HTTP/1.1 101 Switching Protocols\r\n
Upgrade: websocket\r\n
Connection: Upgrade\r\n
Sec-WebSocket-Accept: anTEIFyI/gTepr8Q3okBj81M2/4=\r\n
\r\n
[HTTP response 1/1]
[Time since request: 0.000245010 seconds]
[Request in frame: 4]
Can someone tell me what is wrong with the response? Is my accept value incorrect?
EDIT 1:
The code I use to create the response value. The websocket_key is grabbed from the request prior to this.
const char *magic_string = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11";
int pre_hash_size = 36 + websocket_key.size();
char pre_hash[pre_hash_size];
memcpy(pre_hash, websocket_key.c_str(), websocket_key.size());
memcpy(pre_hash + websocket_key.size(), magic_string, 36);
unique_ptr<Botan::HashFunction> hash1(Botan::HashFunction::create("SHA-1"));
Botan::secure_vector<uint8_t> post_hash = hash1->process(reinterpret_cast<const uint8_t *>(pre_hash), pre_hash_size);
string accept_response = base64_encode(post_hash.data(), post_hash.size());
Here is the base 64 function:
/*
base64.cpp and base64.h
base64 encoding and decoding with C++.
Version: 1.01.00
Copyright (C) 2004-2017 René Nyffenegger
This source code is provided 'as-is', without any express or implied
warranty. In no event will the author be held liable for any damages
arising from the use of this software.
Permission is granted to anyone to use this software for any purpose,
including commercial applications, and to alter it and redistribute it
freely, subject to the following restrictions:
1. The origin of this source code must not be misrepresented; you must not
claim that you wrote the original source code. If you use this source code
in a product, an acknowledgment in the product documentation would be
appreciated but is not required.
2. Altered source versions must be plainly marked as such, and must not be
misrepresented as being the original source code.
3. This notice may not be removed or altered from any source distribution.
René Nyffenegger rene.nyffenegger#adp-gmbh.ch
*/
static const std::string base64_chars =
"ABCDEFGHIJKLMNOPQRSTUVWXYZ"
"abcdefghijklmnopqrstuvwxyz"
"0123456789+/";
std::string base64_encode(unsigned char const *bytes_to_encode, unsigned int in_len)
{
std::string ret;
int i = 0;
int j = 0;
unsigned char char_array_3[3];
unsigned char char_array_4[4];
while (in_len--)
{
char_array_3[i++] = *(bytes_to_encode++);
if (i == 3)
{
char_array_4[0] = (char_array_3[0] & 0xfc) >> 2;
char_array_4[1] = ((char_array_3[0] & 0x03) << 4) + ((char_array_3[1] & 0xf0) >> 4);
char_array_4[2] = ((char_array_3[1] & 0x0f) << 2) + ((char_array_3[2] & 0xc0) >> 6);
char_array_4[3] = char_array_3[2] & 0x3f;
for (i = 0; (i < 4); i++)
ret += base64_chars[char_array_4[i]];
i = 0;
}
}
if (i)
{
for (j = i; j < 3; j++)
char_array_3[j] = '\0';
char_array_4[0] = (char_array_3[0] & 0xfc) >> 2;
char_array_4[1] = ((char_array_3[0] & 0x03) << 4) + ((char_array_3[1] & 0xf0) >> 4);
char_array_4[2] = ((char_array_3[1] & 0x0f) << 2) + ((char_array_3[2] & 0xc0) >> 6);
for (j = 0; (j < i + 1); j++)
ret += base64_chars[char_array_4[j]];
while ((i++ < 3))
ret += '=';
}
return ret;
}

The problem was that when I concatenated the pre_hash string from the websocket key (sent by client) and the magic string (constant), I didn't account for the null terminator that the size() function includes in it's count. an extra space I had inadvertently added when parsing the request header.
Remember kiddies, C++ strings are null terminated and size() reflects that.

Related

retrieving string from boost::tuples::tuple

After reading the article at https://www.boost.org/doc/libs/1_67_0/libs/tuple/doc/html/tuple_users_guide.html
The following note is a problem for me.
Note that extracting tuples with std::string or C-style string elements does not generally work, since the streamed tuple representation may not be unambiguously parseable.
What type should I use to unambiguously parse strings from a stream into a tuple?
When retrieving a std::string from a tuple the string is devided by white space. Which is not desired! Setting a delimiter e.g., a number sign ( # ) does not help.
// typedef tuple
typedef std::string td_current_gmt, td_remote_endpoint,
td_request, td_response, td_elapsed_time;
typedef boost::tuples::tuple<td_current_gmt, td_remote_endpoint,
td_request, td_response, td_elapsed_time> tuple_logging;
// store in tuple
tuple_logging tl{ current_gmt, remote_endpoint,
request, response, elapsed_time };
// write tuple to file
tl = boost::tuples::make_tuple(current_gmt, remote_endpoint,
request, response, elapsed_time);
boost::filesystem::path p = { "logging" };
boost::filesystem::ofstream ofs{ p };
ofs << /*boost::tuples::set_delimiter('#') <<*/ tl;
ofs.close();
// read tuple from file
tuple_logging tlin{ current_gmt, remote_endpoint,
request, response, elapsed_time };
boost::filesystem::ifstream ifs{ p };
//ifs >> boost::tuples::set_delimiter('#');
ifs >> tlin;
The output is (Fri, 16 Aug 2019 06:28:05)
But actually it has to be (Fri, 16 Aug 2019 06:28:05 GMT 192.168.178.14:52832 TRACE / HTTP/1.1 HTTP/1.1 200 OK 8.936800)
Here's the code.
void parse_logfile(
boost::filesystem::ifstream& ifs,
const boost::filesystem::path& p,
std::vector<tuple_logging>& vector_with_tuples
)
{
typedef std::string one_line_from_logging;
one_line_from_logging str;
tuple_logging tlin;
// clear vector
vector_with_tuples.clear();
// open log file for reading
ifs.open(p);
// read one line from log file, until eof
while (std::getline(ifs, str))
{
size_t sBegin = 0, sEnd = 0;
// 1
// first character on a line is '(',
// start at sBegin = 1
sBegin = sEnd + 1;
sEnd = str.find('#', sBegin);
std::string current_gmt_ = str.substr(sBegin, sEnd - sBegin);
// 2
sBegin = sEnd + 1;
sEnd = str.find('#', sBegin);
std::string remote_endpoint_ = str.substr(sBegin, sEnd - sBegin);
// 3
sBegin = sEnd + 1;
sEnd = str.find('#', sBegin);
std::string request_ = str.substr(sBegin, sEnd - sBegin);
// 4
sBegin = sEnd + 1;
sEnd = str.find('#', sBegin);
std::string response_ = str.substr(sBegin, sEnd - sBegin);
// 5
sBegin = sEnd + 1;
// last character on a line is ')'
sEnd = str.find(')', sBegin);
std::string elapsed_time_ = str.substr(sBegin, sEnd - sBegin);
// create tuple from parsed log data out of file
tlin = boost::tuples::make_tuple(
current_gmt_,
remote_endpoint_,
request_,
response_,
elapsed_time_
);
// set tuple into vector
vector_with_tuples.push_back(tlin);
}
// close log file
ifs.close();
}

Is there something special about Sending to Outlook 2013 with MAPI that I'm missing?

I am working on a Simple MAPI interface with Visual Studio 2008 to send email and attachments from an application. Works great with Thunderbird and Outlook 6 but Outlook 2013 is giving me all sorts of grief.
There are two key issues:
1) The email goes to Outlook's outbox but when it sends it bounces back (or appears to as I think this is internal) with the message "None of your e-mail accounts could send to this recipient."
If I compose a new message in Outlook with that exact email address it works fine. I've tried two different outbound SMTP accounts that I know both work, and work manually, but with IMAP they are stuck. The IMAP code I used didn't have originator data and I've hacked that in to try to make it work (the code I'm posting is a bit raw because I'm still trying to figure this out)
2) Outlook will display "A program is trying to send an e-mail message on your behalf".
I've set the the Outlook Trust Centre access to "Never warn about activity" as described in loads of online help but the problem persists. I can't help but think these might be related?
I'm wondering if Outlook 2013 is needing more data than I'm providing or if I'm making some cockeyed assumption.
Note: these test are on three different machines - Thunderbird on my main Win 10 development machine, Outlook 6 on a XP virtual box, and Outlook 2013 is on another Windows 10 machine.
Note on the code: I'm using CPtrArrays to store the data passed by the calling function. You'll see the GetAt() in setting up the recipients.
Thanks!
MapiRecipDesc sender[1];
MapiRecipDesc recipient[50];
MapiFileDesc fileDesc[20];
sender[0].ulRecipClass = MAPI_ORIG;
sender[0].lpszAddress = "me#me.net";
sender[0].lpszName = "Me";
sender[0].lpEntryID = 0;
sender[0].ulEIDSize = 0;
sender[0].ulReserved = 0;
iToCount = 0;
iIndex = 0;
while (iIndex < m_paTo.GetCount()) {
recipient[iToCount].ulRecipClass = MAPI_TO;
recipient[iToCount].lpszAddress = (char *) m_paTo.GetAt(iToCount);
recipient[iToCount].lpszName = (char *) m_paTo.GetAt(iToCount);
recipient[iToCount].lpEntryID = 0;
recipient[iToCount].ulEIDSize = 0;
recipient[iToCount].ulReserved = 0;
iIndex++;
iToCount++;
}
iIndex = 0;
while (iIndex < m_paCC.GetCount()) {
recipient[iToCount].ulRecipClass = MAPI_CC;
recipient[iToCount].lpszAddress = (char *) m_paCC.GetAt(iIndex);
recipient[iToCount].lpszName = (char *) m_paCC.GetAt(iIndex);
recipient[iToCount].lpEntryID = 0;
recipient[iToCount].ulEIDSize = 0;
recipient[iToCount].ulReserved = 0;
iIndex++;
iToCount++;
}
iIndex = 0;
while (iIndex < m_paBCC.GetCount()) {
recipient[iToCount].ulRecipClass = MAPI_BCC;
recipient[iToCount].lpszAddress = (char *) m_paBCC.GetAt(iIndex);
recipient[iToCount].lpszName = (char *) m_paBCC.GetAt(iIndex);
recipient[iToCount].lpEntryID = 0;
recipient[iToCount].ulEIDSize = 0;
recipient[iToCount].ulReserved = 0;
iIndex++;
iToCount++;
}
iFileCount = 0;
iIndex = 0;
while (iIndex < m_paAttachments.GetCount()) {
fileDesc[iFileCount].flFlags = 0;
fileDesc[iFileCount].lpFileType = 0;
fileDesc[iFileCount].lpszFileName = (char *) m_paAttachments.GetAt(iIndex);
fileDesc[iFileCount].lpszPathName = (char *) m_paAttachments.GetAt(iIndex);
fileDesc[iFileCount].nPosition = -1;
fileDesc[iFileCount].ulReserved = 0;
iIndex++;
iFileCount++;
}
TCHAR szSubject[_MAX_PATH];
TCHAR szMessage[5001];
::StrCpy(szSubject, m_sSubject);
::StrCpy(szMessage, m_sMessage);
MapiMessage message;
::ZeroMemory(&message, sizeof(message));
message.lpszSubject = szSubject;
message.nRecipCount = iToCount;
message.lpRecips = recipient;
message.nFileCount = iFileCount;
message.lpFiles = fileDesc;
message.lpszNoteText = szMessage;
message.flFlags = MAPI_SENT | MAPI_UNREAD;
message.lpszConversationID = "123";
message.lpOriginator = sender;
//int nError = SendMail(0, (ULONG_PTR)hWndParent, &message, MAPI_LOGON_UI|MAPI_DIALOG, 0);
int nError = SendMail(0, (ULONG_PTR)hWndParent, &message, MAPI_LOGON_UI, 0);
if (nError != SUCCESS_SUCCESS &&
nError != MAPI_USER_ABORT &&
nError != MAPI_E_LOGIN_FAILURE) {
CString sMessage;
CString sTest = recipient[0].lpszAddress;
sMessage.Format("MapiMail:: SendMail Error code %d Recip count %d: first Recip: %s", nError, message.nRecipCount, sTest);
AfxMessageBox(sMessage);
lLog.WriteString(sMessage);
return false;
}
With Barmak Shemirani's advice I'm posting my own answer. If anyone else is looking for this info, maybe I can save them some time by posting it here in one place.
The "None of your e-mail accounts could send to this recipient." problem can be solved by putting the email address of the recipient in pointy brackets in the name field in the recipient structure. Leave the address field blank. So, for example
::StrCpy(szTo, "<address#email.com>");
recipient[iToCount].ulRecipClass = MAPI_TO;
recipient[iToCount].lpszAddress = 0;
recipient[iToCount].lpszName = szTo;
recipient[iToCount].lpEntryID = 0;
recipient[iToCount].ulEIDSize = 0;
recipient[iToCount].ulReserved = 0;
I've tested this with Outlook 2013 and 2016 and with Outlook 6, Thunderbird, and EM Client and they're all happy with it. Apparently you can put the name as well PersonName <name#email.com>, but I haven't tested that.
The issue of Outlook displaying:
A program is trying to send an e-mail message on your behalf
is a software configuration issue. Most websites advise using the Trust Centre to set the Programmatic Access to allow other applications access via MAPI. You have to set this running the program as administrator but you also have to run Outlook as administrator for it to work
The workaround is to edit the registry and add a DWORD PromptSimpleMAPISend with a value of 2 in:
Computer\HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\x.0\Outlook\Security
Bonus answer. If anyone is wondering about HTML in MAPI, it isn't supported. MAPI apparently pre-dates the common usage of HTML email.
There is a workaround. You can leave your message body blank and instead put your message in html in a file and put that file as the first attachment in the MAPI message. I named my file with an .html extension.
It is a bit of fluke - the email clients pick it up and display the HTML in the body of the email. Your html file will still be an attachment followed by any other attachments.
I've tested with Thunderbird, Outlook, and EM Client. I took a quick look at one web email reader and it did not display the html text (although the attachment was available to read).

Convert connect address with address familiy AF_SYSTEM to human readable string

Background
I'm writing some dtrace program which tracks application socket file descriptors. Aim is to provide logs which help me spot leak of file descriptors in some very complex OS X application.
Here is my other question with very helpful answer.
Problem
I want that my program is logging address to which file descriptor has been connected to. In examples there is a code which partial do what I need: soconnect_mac.d, here is link to github.
soconnect_mac.d works great when applied on Firefox, but it completely fails in case of my application. Quick investigation shown that soconnect_mac.d is able to interpret only AF_INET (value 2) family address and som library used by my application is using AF_SYSTEM (value 32) family address.
I can't find anything which could help me convert received address to something what is human readable.
So far I've got this:
#!/usr/sbin/dtrace -s
inline int af_inet = 2 ; /* AF_INET defined in Kernel/sys/socket.h */
inline int af_inet6 = 30; /* AF_INET6 defined in Kernel/sys/socket.h */
inline int af_system = 32; /* AF_SYSTEM defined in Kernel/sys/socket.h */
… // some stuff
syscall::connect:entry
/pid == $target && isOpened[pid, arg0] == 1/
{
/* assume this is sockaddr_in until we can examine family */
this->s = (struct sockaddr_in *)copyin(arg1, arg2);
this->f = this->s->sin_family;
self->fileDescriptor = arg0;
}
/* this section is copied with pride from "soconnect_mac.d" */
syscall::connect:entry
/this->f == af_inet/
{
/* Convert port to host byte order without ntohs() being available. */
self->port = (this->s->sin_port & 0xFF00) >> 8;
self->port |= (this->s->sin_port & 0xFF) << 8;
/*
* Convert an IPv4 address into a dotted quad decimal string.
* Until the inet_ntoa() functions are available from DTrace, this is
* converted using the existing strjoin() and lltostr(). It's done in
* two parts to avoid exhausting DTrace registers in one line of code.
*/
this->a = (uint8_t *)&this->s->sin_addr;
this->addr1 = strjoin(lltostr(this->a[0] + 0ULL),
strjoin(".",
strjoin(lltostr(this->a[1] + 0ULL),
".")));
this->addr2 = strjoin(lltostr(this->a[2] + 0ULL),
strjoin(".",
lltostr(this->a[3] + 0ULL)));
self->address = strjoin(this->addr1, this->addr2);
}
/* this section is my */
syscall::connect:entry
/this->f == af_system/
{
/* TODO: Problem how to handle AF_SYSTEM address family */
/* Convert port to host byte order without ntohs() being available. */
self->port = (this->s->sin_port & 0xFF00) >> 8;
self->port |= (this->s->sin_port & 0xFF) << 8; // this also doen't work as it should
self->address = "system family address needed here";
}
// a fallback
syscall::connect:entry
/this->f && this->f != af_inet && this->f != af_system/
{
/* Convert port to host byte order without ntohs() being available. */
self->port = (this->s->sin_port & 0xFF00) >> 8;
self->port |= (this->s->sin_port & 0xFF) << 8;
self->address = strjoin("Can't handle family: ", lltostr(this->f));
}
syscall::connect:return
/self->fileDescriptor/
{
this->errstr = err[errno] != NULL ? err[errno] : lltostr(errno);
printf("%Y.%03d FD:%d Status:%s Address:%s Port:%d",
walltimestamp, walltimestamp % 1000000000 / 1000000,
self->fileDescriptor, this->errstr, self->address, self->port);
self->fileDescriptor = 0;
self->address = 0;
self->port = 0;
}
What is even more annoying my code has failed to read port number (I get 512 value instead one of this: 443, 8443, 5061).
IMO problem is first syscall::connect:entry where it is assumed that second argument can be treated as struct sockaddr_in. I'm guessing struct sockaddr_storage should be used in case of AF_SYSTEM address family, but I didn't found any documentation or source code which proves this in direct way.
My section with this->f == af_system condition properly catches events from application I'm investigating.

MQL4: How to read a CSV from a URL

I'm using this URL to fetch some content from a Quandl website:
https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csv?exclude_column_names=true&rows=1&api_key=my_api_key
The Quandl server returns in response to the above request a value of this:
2016-08-01, 144598.0
I need to use the value of 144598.0 within an MQL4 Script, so:
Q1. How do I fetch the content from the URL above to be used within an MQL4 Script?
A very helpful user from SO (https://stackoverflow.com/users/3666197/user3666197) provided the following script (original found at MQL4: Read single value from CSV) (a couple parts added in by myself) to help me achieve this, however, I couldn't get it to work:
// Code created with the help of Stack Overflow question
// https://stackoverflow.com/questions/39279634/mql4-read-single-value-from-csv/39284875#39284875
// Question by p.luck:
// https://stackoverflow.com/users/5551849/p-luck
// Answer by user3666197:
// https://stackoverflow.com/users/3666197/user3666197
void OnStart()
{
string cookie = NULL,
headers;
char post[],
result[];
int res;
/* TODO: *
* Must allow MT4 to access the server URL, *
* you should add URL "https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csv" *
* in the list of allowed URLs *
* ( MT4 -> Tools -> Options -> [Tab]: "Expert Advisors" ): */
string aDataSOURCE_URL = "https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csv";
string aDataSOURCE_API = "?exclude_column_names=true&rows=1&api_key=my_api_key";
//-- Create the body of the POST request for API specifications and API-authorization
ArrayResize( post,
StringToCharArray( aDataSOURCE_API, // string text |--> [in] String to copy.
post, // uchar &array[] <--| [out] Array of uchar type.
0, // int start = 0 |--> [in] Position from which copying starts. Default - 0.
WHOLE_ARRAY, // int count = -1 |--> [in] Number of array elements to copy. Defines length of a resulting string. Default value is -1, which means copying up to the array end, or till terminating '\0'. Terminating zero will also be copied to the recipient array, in this case the size of a dynamic array can be increased if necessary to the size of the string. If the size of the dynamic array exceeds the length of the string, the size of the array will not be reduced.
CP_UTF8 // uint cp = CP_ACP |--> [in] The value of the code page. For the most-used code pages provide appropriate constants.
)
- 1
);
//-- Reset the last error code
ResetLastError();
//-- Loading a html page from Quandl
int timeout = 5000; //-- Timeout below 1000 (1 sec.) is not enough for slow Internet connection
res = WebRequest( "POST", // const string method |--> [in] HTTP method.
aDataSOURCE_URL, // const string URL |--> [in] URL.
cookie, // const string cookie |--> [in] Cookie value.
NULL, // const string referrer |--> [in] Value of the Referer header of the HTTP request.
timeout, // int timeout |--> [in] Timeout in milliseconds.
post, // const char &data |--> [in] Data array of the HTTP message body
ArraySize( post ), // int data_size |--> [in] Size of the data[] array.
result, // char &result <--| [out] An array containing server response data.
headers // string &result_headers <--| [out] Server response headers.
);
//-- Check errors
if ( res == -1 )
{ Print( "WebRequest Error. Error code = ", GetLastError() ); //-- Perhaps the URL is not listed, display a message about the necessity to add the address
MessageBox( "Add the address '" + aDataSOURCE_URL + "' in the list of allowed URLs on tab 'Expert Advisors'", "Error", MB_ICONINFORMATION );
}
else //-- Load was successfull
{
PrintFormat( "The data has been successfully loaded, size = %d bytes.", ArraySize( result ) );
//-- parse the content ---------------------------------------
/*
"2016-08-01, 144598.0"
*/
//-- consume the content -------------------------------------
//...
}
}
I have added the URL of https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csvto the list of allowed URLs in MT4.
If I enable AutoTrading and drag the script, named (tutorial7), on to a chart of USDCAD,M1,I get these messages within the Experts tab:
Script tutorial7 USDCAD,M1: loaded successfuly
tutorial7 USDCAD,M1: initialized
tutorial7 USDCAD,M1: The data has been successfully loaded, size = 0 bytes
tutorial7 USDCAD,M1: uninit reason 0
Surely if the "The data has successfully loaded" it shouldn't say "size = 0 bytes"?
Should this script work correctly if I just copy and paste it straight in to the MetaQuotes Language Editor and compile it?
Apart from adding the URL to the allowed URLs in MT4 and copying and pasting this code in to a script, is there anything else I must do?
If so, how?
I am running the above code as a Script not an Expert Advisor; is this okay?
Q2. Can I set the fetched value of 144598.0 as a variable within my script?
I need to make the value of 144598.0 a variable so that it can be compared to another value.
Would something like this work:
void OnStart()
{
... // above code which fetches the value from the .csv URL
double x = 155876.0 // value manually typed in
y = 144598.0 // value fetched from the .csv URL using the above code
// ignores column 1 consisting of 2016-08-01
if ( x > y ) {
// execute code
}
else {
// execute other code
}
}
A2: YES! This is the easiest part of the journey.A1: BUT!How does it work in practice? How does MetaTrader Terminal actually speak to Quandl, so to get it?
Let me first illustrate the issue of remote, HttpServer-side processing.
There is an easy to prototype program curl ( Linux & DOS versions available ) that will show right inside a terminal window ( or a Windows cmd window ) how the remote HttpServer at Quandl responds to various compositions of the locally assembled requests, communicated over the HTTP-protocol.
Notice, that the just retyped URL produces the whole history to be delivered.
C:\>curl https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csv
DATE,VALUE
2016-08-01,144598.0
2016-07-01,144447.0
...
..
.
1939-03-01,30280.0
1939-02-01,30101.0
1939-01-01,29923.0
Once we add further parameters to the plain URL, the remote-side ( the HttpServer ) changes the reply to just the one row we are interested in:
C:\>curl -G --data rows=1 --data exclude_column_names=true https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csv
2016-08-01,144598.0
Cool, looks great, almost the single value we want!
But here the magic comes.
The real interchange ( dialogue ) between the local process ( curl now, but MetaTrader Terminal later ) looks this way ( using a --verbose option in curl commandline ):
C:\>curl --verbose -G --data rows=1 --data exclude_column_names=true https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csv
* Trying 54.174.87.84...
* Connected to www.quandl.com (54.174.87.84) port 443 (#0)
* schannel: SSL/TLS connection with www.quandl.com port 443 (step 1/3)
* schannel: checking server certificate revocation
* schannel: sending initial handshake data: sending 70 bytes...
* schannel: sent initial handshake data: sent 70 bytes
* schannel: SSL/TLS connection with www.quandl.com port 443 (step 2/3)
* schannel: failed to receive handshake, need more data
* schannel: SSL/TLS connection with www.quandl.com port 443 (step 2/3)
* schannel: encrypted data buffer: offset 4096 length 4096
* schannel: encrypted data length: 4017
* schannel: encrypted data buffer: offset 4017 length 4096
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with www.quandl.com port 443 (step 2/3)
* schannel: encrypted data buffer: offset 4569 length 5041
* schannel: sending next handshake data: sending 318 bytes...
* schannel: SSL/TLS connection with www.quandl.com port 443 (step 2/3)
* schannel: encrypted data buffer: offset 51 length 5041
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with www.quandl.com port 443 (step 3/3)
* schannel: incremented credential handle refcount = 1
* schannel: stored credential handle in session cache
> GET /api/v3/datasets/FRED/PAYEMS.csv?rows=1&exclude_column_names=true HTTP/1.1
> Host: www.quandl.com
> User-Agent: curl/7.45.0
> Accept: */*
>
* schannel: client wants to read 16384 bytes
* schannel: encdata_buffer resized 17408
* schannel: encrypted data buffer: offset 0 length 17408
* schannel: encrypted data got 653
* schannel: encrypted data buffer: offset 653 length 17408
* schannel: decrypted data length: 623
* schannel: decrypted data added: 623
* schannel: decrypted data cached: offset 623 length 16384
* schannel: encrypted data buffer: offset 0 length 17408
* schannel: decrypted data buffer: offset 623 length 16384
* schannel: schannel_recv cleanup
* schannel: decrypted data returned 623
* schannel: decrypted data buffer: offset 0 length 16384
< HTTP/1.1 200 OK
< Cache-Control: private
< Content-Disposition: attachment; filename=FRED-PAYEMS.csv
< Content-Transfer-Encoding: binary
< Content-Type: text/csv
< Date: Wed, 07 Sep 2016 12:47:20 GMT
< ETag: W/"adfdb97850c493cdd03e2036574bc404"
< Server: openresty
< Vary: Origin
< X-API-Version: 2015-04-09
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Rack-CORS: preflight-hit; no-origin
< X-RateLimit-Limit: 50
< X-RateLimit-Remaining: 42
< X-Request-Id: c609e92d-22d2-40e7-b7d4-cacb07467c76
< X-Runtime: 0.023534
< X-XSS-Protection: 1; mode=block
< Content-Length: 20
< Connection: keep-alive
<
2016-08-01,144598.0
* Connection #0 to host www.quandl.com left intact
Notice the row GET /api/v3/datasets/FRED/PAYEMS.csv?rows=1&exclude_column_names=true
So the magic is to make MetaTrader Terminal to assemble the same, together with allowing the URL in the permitted list in the configuration ( that you have done already in the other post ).
Also might have noticed, that the HTTP GET sends just the <HttpServer_relative_part_of_the.URL>
The magic is in making MQL4 code send the same request as was seen above and get the data back.
WebRequest() has to use HTTP GET as the Quandl HttpServer does not respond to a HTTP POST version of the same request example, returning 0 bytes ( just omit the -G switch from the curl examples above ).
Meeting all the conditions at once should result in receiving 2016-08-01,144598.0 and using:
int anAmountOfBytesRECVd = 0; // how many bytes Quandl sent
string headers_FromHttpSERVER; // stores as a string
char anAnswerFromHttpSERVER[]; // stores as a byte-array ( char[] )
double y_value = NULL;
anAmountOfBytesRECVd = WebRequest( "GET", // MUST use HTTP GET <- Quandl tested
...
anAnswerFromHttpSERVER,
headers_FromHttpSERVER
);
y_value = StrToDouble( CharArrayToString( anAnserFromHttpSERVER, // [2|0|1|6|-|0|8|-|0|1|,|1|4|4|5|98|.|0]
11, //-----------------------^_______________( -1 == TILL EndOfSTRING )
-1
)
);

My Windows Socket Program Cannot Open Some Urls That Browsers Such As FireFox, IE etc Can

I have made the following win32 socket program to browse web pages. I am using wingw to avoid dependency on any runtime. To get ipaddresses I ping urls such as www.google.com, www.yahoo.com through command prompt and use those ip addreses in my program. Port is ofcourse 80.
I am able to get default pages of google, yahoo etc by using "GET /\r\n". I am also able to get non-default pages, even those inside directories, such as http://yasini.com/newsite/index.aspx by using "GET /newsite/index.aspx". The output of the program is in the form of html received from webserver, saved on hard disk. This file is later opened in firefox to see how did the communication go.
I have made a test webpage, http://a.domaindlx.com/trysite/hello.asp, which I can open in firefox. Then I ping the domain, a.domaindlx.com and get this ipaddress, 66.36.238.30. I try to access the said page by using "GET /trysite/hello.asp" but get this in response, "No web site is configured at this address. No web site is configured at this address."
I know that the said response is sent by the webserver, so I was able to connect to the webserver. The problem is that the webserver is not recognizing the url I am trying to access. I have used different webpages, both htm and asp and none is accessible.
When trying to open website using ipaddress directly in browser, I get the same error, "No website is configured...".
The basic puzzle is, why are these pages accessible through a browser such as firefox, but not through my code, when my code is essentially a browser, mean open connection with webserver at port 80.
#include windows.h
#include stdio.h
WSADATA ws;
int d;
char aa[1000];
struct sockaddr_in a;
SOCKET s;
int li;
void abc(char *p)
{
FILE *fp = fopen("c:\\data.htm", "a+");
fprintf(fp, "%s\n", p);
fclose(fp);
}
_stdcall WinMain (HINSTANCE i, HINSTANCE j, char * k, int l)
{
d = WSAStartup(0x101, &ws);
sprintf(aa, "WSASTARTUP = %d", d);
abc(aa);
s = socket(AF_INET, SOCK_STREAM, 0);
sprintf(aa, "SOCKET = %d", s);
abc(aa);
a.sin_family = AF_INET;
a.sin_port = htons(80);
//a.sin_addr.s_addr = inet_addr("74.125.236.145");
a.sin_addr.s_addr = inet_addr("66.36.238.30"); //a.domaindlx.com
//a.sin_addr.s_addr = inet_addr("206.225.85.18"); //www.domaindlx.com
//a.sin_addr.s_addr = inet_addr("87.248.122.122"); //www.yahoo.com
//a.sin_addr.s_addr = inet_addr("72.167.153.9"); //www.yasini.com
d = connect(s, (struct sockaddr *) &a, sizeof(a));
strcpy(aa, "GET /trysite/hello.asp\r\n");
strcat(aa, "HTTP 1.0 \r\n\r\n");
send(s, aa, sizeof(aa), 0);
li = 1;
while(li != 0)
{
li = recv(s, aa, 1000, 0);
abc(aa);
}
}
Note: Please enclose the header file names in the include line in angle brackets for the code to work. I had to remove that to property format the html.
The troublesome URL is running on a subdomain. The successful URLs are not. Many webservers host multiple accounts on the same physical IP(s), so they need to know which particular domain/subdomain is being requested in order to access the correct account. You need to include a Host header in your request.
Also note that when you call send() to send the request, you are sending the entire 1000 bytes of the aa buffer, which is wrong. You need to send only what you actualy filled in.
Lastly, you are not really managing the socket very well in general. You need better error handling.
Try this:
#include <windows.h>
#include <stdio.h>
void abc(char *p, int l = -1)
{
FILE *fp = fopen("c:\\data.htm", "a+");
if (fp)
{
if (l == -1) l = strlen(p);
fwrite(p, 1, l, fp);
fclose(fp);
}
}
int WINAPI WinMain (HINSTANCE i, HINSTANCE j, char * k, int l)
{
char aa[1000];
WSADATA ws;
int d = WSAStartup(0x101, &ws);
sprintf(aa, "WSASTARTUP = %d\n", d);
abc(aa);
if (d == 0)
{
SOCKET s = socket(AF_INET, SOCK_STREAM, 0);
sprintf(aa, "SOCKET = %d\n", s);
abc(aa);
if (s != INVALID_SOCKET)
{
char *host = "a.domaindlx.com";
char *file = "/trysite/hello.asp";
struct sockaddr_in a;
memset(&a, 0, sizeof(a));
a.sin_family = AF_INET;
a.sin_port = htons(80);
struct hostent *h = gethostbyname(host);
if (!h)
{
sprintf(aa, "gethostbyname(\"%s\") FAILED\n", host);
abc(aa);
}
else
{
sprintf(aa, "gethostbyname(\"%s\") TYPE = %d\n", host, h->h_addrtype);
abc(aa);
if (h->h_addrtype == AF_INET)
{
a.sin_addr = * (struct in_addr*) h->h_addr;
sprintf(aa, "gethostbyname(\"%s\") IP = %s\n", host, inet_ntoa(a.sin_addr));
abc(aa);
d = connect(s, (struct sockaddr *) &a, sizeof(a));
sprintf(aa, "CONNECT = %d\n", d);
abc(aa);
if (d == 0)
{
sprintf(aa,
"GET %s HTTP/1.0\r\n"
"Host: %s\r\n"
"Connection: close\r\n"
"\r\n",
file, host);
char *p = aa;
int t = strlen(aa);
int li;
do
{
li = send(s, p, t, 0);
if (li < 1)
break;
p += li;
t -= li;
}
while (t > 0);
if (t != 0)
{
abc("SEND FAILED\n");
}
else
{
abc("SEND OK\n");
do
{
li = recv(s, aa, sizeof(aa), 0);
if (li < 1)
break;
abc(aa, li);
}
while (true);
}
}
}
}
closesocket(s);
}
WSACleanup();
}
return 0;
}
I strongly suggest you get a packet sniffer, such as Wireshark. Then you can see EXACTLY what webbrowsers (or any other socket app) is actually sending and receiving. Then you can match that in your code as needed.
There are two problems with your code. The first one is that there should be a space not \r\n before HTTP 1.0. Without this your are sending HTTP 0.9.
The second problem is that some IP addresses are used to host multiple sites and require sending a Host header.
The site that tells you "No web site is configured at this address" may work better if you add the Host: header. Your request to that site should look like this:
"GET /trysite/hello.asp HTTP 1.0\r\nHost: a.domaindlx.com\r\n\r\n"
You're not following the protocol correctly. You want GET /trysite/hello.asp HTTP/1.0\r\n\r\n See here for the full spec.

Resources