MQL4: How to read a CSV from a URL - algorithmic-trading

Question

I'm using this URL to fetch some content from a Quandl website:
https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csv?exclude_column_names=true&rows=1&api_key=my_api_key
The Quandl server returns in response to the above request a value of this:
2016-08-01, 144598.0
I need to use the value of 144598.0 within an MQL4 Script, so:
Q1. How do I fetch the content from the URL above to be used within an MQL4 Script?
A very helpful user from SO (https://stackoverflow.com/users/3666197/user3666197) provided the following script (original found at MQL4: Read single value from CSV) (a couple parts added in by myself) to help me achieve this, however, I couldn't get it to work:
// Code created with the help of Stack Overflow question
// https://stackoverflow.com/questions/39279634/mql4-read-single-value-from-csv/39284875#39284875
// Question by p.luck:
// https://stackoverflow.com/users/5551849/p-luck
// Answer by user3666197:
// https://stackoverflow.com/users/3666197/user3666197
void OnStart()
{
string cookie = NULL,
headers;
char post[],
result[];
int res;
/* TODO: *
* Must allow MT4 to access the server URL, *
* you should add URL "https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csv" *
* in the list of allowed URLs *
* ( MT4 -> Tools -> Options -> [Tab]: "Expert Advisors" ): */
string aDataSOURCE_URL = "https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csv";
string aDataSOURCE_API = "?exclude_column_names=true&rows=1&api_key=my_api_key";
//-- Create the body of the POST request for API specifications and API-authorization
ArrayResize( post,
StringToCharArray( aDataSOURCE_API, // string text |--> [in] String to copy.
post, // uchar &array[] <--| [out] Array of uchar type.
0, // int start = 0 |--> [in] Position from which copying starts. Default - 0.
WHOLE_ARRAY, // int count = -1 |--> [in] Number of array elements to copy. Defines length of a resulting string. Default value is -1, which means copying up to the array end, or till terminating '\0'. Terminating zero will also be copied to the recipient array, in this case the size of a dynamic array can be increased if necessary to the size of the string. If the size of the dynamic array exceeds the length of the string, the size of the array will not be reduced.
CP_UTF8 // uint cp = CP_ACP |--> [in] The value of the code page. For the most-used code pages provide appropriate constants.
)
- 1
);
//-- Reset the last error code
ResetLastError();
//-- Loading a html page from Quandl
int timeout = 5000; //-- Timeout below 1000 (1 sec.) is not enough for slow Internet connection
res = WebRequest( "POST", // const string method |--> [in] HTTP method.
aDataSOURCE_URL, // const string URL |--> [in] URL.
cookie, // const string cookie |--> [in] Cookie value.
NULL, // const string referrer |--> [in] Value of the Referer header of the HTTP request.
timeout, // int timeout |--> [in] Timeout in milliseconds.
post, // const char &data |--> [in] Data array of the HTTP message body
ArraySize( post ), // int data_size |--> [in] Size of the data[] array.
result, // char &result <--| [out] An array containing server response data.
headers // string &result_headers <--| [out] Server response headers.
);
//-- Check errors
if ( res == -1 )
{ Print( "WebRequest Error. Error code = ", GetLastError() ); //-- Perhaps the URL is not listed, display a message about the necessity to add the address
MessageBox( "Add the address '" + aDataSOURCE_URL + "' in the list of allowed URLs on tab 'Expert Advisors'", "Error", MB_ICONINFORMATION );
}
else //-- Load was successfull
{
PrintFormat( "The data has been successfully loaded, size = %d bytes.", ArraySize( result ) );
//-- parse the content ---------------------------------------
/*
"2016-08-01, 144598.0"
*/
//-- consume the content -------------------------------------
//...
}
}
I have added the URL of https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csvto the list of allowed URLs in MT4.
If I enable AutoTrading and drag the script, named (tutorial7), on to a chart of USDCAD,M1,I get these messages within the Experts tab:
Script tutorial7 USDCAD,M1: loaded successfuly
tutorial7 USDCAD,M1: initialized
tutorial7 USDCAD,M1: The data has been successfully loaded, size = 0 bytes
tutorial7 USDCAD,M1: uninit reason 0
Surely if the "The data has successfully loaded" it shouldn't say "size = 0 bytes"?
Should this script work correctly if I just copy and paste it straight in to the MetaQuotes Language Editor and compile it?
Apart from adding the URL to the allowed URLs in MT4 and copying and pasting this code in to a script, is there anything else I must do?
If so, how?
I am running the above code as a Script not an Expert Advisor; is this okay?
Q2. Can I set the fetched value of 144598.0 as a variable within my script?
I need to make the value of 144598.0 a variable so that it can be compared to another value.
Would something like this work:
void OnStart()
{
... // above code which fetches the value from the .csv URL
double x = 155876.0 // value manually typed in
y = 144598.0 // value fetched from the .csv URL using the above code
// ignores column 1 consisting of 2016-08-01
if ( x > y ) {
// execute code
}
else {
// execute other code
}
}

Answer 1

A2: YES! This is the easiest part of the journey.A1: BUT!How does it work in practice? How does MetaTrader Terminal actually speak to Quandl, so to get it?
Let me first illustrate the issue of remote, HttpServer-side processing.
There is an easy to prototype program curl ( Linux & DOS versions available ) that will show right inside a terminal window ( or a Windows cmd window ) how the remote HttpServer at Quandl responds to various compositions of the locally assembled requests, communicated over the HTTP-protocol.
Notice, that the just retyped URL produces the whole history to be delivered.
C:\>curl https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csv
DATE,VALUE
2016-08-01,144598.0
2016-07-01,144447.0
...
..
.
1939-03-01,30280.0
1939-02-01,30101.0
1939-01-01,29923.0
Once we add further parameters to the plain URL, the remote-side ( the HttpServer ) changes the reply to just the one row we are interested in:
C:\>curl -G --data rows=1 --data exclude_column_names=true https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csv
2016-08-01,144598.0
Cool, looks great, almost the single value we want!
But here the magic comes.
The real interchange ( dialogue ) between the local process ( curl now, but MetaTrader Terminal later ) looks this way ( using a --verbose option in curl commandline ):
C:\>curl --verbose -G --data rows=1 --data exclude_column_names=true https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csv
* Trying 54.174.87.84...
* Connected to www.quandl.com (54.174.87.84) port 443 (#0)
* schannel: SSL/TLS connection with www.quandl.com port 443 (step 1/3)
* schannel: checking server certificate revocation
* schannel: sending initial handshake data: sending 70 bytes...
* schannel: sent initial handshake data: sent 70 bytes
* schannel: SSL/TLS connection with www.quandl.com port 443 (step 2/3)
* schannel: failed to receive handshake, need more data
* schannel: SSL/TLS connection with www.quandl.com port 443 (step 2/3)
* schannel: encrypted data buffer: offset 4096 length 4096
* schannel: encrypted data length: 4017
* schannel: encrypted data buffer: offset 4017 length 4096
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with www.quandl.com port 443 (step 2/3)
* schannel: encrypted data buffer: offset 4569 length 5041
* schannel: sending next handshake data: sending 318 bytes...
* schannel: SSL/TLS connection with www.quandl.com port 443 (step 2/3)
* schannel: encrypted data buffer: offset 51 length 5041
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with www.quandl.com port 443 (step 3/3)
* schannel: incremented credential handle refcount = 1
* schannel: stored credential handle in session cache
> GET /api/v3/datasets/FRED/PAYEMS.csv?rows=1&exclude_column_names=true HTTP/1.1
> Host: www.quandl.com
> User-Agent: curl/7.45.0
> Accept: */*
>
* schannel: client wants to read 16384 bytes
* schannel: encdata_buffer resized 17408
* schannel: encrypted data buffer: offset 0 length 17408
* schannel: encrypted data got 653
* schannel: encrypted data buffer: offset 653 length 17408
* schannel: decrypted data length: 623
* schannel: decrypted data added: 623
* schannel: decrypted data cached: offset 623 length 16384
* schannel: encrypted data buffer: offset 0 length 17408
* schannel: decrypted data buffer: offset 623 length 16384
* schannel: schannel_recv cleanup
* schannel: decrypted data returned 623
* schannel: decrypted data buffer: offset 0 length 16384
< HTTP/1.1 200 OK
< Cache-Control: private
< Content-Disposition: attachment; filename=FRED-PAYEMS.csv
< Content-Transfer-Encoding: binary
< Content-Type: text/csv
< Date: Wed, 07 Sep 2016 12:47:20 GMT
< ETag: W/"adfdb97850c493cdd03e2036574bc404"
< Server: openresty
< Vary: Origin
< X-API-Version: 2015-04-09
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Rack-CORS: preflight-hit; no-origin
< X-RateLimit-Limit: 50
< X-RateLimit-Remaining: 42
< X-Request-Id: c609e92d-22d2-40e7-b7d4-cacb07467c76
< X-Runtime: 0.023534
< X-XSS-Protection: 1; mode=block
< Content-Length: 20
< Connection: keep-alive
<
2016-08-01,144598.0
* Connection #0 to host www.quandl.com left intact
Notice the row GET /api/v3/datasets/FRED/PAYEMS.csv?rows=1&exclude_column_names=true
So the magic is to make MetaTrader Terminal to assemble the same, together with allowing the URL in the permitted list in the configuration ( that you have done already in the other post ).
Also might have noticed, that the HTTP GET sends just the <HttpServer_relative_part_of_the.URL>
The magic is in making MQL4 code send the same request as was seen above and get the data back.
WebRequest() has to use HTTP GET as the Quandl HttpServer does not respond to a HTTP POST version of the same request example, returning 0 bytes ( just omit the -G switch from the curl examples above ).
Meeting all the conditions at once should result in receiving 2016-08-01,144598.0 and using:
int anAmountOfBytesRECVd = 0; // how many bytes Quandl sent
string headers_FromHttpSERVER; // stores as a string
char anAnswerFromHttpSERVER[]; // stores as a byte-array ( char[] )
double y_value = NULL;
anAmountOfBytesRECVd = WebRequest( "GET", // MUST use HTTP GET <- Quandl tested
...
anAnswerFromHttpSERVER,
headers_FromHttpSERVER
);
y_value = StrToDouble( CharArrayToString( anAnserFromHttpSERVER, // [2|0|1|6|-|0|8|-|0|1|,|1|4|4|5|98|.|0]
11, //-----------------------^_______________( -1 == TILL EndOfSTRING )
-1
)
);