FTP client on ethernet shield arduino - ftp

I want to do a project in which the data are saved on the SD card, and then use the ethernet shield these data are sent to an ftp server, using an FTP client arduino. the server i have a free hosting.
Here is the data server from ftptest:
Status: Resolving address of cba.pl
Status: Connecting to 95.211.144.68
Warning: The entered address does not resolve to an IPv6 address.
Status: Connected, waiting for welcome message...
Reply: 220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
Reply: 220-You are user number 57 of 200 allowed.
Reply: 220-Local time is now 16:30. Server port: 21.
Reply: 220-This is a private system - No anonymous login
Reply: 220-IPv6 connections are also welcome on this server.
Reply: 220 You will be disconnected after 5 minutes of inactivity.
Command: CLNT https://ftptest.net on behalf of 2a02:a311:c020:3200:c10d:18e1:36a5:8e2
Reply: 530 You aren't logged in
Command: AUTH TLS
Reply: 234 AUTH TLS OK.
Status: Performing TLS handshake...
Status: TLS handshake successful, verifying certificate...
Status: Received 2 certificates from server.
Status: cert[0]: subject='CN=www.cba.pl' issuer='C=US,O=Let\27s Encrypt,CN=Let\27s Encrypt Authority X3'
Status: cert[1]: subject='C=US,O=Let\27s Encrypt,CN=Let\27s Encrypt Authority X3' issuer='O=Digital Signature Trust Co.,CN=DST Root CA X3'
Command: USER wsalkowski
Reply: 331 User wsalkowski OK. Password required
Command: PASS ********
Reply: 230-Your bandwidth usage is restricted
Reply: 230-OK. Current restricted directory is /
Reply: 230 Max allowed filesize is 10485760 bytes
Command: SYST
Reply: 215 UNIX Type: L8
Command: FEAT
Reply: 211-Extensions supported:
Reply: EPRT
Reply: IDLE
Reply: MDTM
Reply: SIZE
Reply: MFMT
Reply: REST STREAM
Reply: MLST type*;size*;sizd*;modify*;UNIX.mode*;UNIX.uid*;UNIX.gid*;unique*;
Reply: MLSD
Reply: AUTH TLS
Reply: PBSZ
Reply: PROT
Reply: UTF8
Reply: TVFS
Reply: ESTA
Reply: PASV
Reply: EPSV
Reply: SPSV
Reply: ESTP
Reply: 211 End.
Command: PBSZ 0
Reply: 200 PBSZ=0
Command: PROT P
Reply: 200 Data protection level set to "private"
Command: PWD
Reply: 257 "/" is your current location
Status: Current path is /
Command: TYPE I
Reply: 200 TYPE is now 8-bit binary
Command: PASV
Reply: 227 Entering Passive Mode (95,211,144,68,218,36)
Command: MLSD
Status: Data connection established, performing TLS handshake...
Reply: 150 Accepted data connection
Status: TLS handshake successful, verifying certificate...
Status: Received 2 certificates from server.
Status: cert[0]: subject='CN=www.cba.pl' issuer='C=US,O=Let\27s Encrypt,CN=Let\27s Encrypt Authority X3'
Status: cert[1]: subject='C=US,O=Let\27s Encrypt,CN=Let\27s Encrypt Authority X3' issuer='O=Digital Signature Trust Co.,CN=DST Root CA X3'
Status: TLS session of transfer connection has been resumed.
Listing: type=cdir;sizd=4096;modify=20160501192730;UNIX.mode=0755;UNIX.uid=0;UNIX.gid=0;unique=803g16f353ca; .
Listing: type=pdir;sizd=4096;modify=20160501192730;UNIX.mode=0755;UNIX.uid=0;UNIX.gid=0;unique=803g16f353ca; ..
Listing: type=dir;sizd=4096;modify=20161031125022;UNIX.mode=0700;UNIX.uid=1098695;UNIX.gid=33;unique=803g2259f68; wsalkowski.cba.pl
Reply: 226-Options: -a -l
Reply: 226 3 matches total
Status: Success
Results
Your server is working and assorted routers/firewalls have been correctly configured for explicit FTP over TLS as performed by this test. However there have been warnings about compatibility issues, not all users will be able to use your server.
For maximum compatibility, consider resolving these warnings.
here results from serial monitor arduino ide
Ready. Press f or r
kkksdsSD opened
Command connected
220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
220-You are user number 52 of 200 allowed.
220-Local time is now 15:32. Server port: 21.
220-This is a private system - No anonymous login
220-IPv6 connections are also welcome on this server.
220 You will be disconnected after 5 minutes of inactivity.
331 User wsalkowski OK. Password required
230-Your bandwidth usage is restricted
230-OK. Current restricted directory is /
230 Max allowed filesize is 10485760 bytes
215 UNIX Type: L8
227 Entering Passive Mode (95,211,144,65,208,115)
Data port: 53363
Data connected
553 Can't open that file: No such file or directory
221-Goodbye. You uploaded 0 and downloaded 0 kbytes.
221 Logout.
Command disconnected
SD closed
FTP FAIL
SD opened
Command connected
220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
220-You are user number 53 of 200 allowed.
220-Local time is now 15:32. Server port: 21.
220-This is a private system - No anonymous login
220-IPv6 connections are also welcome on this server.
220 You will be disconnected after 5 minutes of inactivity.
331 User wsalkowski OK. Password required
230-Your bandwidth usage is restricted
230-OK. Current restricted directory is /
230 Max allowed filesize is 10485760 bytes
215 UNIX Type: L8
227 Entering Passive Mode (95,211,144,65,208,248)
Data port: 53496
Data connected
553 Can't open that file: No such file or directory
221-Goodbye. You uploaded 0 and downloaded 0 kbytes.
221 Logout.
Command disconnected
SD closed
FTP FAIL
and here code arduino (the code i have from http://playground.arduino.cc/Code/FTP)
/*
FTP passive client for IDE v1.0.1 and w5100/w5200
Posted October 2012 by SurferTim
Modified 6 June 2015 by SurferTim
*/
#include
#include
#include
// comment out next line to write to SD from FTP server
#define FTPWRITE
// this must be unique
byte mac[] = { 0x90, 0xA2, 0xDA, 0x00, 0x59, 0x67 } ;
// change to your network settings
IPAddress ip( 192, 168, 0, 25 );
IPAddress gateway( 192, 168, 0, 1 );
IPAddress subnet( 255, 255, 255, 0 );
// change to your server
IPAddress server( 95, 211, 144, 65);
EthernetClient client;
EthernetClient dclient;
char outBuf[128];
char outCount;
// change fileName to your file (8.3 format!)
char fileName[13] = "POMIARY.txt";
void setup()
{
Serial.begin(9600);
digitalWrite(10,HIGH);
if(SD.begin(4) == 0)
{
Serial.println(F("SD init fail"));
}
Ethernet.begin(mac, ip, gateway, gateway, subnet);
digitalWrite(10,HIGH);
delay(2000);
Serial.println(F("Ready. Press f or r"));
}
void loop()
{
byte inChar;
inChar = Serial.read();
if(inChar == 'f')
{
if(doFTP()) Serial.println(F("FTP OK"));
else Serial.println(F("FTP FAIL"));
}
if(inChar == 'r')
{
readSD();
}
}
File fh;
byte doFTP()
{
#ifdef FTPWRITE
fh = SD.open(fileName,FILE_READ);
#else
SD.remove(fileName);
fh = SD.open(fileName,FILE_WRITE);
#endif
if(!fh)
{
Serial.println(F("SD open fail"));
return 0;
}
#ifndef FTPWRITE
if(!fh.seek(0))
{
Serial.println(F("Rewind fail"));
fh.close();
return 0;
}
#endif
Serial.println(F("SD opened"));
if (client.connect(server,21)) {
Serial.println(F("Command connected"));
}
else {
fh.close();
Serial.println(F("Command connection failed"));
return 0;
}
if(!eRcv()) return 0;
// Change to your user and password
client.write("USER wsalkowski\r\n");
if(!eRcv()) return 0;
client.write("PASS pass\r\n");
if(!eRcv()) return 0;
client.write("SYST\r\n");
if(!eRcv()) return 0;
client.write("PASV\r\n");
if(!eRcv()) return 0;
char *tStr = strtok(outBuf,"(,");
int array_pasv[6];
for ( int i = 0; i 63)
{
dclient.write(clientBuf,64);
clientCount = 0;
}
}
if(clientCount > 0) dclient.write(clientBuf,clientCount);
#else
while(dclient.connected())
{
while(dclient.available())
{
char c = dclient.read();
fh.write(c);
Serial.write(c);
}
}
#endif
dclient.stop();
Serial.println(F("Data disconnected"));
if(!eRcv()) return 0;
client.println(F("QUIT"));
if(!eRcv()) return 0;
client.stop();
Serial.println(F("Command disconnected"));
fh.close();
Serial.println(F("SD closed"));
return 1;
}
byte eRcv()
{
byte respCode;
byte thisByte;
while(!client.available()) delay(1);
respCode = client.peek();
outCount = 0;
while(client.available())
{
thisByte = client.read();
Serial.write(thisByte);
if(outCount = '4')
{
efail();
return 0;
}
return 1;
}
void efail()
{
byte thisByte = 0;
client.println(F("QUIT"));
while(!client.available()) delay(1);
while(client.available())
{
thisByte = client.read();
Serial.write(thisByte);
}
client.stop();
Serial.println(F("Command disconnected"));
fh.close();
Serial.println(F("SD closed"));
}
void readSD()
{
fh = SD.open(fileName,FILE_READ);
if(!fh)
{
Serial.println(F("SD open fail"));
return;
}
while(fh.available())
{
Serial.write(fh.read());
}
fh.close();
}
If someone tells me what can go and what i am doing wrong ?
i can throw files using filezilla, but only to the folder wsalkowski.cba.pl
whether the problem is lack of access to the path /wsalkowski.cba.pl from the ftp client arduino? That is, the file with the default sd card is thrown to the root folder /, which does not have permission chmod?
please help me , and sorry to my english

Let's try to change your FTP server. It's could be problem in server if you log into your FTP.

Related

Not getting error response of sent packet from ESP32C3 LwIP TCP server

I am facing issue of not receiving error in return of "send()" function in case of TCP socket disconnected at client side. I am using example TCP server(ESP32C3) by using LwIP stack (version v2.1.3).
To debug this issue I have added delay of 50ms after sending data and then send same data again on same socket then I am getting error of -14 which is as expected .
I have modified default TCP server example which is given by ESP-IDF to send 256 bytes continuously.
Any specific time and function for which we need to wait and then check for ACK/NACK from TCP socket of packet which server sent to client?
while (to_write > 0)
{
vTaskDelay(20000 / portTICK_PERIOD_MS); // disconnecting TCP client manually (Hercules.exe)
int written = send(sock, rx_buffer + (len - to_write), 256, 0);
vTaskDelay(50 / portTICK_PERIOD_MS);
written = send(sock, rx_buffer + (len - to_write), 256, 0);
if (written < 0)
{
ESP_LOGE(TAG, "Error occurred during sending: errno %d", errno);
}
ESP_LOGE(TAG, "Error occurred during sending: errno %d", errno); // not getting error 1st time
errno = 0;
to_write -= written;
}
I have disabled nagle's algorithm .
int val = true;
if (!(setsockopt(sock_internal, IPPROTO_TCP, TCP_NODELAY, (char *)&val, sizeof(int)) == ESP_OK))
{
ESP_LOGE(TAG, "failed to set tcp no delay");
}

Localhost UDP client not receiving packets from UDP proxy [closed]

Closed. This question is not reproducible or was caused by typos. It is not currently accepting answers.
This question was caused by a typo or a problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers.
Closed 4 years ago.
Improve this question
I have a proxy that acts between a Minecraft (Windows 10 Edition) client and a server, the protocol is UDP. The client broadcasts an unconnected ping, then server replies with an unconnected pong, everything with that is okay. The thing is when the server sends the packet to my proxy, my proxy sends it to client but for some reason something happens during that part; either my proxy is not sending the packet or the client is not receiving the packet, but most likely it's the second option.
Edit: I got the unconnected pings and pongs working, now the server shows online on the server list, now the problem is mainly the open connection requests/replies. How I got the the pings and pongs working was I re-encoded the buffers and send them instead of sending them raw.
Here you can see from wireshark, the minecraft client sends an unconnected ping the proxy:
NO. Time. Source. Dest. Proto. Len. Packet
417 10.452413 10.0.0.248 10.0.0.255 RakNet 75 Unconnected Ping (client -> proxy)
430 10.457000 10.0.0.248 x.x.x.x RakNet 610 Unconnected Ping (proxy -> server)
431 10.587214 x.x.x.x 10.0.0.248 RakNet 212 Unconnected Pong (server -> proxy -> client)
Now the proxy receive the unconnected pong, send it to the client, and the client doesn't receive it, I can confirm this because on the game the server shows offline and doesn't show any data:
Localhost Server Screenshot
In my code, I first bind the proxy on port 19132 and set the server address I want to communicate with:
var config = NewConfig()
var proxy = Proxy{}
var err error
proxy.UDPConn, err = net.ListenUDP("udp", &net.UDPAddr{IP: net.ParseIP(config.BindAddr), Port: config.BindPort})
if err != nil {
Panic(err.Error())
os.Exit(1)
}
Info("Starting proxy on " + config.BindAddr + ":" + strconv.Itoa(config.BindPort))
addrs, err := net.LookupHost(config.ServerAddr)
if err != nil {
Panic(err.Error())
os.Exit(1)
}
conn := NewConnection(&proxy)
conn.server.SetAddress(net.UDPAddr{IP: net.ParseIP(addrs[0]), Port: config.ServerPort})
conn.HandleIncomingPackets()
Now my proxy starts receiving and sending packets automatically.
for true {
buffer := make([]byte, 2048)
_, addr, err := conn.proxy.UDPConn.ReadFromUDP(buffer)
if err != nil {
Alert(err.Error())
continue
}
MessageId := buffer[0]
Debug("Message Id : " + strconv.Itoa(int(MessageId))) // this is the packet id
if conn.client.IsConnected() { // client is connected
if conn.IsServer(*addr) {
conn.pkHandler.HandleIncomingPacket(buffer, conn.client) // if server send to client
}else{
conn.pkHandler.HandleIncomingPacket(buffer, conn.server) // if client send to server
}
} else {
switch MessageId {
case byte(IdUnconnectedPingOpenConnection):
conn.handleUnconnectedPing(*addr, buffer) // send this server
break
case byte(IdUnconnectedPongOpenConnection):
conn.handleUnconnectedPong(*addr, buffer) // parse server data and send to client
break
case byte(IdOpenConnectionRequest1):
conn.handleConnectionRequest1(*addr, buffer) // connect client and send to server
break
//case byte(IdOpenConnectionReply1):
// conn.handleConnectionReply1(*addr, buffer)
// break
}
}
}
This is the message id log:
[2018-06-10 13:52:12][Log/DEBUG]: Message Id : 1
[2018-06-10 13:52:12][Log/INFO]: Received unconnected ping from client address: 10.0.0.248
[2018-06-10 13:52:12][Log/DEBUG]: Message Id : 28
[2018-06-10 13:52:12][Log/INFO]: Received unconnected pong from server address: x.x.x.x
[2018-06-10 13:52:13][Log/DEBUG]: Message Id : 1
[2018-06-10 13:52:13][Log/INFO]: Received unconnected ping from client address: 10.0.0.248
[2018-06-10 13:52:13][Log/DEBUG]: Message Id : 28
[2018-06-10 13:52:13][Log/INFO]: Received unconnected pong from server address: x.x.x.x
Another way I confirmed the client is not receiving packets is that when in the game I click the server in the server list, the client sends an open connection request 1 (5) and the server replies with open connection reply 1 (6), the client is supposed to receive this and continue with an open connection request 2 (7) then finally the server replies with open connection reply 2 (8), but the client never sends open connection request 2 (7), because it never got the open connection reply 1 (6) from the proxy and hence times out and disconnects, here is a log showing this:
[2018-06-10 11:07:46][Log/DEBUG]: Message Id : 5
[5 0 255 255 0 254 254 254 254 253 253 253 253 18 52 86 120 8]
[2018-06-10 11:07:47][Log/DEBUG]: Message Id : 6
[6 0 255 255 0 254 254 254 254 253 253 253 253 18 52 86 120 36 149 162 237 197 55 226 161 0 8 28]
[2018-06-10 11:07:47][Log/DEBUG]: Message Id : 5
[5 0 255 255 0 254 254 254 254 253 253 253 253 18 52 86 120 8]
[2018-06-10 11:07:47][Log/DEBUG]: Message Id : 6
[6 0 255 255 0 254 254 254 254 253 253 253 253 18 52 86 120 36 149 162 237 197 55 226 161 0 8 28]
[2018-06-10 11:07:47][Log/DEBUG]: Message Id : 1
[1 0 0 0 0 3 3 91 191 0 255 255 0 254 254 254 254 253 253 253 253 18 52 86 120 191 216 14 215 31 123 8 249]
After so much debugging and testing, I figured I just needed to continue sending the datagram buffers, if it came from the server send it to client and vice-versa. If I cancelled the packet from sending I needed to send an ACK. It was all that simple.

FTP Arduino issue with ESP8266

Trying to do FTP with my router from an ESP8266 WiFi-board and using the Arduino-IDE, I keep getting the following error message:
331 Password required for anonymous.
My code looks like this:
if (client.connect(server, 21)) { // 21 = FTP server
Serial.println(F("Command connected FIRST TIME"));
} else {
Serial.println(F("Command connection failed FIRST TIME"));
}
eRcv();
Serial.println("OUTPUT BUFFER 1");
Serial.println(outBuf);
client.println(F("USER anonymous"));
eRcv();
Serial.println("OUTPUT BUFFER 2");
Serial.println(outBuf);
client.println(F("PASS anonymous"));
eRcv();
Serial.println("OUTPUT BUFFER 3");
Serial.println(outBuf);
client.println(F("SYST"));
eRcv();
Serial.println("OUTPUT BUFFER 4");
Serial.println(outBuf);
client.println(F("Type I"));
eRcv();
My log looks like that:
WiFi connected; IP address: 192.168.178.33
Command connected FIRST TIME
OUTPUT BUFFER 1
220 FRITZ!Box7490 FTP server ready.
OUTPUT BUFFER 2
331 Password required for anonymous.
As you can see, the error message I receive (i.e. err 331) happens already at cmd nr 2 (i.e. "PASS anonymous2).
The router is set to accept an anonymous FTP (that should not be the problem). The router, of course, is set to allow FTP.
I read something about a "passive mode" (client.println(F("PASV"));) but it seems to me that the "PASS anonymous" should go through independent of PASV-mode ore not. Is this correct ?
Are there any other suggestions of what to do here ?
Much appreciated!
P.S. For completion, the FTP-receive (delivering the "outBuf" from the example-code above) looks like this:
//-------------- FTP receive
byte eRcv() {
byte respCode;
byte thisByte;
long StartTimeoutTime = millis();
while (!client.available() && (millis() - StartTimeoutTime < 1000))
{ // wait for answer with 1 second timeout
delay(1);
}
if (millis() - StartTimeoutTime >= 1000)
{
efail();
return 0;
}
respCode = client.peek();
outCount = 0;
while (client.available()) {
thisByte = client.read();
//Serial.write(thisByte);
if (outCount < 127) {
outBuf[outCount] = thisByte;
outCount++;
outBuf[outCount] = 0;
}
}
if (respCode >= '4') {
efail();
return 0;
}
return 1;
} // eRcv()
Anonymous authentication with FTP still requires that you send a username and a password. Traditionally the username is anonymous and an email address is used as a password. Something like user#test.com works fine. Here is a link to RFC 959, File Transfer Protocol.
From here it looks like you might not be waiting long enough for the server to send the 220 message before you send the USER. After you connect, wait for the server to finish sending its welcome message. Then send your USER, wait for the 331, then send your PASS. The server might also be sending multiple strings for the first message. Try logging into the FTP server with the commandline client for your o/s and see exactly what it's sending you, and adjust your code for that.

SMTP Error (250): Authentication failed through roundcubemail

I am pretty new on setting up webservices and i only have basic knowledge on networks, and i am trying to set one web service on a local server, to receive and send mail through an mail server hosted in my ISP.
Setup:
- OSX
- Roundcubemail
- Server app
- Dinamic IP with No-IP link
Here is my default.inc.php:
// ----------------------------------
// SMTP
// ----------------------------------
// SMTP server host (for sending mails).
// To use SSL/TLS connection, enter hostname with prefix ssl:// or tls://
// If left blank, the PHP mail() function is used
// Supported replacement variables:
// %h - user's IMAP hostname
// %n - hostname ($_SERVER['SERVER_NAME'])
// %t - hostname without the first part
// %d - domain (http hostname $_SERVER['HTTP_HOST'] without the first part)
// %z - IMAP domain (IMAP hostname without the first part)
// For example %n = mail.domain.tld, %t = domain.tld
$config['smtp_server'] = ‘%n’;
// $config['smtp_server'] = ‘’;
// SMTP port (default is 25; use 587 for STARTTLS or 465 for the
// deprecated SSL over SMTP (aka SMTPS))
$config['smtp_port'] = 25;
// SMTP username (if required) if you use %u as the username Roundcube
// will use the current username for login
$config['smtp_user'] = '%u';
// $config['smtp_user'] = '';
// SMTP password (if required) if you use %p as the password Roundcube
// will use the current user's password for login
$config['smtp_pass'] = '%p';
// $config['smtp_pass'] = '';
// SMTP AUTH type (DIGEST-MD5, CRAM-MD5, LOGIN, PLAIN or empty to use
// best server supported one)
$config['smtp_auth_type'] = ‘LOGIN’;
// $config['smtp_auth_type'] = ‘’;
// Optional SMTP authentication identifier to be used as authorization proxy
$config['smtp_auth_cid'] = null;
// Optional SMTP authentication password to be used for smtp_auth_cid
$config['smtp_auth_pw'] = null;
// SMTP HELO host
// Hostname to give to the remote server for SMTP 'HELO' or 'EHLO' messages
// Leave this blank and you will get the server variable 'server_name' or
// localhost if that isn't defined.
$config['smtp_helo_host'] = '';
// SMTP connection timeout, in seconds. Default: 0 (use default_socket_timeout)
// Note: There's a known issue where using ssl connection with
// timeout > 0 causes connection errors ([url]https://bugs.php.net/bug.php?id=54511[/url])
$config['smtp_timeout'] = 0;
// SMTP socket context options
// See [url]http://php.net/manual/en/context.ssl.php[/url]
// The example below enables server certificate validation, and
// requires 'smtp_timeout' to be non zero.
// $config['smtp_conn_options'] = array(
// 'ssl' => array(
// 'verify_peer' => true,
// 'verify_depth' => 3,
// 'cafile' => '/etc/openssl/certs/ca.crt',
// ),
// );
$config['smtp_conn_options'] = null;
And i am getting a:
SMTP Error (250): Authentication failed.
Without being possible to send mails to no one.
And If i change those to default:
$config['smtp_server'] = ‘’;
$config['smtp_port'] = 25;
$config['smtp_user'] = '';
$config['smtp_pass'] = '';
$config['smtp_auth_type'] = ‘’;
I would have a different output, here i am able to send and receive mails between my several domains in my ISP, but i am unable to send mails to domains such as gmail.com, receiving the following error:
SMTP Error (550): Failed to add recipient "XXX" (Please turn on SMTP Authentication in your mail client.
XXX (XXX.local) [XXX]:XXX
is not permitted to relay through this server without authentication.).
I have also activated the smtp log, but cannot see where is the problem:
[17-Oct-2016 11:16:24 +0100]: <sbg56vpg> Recv: 220-cpanel.X.pt ESMTP Exim 4.87 #1 Mon, 17 Oct 2016 11:16:24 +0100
[17-Oct-2016 11:16:24 +0100]: <sbg56vpg> Recv: 220-We do not authorize the use of this system to transport unsolicited,
[17-Oct-2016 11:16:24 +0100]: <sbg56vpg> Recv: 220 and/or bulk e-mail.
[17-Oct-2016 11:16:24 +0100]: <sbg56vpg> Send: EHLO X.local
[17-Oct-2016 11:16:24 +0100]: <sbg56vpg> Recv: 250-cpanel.X.pt Hello X.pt [X]
[17-Oct-2016 11:16:24 +0100]: <sbg56vpg> Recv: 250-SIZE 52428800
[17-Oct-2016 11:16:24 +0100]: <sbg56vpg> Recv: 250-8BITMIME
[17-Oct-2016 11:16:24 +0100]: <sbg56vpg> Recv: 250-PIPELINING
[17-Oct-2016 11:16:24 +0100]: <sbg56vpg> Recv: 250-AUTH PLAIN LOGIN
[17-Oct-2016 11:16:24 +0100]: <sbg56vpg> Recv: 250-STARTTLS
[17-Oct-2016 11:16:24 +0100]: <sbg56vpg> Recv: 250 HELP
[17-Oct-2016 11:16:24 +0100]: <sbg56vpg> Send: RSET
[17-Oct-2016 11:16:24 +0100]: <sbg56vpg> Recv: 250 Reset OK
[17-Oct-2016 11:16:24 +0100]: <sbg56vpg> Send: QUIT
[17-Oct-2016 11:16:24 +0100]: <sbg56vpg> Recv: 221 cpanel.X.pt closing connection
I discover that SMTP works with authentication through SPF and DKIM but dont know if that is interfering with SMTP auth.
I FINALLY found what was doing this, it was my server app, i had to leave Mail turned on but without any relay outgoing mail through ISP activated, and i could not leave Mail turned off. :D

MQL4: How to read a CSV from a URL

I'm using this URL to fetch some content from a Quandl website:
https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csv?exclude_column_names=true&rows=1&api_key=my_api_key
The Quandl server returns in response to the above request a value of this:
2016-08-01, 144598.0
I need to use the value of 144598.0 within an MQL4 Script, so:
Q1. How do I fetch the content from the URL above to be used within an MQL4 Script?
A very helpful user from SO (https://stackoverflow.com/users/3666197/user3666197) provided the following script (original found at MQL4: Read single value from CSV) (a couple parts added in by myself) to help me achieve this, however, I couldn't get it to work:
// Code created with the help of Stack Overflow question
// https://stackoverflow.com/questions/39279634/mql4-read-single-value-from-csv/39284875#39284875
// Question by p.luck:
// https://stackoverflow.com/users/5551849/p-luck
// Answer by user3666197:
// https://stackoverflow.com/users/3666197/user3666197
void OnStart()
{
string cookie = NULL,
headers;
char post[],
result[];
int res;
/* TODO: *
* Must allow MT4 to access the server URL, *
* you should add URL "https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csv" *
* in the list of allowed URLs *
* ( MT4 -> Tools -> Options -> [Tab]: "Expert Advisors" ): */
string aDataSOURCE_URL = "https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csv";
string aDataSOURCE_API = "?exclude_column_names=true&rows=1&api_key=my_api_key";
//-- Create the body of the POST request for API specifications and API-authorization
ArrayResize( post,
StringToCharArray( aDataSOURCE_API, // string text |--> [in] String to copy.
post, // uchar &array[] <--| [out] Array of uchar type.
0, // int start = 0 |--> [in] Position from which copying starts. Default - 0.
WHOLE_ARRAY, // int count = -1 |--> [in] Number of array elements to copy. Defines length of a resulting string. Default value is -1, which means copying up to the array end, or till terminating '\0'. Terminating zero will also be copied to the recipient array, in this case the size of a dynamic array can be increased if necessary to the size of the string. If the size of the dynamic array exceeds the length of the string, the size of the array will not be reduced.
CP_UTF8 // uint cp = CP_ACP |--> [in] The value of the code page. For the most-used code pages provide appropriate constants.
)
- 1
);
//-- Reset the last error code
ResetLastError();
//-- Loading a html page from Quandl
int timeout = 5000; //-- Timeout below 1000 (1 sec.) is not enough for slow Internet connection
res = WebRequest( "POST", // const string method |--> [in] HTTP method.
aDataSOURCE_URL, // const string URL |--> [in] URL.
cookie, // const string cookie |--> [in] Cookie value.
NULL, // const string referrer |--> [in] Value of the Referer header of the HTTP request.
timeout, // int timeout |--> [in] Timeout in milliseconds.
post, // const char &data |--> [in] Data array of the HTTP message body
ArraySize( post ), // int data_size |--> [in] Size of the data[] array.
result, // char &result <--| [out] An array containing server response data.
headers // string &result_headers <--| [out] Server response headers.
);
//-- Check errors
if ( res == -1 )
{ Print( "WebRequest Error. Error code = ", GetLastError() ); //-- Perhaps the URL is not listed, display a message about the necessity to add the address
MessageBox( "Add the address '" + aDataSOURCE_URL + "' in the list of allowed URLs on tab 'Expert Advisors'", "Error", MB_ICONINFORMATION );
}
else //-- Load was successfull
{
PrintFormat( "The data has been successfully loaded, size = %d bytes.", ArraySize( result ) );
//-- parse the content ---------------------------------------
/*
"2016-08-01, 144598.0"
*/
//-- consume the content -------------------------------------
//...
}
}
I have added the URL of https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csvto the list of allowed URLs in MT4.
If I enable AutoTrading and drag the script, named (tutorial7), on to a chart of USDCAD,M1,I get these messages within the Experts tab:
Script tutorial7 USDCAD,M1: loaded successfuly
tutorial7 USDCAD,M1: initialized
tutorial7 USDCAD,M1: The data has been successfully loaded, size = 0 bytes
tutorial7 USDCAD,M1: uninit reason 0
Surely if the "The data has successfully loaded" it shouldn't say "size = 0 bytes"?
Should this script work correctly if I just copy and paste it straight in to the MetaQuotes Language Editor and compile it?
Apart from adding the URL to the allowed URLs in MT4 and copying and pasting this code in to a script, is there anything else I must do?
If so, how?
I am running the above code as a Script not an Expert Advisor; is this okay?
Q2. Can I set the fetched value of 144598.0 as a variable within my script?
I need to make the value of 144598.0 a variable so that it can be compared to another value.
Would something like this work:
void OnStart()
{
... // above code which fetches the value from the .csv URL
double x = 155876.0 // value manually typed in
y = 144598.0 // value fetched from the .csv URL using the above code
// ignores column 1 consisting of 2016-08-01
if ( x > y ) {
// execute code
}
else {
// execute other code
}
}
A2: YES! This is the easiest part of the journey.A1: BUT!How does it work in practice? How does MetaTrader Terminal actually speak to Quandl, so to get it?
Let me first illustrate the issue of remote, HttpServer-side processing.
There is an easy to prototype program curl ( Linux & DOS versions available ) that will show right inside a terminal window ( or a Windows cmd window ) how the remote HttpServer at Quandl responds to various compositions of the locally assembled requests, communicated over the HTTP-protocol.
Notice, that the just retyped URL produces the whole history to be delivered.
C:\>curl https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csv
DATE,VALUE
2016-08-01,144598.0
2016-07-01,144447.0
...
..
.
1939-03-01,30280.0
1939-02-01,30101.0
1939-01-01,29923.0
Once we add further parameters to the plain URL, the remote-side ( the HttpServer ) changes the reply to just the one row we are interested in:
C:\>curl -G --data rows=1 --data exclude_column_names=true https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csv
2016-08-01,144598.0
Cool, looks great, almost the single value we want!
But here the magic comes.
The real interchange ( dialogue ) between the local process ( curl now, but MetaTrader Terminal later ) looks this way ( using a --verbose option in curl commandline ):
C:\>curl --verbose -G --data rows=1 --data exclude_column_names=true https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csv
* Trying 54.174.87.84...
* Connected to www.quandl.com (54.174.87.84) port 443 (#0)
* schannel: SSL/TLS connection with www.quandl.com port 443 (step 1/3)
* schannel: checking server certificate revocation
* schannel: sending initial handshake data: sending 70 bytes...
* schannel: sent initial handshake data: sent 70 bytes
* schannel: SSL/TLS connection with www.quandl.com port 443 (step 2/3)
* schannel: failed to receive handshake, need more data
* schannel: SSL/TLS connection with www.quandl.com port 443 (step 2/3)
* schannel: encrypted data buffer: offset 4096 length 4096
* schannel: encrypted data length: 4017
* schannel: encrypted data buffer: offset 4017 length 4096
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with www.quandl.com port 443 (step 2/3)
* schannel: encrypted data buffer: offset 4569 length 5041
* schannel: sending next handshake data: sending 318 bytes...
* schannel: SSL/TLS connection with www.quandl.com port 443 (step 2/3)
* schannel: encrypted data buffer: offset 51 length 5041
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with www.quandl.com port 443 (step 3/3)
* schannel: incremented credential handle refcount = 1
* schannel: stored credential handle in session cache
> GET /api/v3/datasets/FRED/PAYEMS.csv?rows=1&exclude_column_names=true HTTP/1.1
> Host: www.quandl.com
> User-Agent: curl/7.45.0
> Accept: */*
>
* schannel: client wants to read 16384 bytes
* schannel: encdata_buffer resized 17408
* schannel: encrypted data buffer: offset 0 length 17408
* schannel: encrypted data got 653
* schannel: encrypted data buffer: offset 653 length 17408
* schannel: decrypted data length: 623
* schannel: decrypted data added: 623
* schannel: decrypted data cached: offset 623 length 16384
* schannel: encrypted data buffer: offset 0 length 17408
* schannel: decrypted data buffer: offset 623 length 16384
* schannel: schannel_recv cleanup
* schannel: decrypted data returned 623
* schannel: decrypted data buffer: offset 0 length 16384
< HTTP/1.1 200 OK
< Cache-Control: private
< Content-Disposition: attachment; filename=FRED-PAYEMS.csv
< Content-Transfer-Encoding: binary
< Content-Type: text/csv
< Date: Wed, 07 Sep 2016 12:47:20 GMT
< ETag: W/"adfdb97850c493cdd03e2036574bc404"
< Server: openresty
< Vary: Origin
< X-API-Version: 2015-04-09
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Rack-CORS: preflight-hit; no-origin
< X-RateLimit-Limit: 50
< X-RateLimit-Remaining: 42
< X-Request-Id: c609e92d-22d2-40e7-b7d4-cacb07467c76
< X-Runtime: 0.023534
< X-XSS-Protection: 1; mode=block
< Content-Length: 20
< Connection: keep-alive
<
2016-08-01,144598.0
* Connection #0 to host www.quandl.com left intact
Notice the row GET /api/v3/datasets/FRED/PAYEMS.csv?rows=1&exclude_column_names=true
So the magic is to make MetaTrader Terminal to assemble the same, together with allowing the URL in the permitted list in the configuration ( that you have done already in the other post ).
Also might have noticed, that the HTTP GET sends just the <HttpServer_relative_part_of_the.URL>
The magic is in making MQL4 code send the same request as was seen above and get the data back.
WebRequest() has to use HTTP GET as the Quandl HttpServer does not respond to a HTTP POST version of the same request example, returning 0 bytes ( just omit the -G switch from the curl examples above ).
Meeting all the conditions at once should result in receiving 2016-08-01,144598.0 and using:
int anAmountOfBytesRECVd = 0; // how many bytes Quandl sent
string headers_FromHttpSERVER; // stores as a string
char anAnswerFromHttpSERVER[]; // stores as a byte-array ( char[] )
double y_value = NULL;
anAmountOfBytesRECVd = WebRequest( "GET", // MUST use HTTP GET <- Quandl tested
...
anAnswerFromHttpSERVER,
headers_FromHttpSERVER
);
y_value = StrToDouble( CharArrayToString( anAnserFromHttpSERVER, // [2|0|1|6|-|0|8|-|0|1|,|1|4|4|5|98|.|0]
11, //-----------------------^_______________( -1 == TILL EndOfSTRING )
-1
)
);

Resources