ESP32 send large file (http post) to server from LittleFS - https

I'm trying to post a (large) json file that's stored in the SPIFFS (LittleFS- of an arduino.
I first of all, fetch the JSON from a local IP address, which can change or is dynamic. Then, that json file gets stored in the internal flash of my ESP32 and sent over to our servers.
The goal is to have an ESP32 act as a bridge for local addresses.
For example:
Locally, there is a server running, that server is never accessible from outside of a network, but it is from within. We place an ESP32 in that network, to be able to read out data from the internal device and pass them through to our servers, for further processing.
The current code I have is the following;
HTTPClient http;
if (LittleFS.exists("/downloadedFile.json")) {
LittleFS.remove("/downloadedFile.json");
}
File f = LittleFS.open("/downloadedFile.json", "w", true);
if (f) {
String url;
// Build the URL...
if (username != "" && password != "") {
url = "http://" + username + ":" + password + "#" + ip + ":" + String(port) + "/data/file.json";
} else {
url = "http://" + ip + ":" + String(port) + "/data/file.json";
}
http.begin(url);
int httpCode = http.GET();
if (httpCode > 0) {
if (httpCode == HTTP_CODE_OK) {
http.writeToStream(&f);
}
} else {
// failed.
}
f.close();
} else {
// Unable to open file
}
http.end();
if (LittleFS.exists("/downloadedFile.json")) {
File f = LittleFS.open("/downloadedFile.json", "r");
if (f) {
WiFiClienSecure client;
client.setInsecure(); // test
if (!client.connect("my.somain.com", 443)) {
// failed to connect. (never gets in here)
return;
}
client.println("POST /path/to/my/endpoint HTTP/1.1");
client.println("Host: my.somain.com");
client.println("Content-Type: application/json");
client.println("Connection: keep-alive");
client.println("Content-Length: " + String(f.size()));
client.println();
// Build the body
int len = 0;
int total = 0;
uint8_t buf[1025] = {};
size_t _len = 256;
do {
len = f.read(buf, _len);
client.write(buf, len);
total += len;
} while (len > 0);
client.println(); // end line
client.stop();
f.close();
}
}
Downloading the file always works, except for sometimes random crashing the whole system. But that's the least of my concerns.
Locally I got it working to download and re-upload the file. Struggle is when I try to achieve the same result on the server (which uses https)
Only difference is this;
Server
Local
WiFiClienSecure client;
WiFiClient client;
client.setInsecure();
client.connect("my.somain.com", 443)
client.connect("IP", 80)
The main problem is that it doesn't work on the server and that it works really unstable. the file size (when I do f.size()) is 310831.
Any help and thoughts on getting this stable is appriciated!

Related

ESP32-S2 download BMP file from url

i'm using ESP32-S2 with sd card SPI.
I'm usind SD.h library
Wifi connection
and i need to dowload a BMP form url http://example.com/test.bmp ( or in future generic binary file ) and store in SD Card.
I'm able to connect but not to save file streaming coming from url
void downloadImage(){
// Connect to external web server
Serial.println("Checking if image available");
String imageName="bottone.bmp";
String fwVersionURL = FWURL + imageName;
client.begin(fwVersionURL);
// Get file, just to check if each reachable
int resp = client.GET();
Serial.print("Response: ");
Serial.println(resp);
// If file is reachable, start downloading
if(resp == 200){
// get length of document (is -1 when Server sends no Content-Length header)
totImageLength = client.getSize();
// transfer to local variable
int len = totImageLength;
Serial.printf("FW Size: %u\n",totImageLength);
// create buffer for read
uint8_t buff[totImageLength] = { 0 };
// get tcp stream
WiFiClient * stream = client.getStreamPtr();
// read all data from server
Serial.println("Updating firmware...");
while(client.connected() && (len > 0 || len == -1)) {
// get available data size
size_t size = stream->available();
if(size) {
// read up to 128 byte
int c = stream->readBytes(buff, ((size > sizeof(buff)) ? sizeof(buff) : size));
// pass to function
}
delay(1);
}
File remoteImg = SD.open("/"+imageName, FILE_WRITE);
remoteImg.write(buff, totImageLength);
remoteImg.close();
}else{
Serial.println("Cannot download file.");
}
client.end();
}

Does Ocelot supports secure Websockets (wss)

When I tried using Ocelot as a WebSocket proxy, I could not get it working for wss. I was able to see it working for ws.
When we are trying to proxy for wss getting decrypt operation failed while reading the bytes at the server side socket. With plan ws I am able to get this working.
Ocelot config as follows, where wss proxying is specified:
{
"Routes": [
{
"DownstreamPathTemplate": "/ws",
"UpstreamPathTemplate": "/{anything}",
"DownstreamScheme": "wss",
"DownstreamHostAndPorts": [
{
"Host": "127.0.0.1",
"Port": 8080
}
]
}
]
}
Websocket Server code which listens on port 8080:
using System;
using System.IO;
using System.Net;
using System.Net.Security;
using System.Net.Sockets;
using System.Security.Authentication;
using System.Security.Cryptography.X509Certificates;
using System.Text;
using System.Text.RegularExpressions;
class Server
{
public static void Main()
{
string ip = "127.0.0.1";
int port = 8080;
var server = new TcpListener(IPAddress.Parse(ip), port);
server.Start();
Console.WriteLine("Server has started on {0}:{1}, Waiting for a connection...", ip, port);
TcpClient client = server.AcceptTcpClient();
Console.WriteLine("A client connected.");
byte[] pfxData = File.ReadAllBytes(#"C:\Users\e409316\Desktop\test.pfx");
var cert = new X509Certificate2(pfxData, "Password1", X509KeyStorageFlags.UserKeySet | X509KeyStorageFlags.Exportable);
Stream sourceTcpStream = new SslStream(client.GetStream(), false);
(sourceTcpStream as SslStream).AuthenticateAsServer(
cert,
false,
SslProtocols.Tls12, true);
Stream stream = sourceTcpStream;//client.GetStream();
//Stream stream = client.GetStream();
// enter to an infinite cycle to be able to handle every change in stream
while (true)
{
//while (!stream.DataAvailable) ;
while (client.Available < 3) ; // match against "get"
byte[] bytes = new byte[client.Available];
stream.Read(bytes, 0, client.Available);
string s = Encoding.UTF8.GetString(bytes);
if (Regex.IsMatch(s, "^GET", RegexOptions.IgnoreCase))
{
Console.WriteLine("=====Handshaking from client=====\n{0}", s);
// 1. Obtain the value of the "Sec-WebSocket-Key" request header without any leading or trailing whitespace
// 2. Concatenate it with "258EAFA5-E914-47DA-95CA-C5AB0DC85B11" (a special GUID specified by RFC 6455)
// 3. Compute SHA-1 and Base64 hash of the new value
// 4. Write the hash back as the value of "Sec-WebSocket-Accept" response header in an HTTP response
string swk = Regex.Match(s, "Sec-WebSocket-Key: (.*)").Groups[1].Value.Trim();
string swka = swk + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11";
byte[] swkaSha1 = System.Security.Cryptography.SHA1.Create().ComputeHash(Encoding.UTF8.GetBytes(swka));
string swkaSha1Base64 = Convert.ToBase64String(swkaSha1);
// HTTP/1.1 defines the sequence CR LF as the end-of-line marker
byte[] response = Encoding.UTF8.GetBytes(
"HTTP/1.1 101 Switching Protocols\r\n" +
"Connection: Upgrade\r\n" +
"Upgrade: websocket\r\n" +
"Sec-WebSocket-Accept: " + swkaSha1Base64 + "\r\n\r\n");
stream.Write(response, 0, response.Length);
}
else
{
bool fin = (bytes[0] & 0b10000000) != 0,
mask = (bytes[1] & 0b10000000) != 0; // must be true, "All messages from the client to the server have this bit set"
int opcode = bytes[0] & 0b00001111, // expecting 1 - text message
msglen = bytes[1] - 128, // & 0111 1111
offset = 2;
if (msglen == 126)
{
// was ToUInt16(bytes, offset) but the result is incorrect
msglen = BitConverter.ToUInt16(new byte[] { bytes[3], bytes[2] }, 0);
offset = 4;
}
else if (msglen == 127)
{
Console.WriteLine("TODO: msglen == 127, needs qword to store msglen");
// i don't really know the byte order, please edit this
// msglen = BitConverter.ToUInt64(new byte[] { bytes[5], bytes[4], bytes[3], bytes[2], bytes[9], bytes[8], bytes[7], bytes[6] }, 0);
// offset = 10;
}
if (msglen == 0)
Console.WriteLine("msglen == 0");
else if (mask)
{
byte[] decoded = new byte[msglen];
byte[] masks = new byte[4] { bytes[offset], bytes[offset + 1], bytes[offset + 2], bytes[offset + 3] };
offset += 4;
for (int i = 0; i < msglen; ++i)
decoded[i] = (byte)(bytes[offset + i] ^ masks[i % 4]);
string text = Encoding.UTF8.GetString(decoded);
Console.WriteLine("{0}", text);
}
else
Console.WriteLine("mask bit not set");
Console.WriteLine();
}
}
}
}
Websocket Client code which tries to connect to ocelot endpoint (upstream endpoint on port 5000):
ClientWebSocket client = new ClientWebSocket();
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
client.ConnectAsync(new Uri("wss://127.0.0.1:5000/"), CancellationToken.None).Wait();
var buffer = new byte[]{1,2,3};
client.SendAsync(new ArraySegment<byte>(buffer), WebSocketMessageType.Text, true,CancellationToken.None);
Error: The decryption operation failed
Found it to be a certificate validity issue. Solved it by using a trusted certificate.

ESP32 OTA via ETH

I have an running and working system which is an ESP32 with an LAN8720 to communicate over the internet.
Now just plane HTTP requests (via the WiFiClientSecure client) work like a charm. But I also need (over https) to update the device.
Now I currently have this code:
#include <Arduino.h>
#include <Update.h>
#include <HTTPUpdate.h>
WiFiClientSecure otaClient;
Serial.println("Preparing to update");
// Do OTA update
otaClient.setInsecure(); //skip verification of SSL cert
if (!otaClient.connect(DIGITAL_HQ_BASE_URL, 443)) {
Serial.println("Could not connect.");
}
otaClient.print("GET "); // watch the space!
otaClient.print(DIGITAL_HQ_BINARY_ENDPOINT); // API endpoint
otaClient.println(" HTTP/1.1"); // watch the space!
otaClient.print("Host: ");
otaClient.println(DIGITAL_HQ_BASE_URL);
otaClient.println("Connection: keep-alive"); // Don't close, since we need to perform OTA
otaClient.print("User-Agent: ");
otaClient.println(DIGITAL_HQ_USER_AGENT);
otaClient.println("Cache-Control: no-cache");
otaClient.println();
unsigned long timeout = millis();
while (otaClient.available() == 0) {
if (millis() - timeout > 5000) {
Serial.println("Client Timeout !");
otaClient.stop();
return;
}
}
while (otaClient.available()) {
// read line till /n
String line = otaClient.readStringUntil('\n');
// remove space, to check if the line is end of headers
line.trim();
// if the the line is empty,
// this is end of headers
// break the while and feed the
// remaining `client` to the
// Update.writeStream();
if (!line.length()) {
//headers ended
break; // and get the OTA started
}
// Check if the HTTP Response is 200
// else break and Exit Update
if (line.startsWith("HTTP/1.1")) {
if (line.indexOf("200") < 0) {
Serial.println("Got a non 200 status code from server. Exiting OTA Update.");
break;
}
}
// extract headers here
// Start with content length
if (line.startsWith("Content-Length: ")) {
contentLength = atol((getHeaderValue(line, "Content-Length: ")).c_str());
Serial.println("Got " + String(contentLength) + " bytes from server");
}
// Next, the content type
if (line.startsWith("Content-Type: ")) {
String contentType = getHeaderValue(line, "Content-Type: ");
Serial.println("Got " + contentType + " payload.");
if (contentType == "application/octet-stream") {
isValidContentType = true;
}
}
}
Serial.println("contentLength : " + String(contentLength) + ", isValidContentType : " + String(isValidContentType));
if (contentLength && isValidContentType) {
// Check if there is enough to OTA Update
bool canBegin = Update.begin(contentLength);
// If yes, begin
if (canBegin) {
Serial.println("Begin OTA. This may take 2 - 5 mins to complete. Things might be quite for a while.. Patience!");
// No activity would appear on the Serial monitor
// So be patient. This may take 2 - 5mins to complete
size_t written = Update.writeStream(otaClient);
if (written == contentLength) {
Serial.println("Written : " + String(written) + " successfully");
} else {
Serial.println("Written only : " + ConvertFormatBytes(written) + "/" + ConvertFormatBytes(contentLength));
// retry??
// execOTA();
}
if (Update.end()) {
Serial.println("OTA done!");
if (Update.isFinished()) {
Serial.println("Update successfully completed. Rebooting.");
ESP.restart();
} else {
Serial.println("Update not finished? Something went wrong!");
}
} else {
Serial.println("Error Occurred. Error #: " + String(Update.getError()));
}
} else {
// not enough space to begin OTA
// Understand the partitions and
// space availability
Serial.println("Not enough space to begin OTA");
otaClient.stop();
}
} else {
Serial.println("There was no content in the response");
otaClient.stop();
}
This runs without errors, but is frozen on the Preparing to update console message. Anyone who has an idea what I'm doing wrong here?
The file NEEDS to come from an https domain.

ESP32 WifiClientSecure no response

I'm working on a project using a google script to publish images from an ESP32-CAM to google drive.
I keep getting the error "no response", no matter how long I set the wait peroid to. I tried using http and https(wificlient and wificlientsecure) but it doesn't work no matter what.
What could be the issue?
WiFiClientSecure clienthttps;
Serial.println("Connect to " + String(myDomain));
if (clienthttps.connect(myDomain, 443)) {
Serial.println("Connection successful");
camera_fb_t * fb = NULL;
fb = esp_camera_fb_get();
if (!fb) {
Serial.println("Camera capture failed");
delay(1000);
ESP.restart();
return;
}
char *input = (char *)fb->buf;
char output[base64_enc_len(3)];
String imageFile = "";
for (int i = 0; i < fb->len; i++) {
base64_encode(output, (input++), 3);
if (i % 3 == 0) imageFile += urlencode(String(output));
}
String Data = myFilename + mimeType + myImage;
esp_camera_fb_return(fb);
Serial.println("Send a captured image to Google Drive.");
Serial.println("POST " + myScript + " HTTP/1.1");
Serial.println("Host: " + String(myDomain));
Serial.println("Content-Length: " + String(Data.length() + imageFile.length()));
Serial.println("Content-Type: application/x-www-form-urlencoded");
Serial.println(Data);
Serial.println();
clienthttps.println("POST " + myScript + " HTTP/1.1");
clienthttps.println("Host: " + String(myDomain));
clienthttps.println("Content-Length: " + String(Data.length() + imageFile.length()));
clienthttps.println("Content-Type: application/x-www-form-urlencoded");
clienthttps.println();
clienthttps.print(Data);
int Index;
for (Index = 0; Index < imageFile.length(); Index = Index + 1000) {
clienthttps.print(imageFile.substring(Index, Index + 1000));
}
Serial.println("Waiting for response.");
long int StartTime = millis();
while (!clienthttps.available()) {
Serial.print(".");
delay(100);
if ((StartTime + waitingTime) < millis()) {
Serial.println();
Serial.println("No response.");
//If you have no response, maybe need a greater value of waitingTime
break;
}
}
Serial.println();
while (clienthttps.available()) {
Serial.print(char(clienthttps.read()));
}
} else {
Serial.println("Connected to " + String(myDomain) + " failed.");
}
clienthttps.stop();
Note that in the latest "github.com/espressif/arduino-esp32" (as at 17Jan23) the command
"WiFiClientSecure.println();" (your "clienthttps.println()") seems to corrupt the SSL-buffer somehow, and although the command returns after a bit, the SSL-connection fails at that point (this didn't happen in earlier versions of the software).
The solution which worked for me was to ensure I at very least sent "WiFiClientSecure.println(" ");" (ie. " \r\n") as this seems to work.
(This solution appears also often necessary if sending data over a GPRS connection because in that instance a ".println()" seems to be ingored!)

Unable to connect https protocol with ESP8266 using WifiClientSecure

I'm trying to get ESP8266 position by unwiredlabs. I followed with this introduction.
This is my arduino code:
#include <ESP8266HTTPClient.h>
#include <ArduinoJson.h>
#include "ESP8266WiFi.h"
char myssid[] = "Your wifi/hotspot name";
char mypass[] = "Your password";
const char* Host = "www.unwiredlabs.com";
String endpoint = "/v2/process.php";
String token = "d99cccda52ec0b";
String jsonString = "{\n";
double latitude = 0.0;
double longitude = 0.0;
double accuracy = 0.0;
void setup(){
Serial.begin(115200);
// Set WiFi to station mode and disconnect from an AP if it was previously connected
WiFi.mode(WIFI_STA);
WiFi.disconnect();
Serial.println("Setup done");
// We start by connecting to a WiFi network
Serial.print("Connecting to ");
Serial.println(myssid);
WiFi.begin(myssid, mypass);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}
Serial.println(".");
}
void loop() {
char bssid[6];
DynamicJsonBuffer jsonBuffer;
// WiFi.scanNetworks will return the number of networks found
int n = WiFi.scanNetworks();
Serial.println("scan done");
if (n == 0 ) {
Serial.println("No networks available");
} else {
Serial.print(n);
Serial.println(" networks found");
}
// now build the jsonString...
jsonString = "{\n";
jsonString += "\"token\" : \"";
jsonString += token;
jsonString += "\",\n";
jsonString += "\"id\" : \"saikirandevice01\",\n";
jsonString += "\"wifi\": [\n";
for (int j = 0; j < n; ++j) {
jsonString += "{\n";
jsonString += "\"bssid\" : \"";
jsonString += (WiFi.BSSIDstr(j));
jsonString += "\",\n";
jsonString += "\"signal\": ";
jsonString += WiFi.RSSI(j);
jsonString += "\n";
if (j < n - 1) {
jsonString += "},\n";
} else {
jsonString += "}\n";
}
}
jsonString += ("]\n");
jsonString += ("}\n");
Serial.println(jsonString);
WiFiClientSecure client;
//Connect to the client and make the api call
Serial.println("Requesting URL: https://" + (String)Host + endpoint);
if (client.connect(Host, 443)) {
Serial.println("Connected");
client.println("POST " + endpoint + " HTTP/1.1");
client.println("Host: " + (String)Host);
client.println("Connection: close");
client.println("Content-Type: application/json");
client.println("User-Agent: Arduino/1.0");
client.print("Content-Length: ");
client.println(jsonString.length());
client.println();
client.print(jsonString);
delay(500);
}
//Read and parse all the lines of the reply from server
while (client.available()) {
String line = client.readStringUntil('\r');
JsonObject& root = jsonBuffer.parseObject(line);
if (root.success()) {
latitude = root["lat"];
longitude = root["lon"];
accuracy = root["accuracy"];
Serial.println();
Serial.print("Latitude = ");
Serial.println(latitude, 6);
Serial.print("Longitude = ");
Serial.println(longitude, 6);
Serial.print("Accuracy = ");
Serial.println(accuracy);
}
}
Serial.println("closing connection");
Serial.println();
client.stop();
delay(5000);
}
When code had been flashed to esp8266, it showed that could not to connect to https://www.instructables.com/v2/process.php.
ESP serial output:
... // some initial setup string
Requesting URL: https://unwiredlabs.com/v2/process.php
// if connected, "connected" was printed here, but not
closing connection
Then, I tried to use url https://unwiredlabs.com/v2/process.php on chrome browser. This is message:
{
status: "error",
message: "Invalid request",
balance: 0,
help: "Check for misplaced commas and use double quotes to encapsulate strings"
}
This proved that this url existed, but when i tried on Postman, it showed:
Then, I turned off SSL certificate verifycation on Postman. It responsed with a 403 Forbidden error.
So i think the reason caused problem is SSL certificate verifycation of WifiClientSecure.
Anyone can help?
SSL - at least the way this code is trying to use it - requires the Fingerprint of the site you're trying to connect to. The code needs to tell its client object the fingerprint of the site before trying to connect to that site.
Step 1: manually retrieve the fingerprint from the site:
I browsed to https://www.unwiredlabs.com in Chrome and copied the site cerficate, then used openSSL in git bash on Windows to extract the fingerprint:
openssl x509 -noout -fingerprint -sha1 -inform pem -in certificate-file.cer > fingerprint.txt
I then edited the resulting fingerprint.txt file, replacing each ':' with a space.
Search the net for details on how to copy the cite certificate using Chrome, or whatever browser you're using.
Step 2: edit the code to add the fingerprint to the Sketch:
I added the constant 'sslFingerprint', and added a call to client.setFingerprint() just before calling client.connect().
I then removed the code unrelated to connecting to the site, creating an example Sketch that illustrates a successful connection to unwiredlabs.com:
#include <ESP8266HTTPClient.h>
#include "ESP8266WiFi.h"
// The SSL Fingerprint of https://www.unwiredlabs.com
// Certificate expires ‎October ‎9, ‎2020
const char *sslFingerprint
= "C3 3E 2F 21 CB 15 4E 02 D5 27 E5 F6 EF FB 31 AE 91 51 A3 5D";
char myssid[] = "yourWiFiSSID";
char mypass[] = "yourWiFiPassword";
const char* Host = "www.unwiredlabs.com";
String endpoint = "/v2/process.php";
void setup() {
Serial.begin(9600);
// Set WiFi to station mode and disconnect from an AP if it was previously connected
WiFi.mode(WIFI_STA);
WiFi.disconnect();
Serial.println("Setup done");
// We start by connecting to a WiFi network
Serial.print("Connecting to ");
Serial.println(myssid);
WiFi.begin(myssid, mypass);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}
Serial.println(".");
}
void loop() {
WiFiClientSecure client;
//Connect to the client and make the api call
Serial.println("Requesting URL: https://" + (String)Host + endpoint);
client.setFingerprint(sslFingerprint);
if (client.connect(Host, 443)) {
Serial.println("Connected");
}
Serial.println("closing connection");
Serial.println();
client.stop();
delay(5000);
}
When run on a Sparkfun ESP8266 Thing Dev board, the Sketch produces this output:
......
Requesting URL: https://www.unwiredlabs.com/v2/process.php
Connected
closing connection

Resources