Cannot read contents of REST API response over HTTPS on ESP32 (Arduino) - https

Question

I am trying to invoke a GET request to an endpoint that I have implemented on the restdb.io from an ESP32 (LilyGo) over HTTPS. My REST API endpoint is working correctly as I can test from my computer using cURL and the body of the response is a very simple JSON structure as shown below:
curl -k -i -v -H "Content-Type: application/json" -H "x-apikey: MY_KEY_STRING" -H "host: MY-RESTDB.restdb.io:443" -X GET "https://MY-RESTDB.restdb.io/views/incrementTaps?card_id=abcdefg&company_id=brasstaps"
Trying 188.166.28.84:443...
Connected MY-RESTDB.restdb.io (188.166.28.84) port 443 (#0)
schannel: disabled automatic use of client certificate
schannel: ALPN, offering http/1.1
schannel: ALPN, server accepted to use http/1.1
GET /views/incrementTaps?card_id=abcdefg&company_id=brasstaps HTTP/1.1
Host: MY-RESTDB.restdb.io:443
User-Agent: curl/7.79.1
Accept: */*
Content-Type: application/json
x-apikey: MY_KEY_STRING
Mark bundle as not supporting multiuse
HTTP/1.1 200 OK
x-xss-protection: 1; mode=block
x-download-options: noopen
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
x-dns-prefetch-control: off
access-control-allow-origin: *
content-type: application/json; charset=utf-8
content-length: 36
etag: W/"24-Z+nbeNAzC5yvac6Rt1EIVJ5CePw"
vary: Accept-Encoding
date: Mon, 20 Jun 2022 14:24:53 GMT
connection: close
{"tap_threshold":8,"current_taps":3}
Closing connection 0
However when I implement on ESP-32(LilyGo) (Arduino) using the embedded SIM800 over GPRS connection I then see an issue.
I am using TinyGSM library to set-up GPRS connection successfully. I then use ArduinoHttpClient library to attempt to complete the GET as follows:
Declarations:
TinyGsm modem(SerialAT);
static const char host[] = "MY-RESTDB.restdb.io";
static int port=443;
char *resource="/views/incrementTaps?card_id=abcdefg&company_id=brasstaps";
Code:
TinyGsmClientSecure client(modem);
HttpClient httpClient = HttpClient(client, host, port);
Then later in the code
httpClient.beginRequest();
httpClient.connectionKeepAlive();
if (httpClient.get(requestString)==HTTP_SUCCESS )
{
httpClient.sendHeader("x-apikey","MY_KEY_STRING");
httpClient.sendHeader("host","MY-RESTDB.restdb.io");
httpClient.sendHeader("accept","*/*");
httpClient.sendHeader(HTTP_HEADER_CONNECTION,"keep-alive");
httpClient.endRequest();
int startTime=millis();
while (millis()-startTime<10000)
{
int readResult = httpClient.read();
if (readResult>=0)
{
Serial.println(readResult);
}
}
}
I am able to see using restdb.io service inspector that the API call from the ESP-32 is being received at the restdb.io server and that the expected response is sent back containing the JSON structure.
However, on my ESP-32, when I look at DEBUG of AT commands it appears to show the TCP connection being set-up to the IP address of restDB.io server (with SSL being enabled) and the GET request being invoked including the required headers (see AT command debug logs up to timestamp 15:03:13.332 below).
14:58:33.871 -> AT+CGATT?
14:58:33.918 ->
14:58:33.918 -> +CGATT: 1
14:58:33.918 ->
14:58:33.918 -> OK
14:58:33.918 -> AT+CIFSR;E0
14:58:34.011 ->
14:58:34.011 -> 10.118.42.176
14:58:34.011 ->
14:58:34.011 -> OK
14:58:34.011 -> AT+CGATT?
14:58:34.058 ->
14:58:34.058 -> +CGATT: 1
14:58:34.058 ->
14:58:34.058 -> OK
14:58:34.058 -> AT+CIFSR;E0
14:58:34.104 ->
14:58:34.104 -> 10.118.42.176
14:58:34.104 ->
14:58:34.104 -> OK
14:58:34.104 -> AT+CIPRXGET=4,0
14:58:34.151 ->
14:58:34.151 -> +CIPRXGET: 4,0,0
14:58:34.151 ->
14:58:34.151 -> OK
14:58:34.151 -> AT+CIPSTATUS=0
14:58:34.244 ->
14:58:34.244 -> +CIPSTATUS: 0,,"","","","INITIAL"
14:58:34.244 ->
14:58:34.244 -> OK
14:58:34.244 -> AT+CIPCLOSE=0,1
14:58:34.291 ->
14:58:34.291 -> +CME ERROR: operation not allowed
14:58:34.291 -> AT+CIPSSL=1
14:58:34.336 ->
14:58:34.336 -> OK
14:58:34.336 -> AT+CIPSTART=0,"TCP","MY_RESTDB.restdb.io",443
14:58:34.430 ->
14:58:34.430 -> OK
14:58:40.342 ->
14:58:40.342 -> 0, CONNECT OK
14:58:40.342 -> AT+CIPSEND=0,3
14:58:40.342 ->
14:58:40.342 -> >GET
14:58:40.389 -> DATA ACCEPT:0,3
14:58:40.389 -> AT+CIPSEND=0,1
14:58:40.436 ->
14:58:40.436 -> >
14:58:40.483 -> DATA ACCEPT:0,1
14:58:40.483 -> AT+CIPSEND=0,64
14:58:40.528 ->
14:58:40.528 -> >/views/incrementTaps?card_id=123456789&company_id=brasstaps
14:58:40.622 -> DATA ACCEPT:0,64
14:58:40.622 -> AT+CIPSEND=0,9
14:58:40.668 ->
14:58:40.668 -> > HTTP/1.1
14:58:40.715 -> DATA ACCEPT:0,9
14:58:40.715 -> AT+CIPSEND=0,2
14:58:40.763 ->
14:58:40.763 -> >
14:58:40.763 ->
14:58:40.809 -> DATA ACCEPT:0,2
14:58:40.809 -> AT+CIPSEND=0,6
14:58:40.857 ->
14:58:40.857 -> >Host:
14:58:40.904 -> DATA ACCEPT:0,6
14:58:40.904 -> AT+CIPSEND=0,24
14:58:40.951 ->
14:58:40.951 -> >MY_RESTDB.restdb.io
14:58:40.997 -> DATA ACCEPT:0,24
14:58:40.997 -> AT+CIPSEND=0,1
14:58:41.044 ->
14:58:41.044 -> >:
14:58:41.090 -> DATA ACCEPT:0,1
14:58:41.090 -> AT+CIPSEND=0,3
14:58:41.137 ->
14:58:41.137 -> >443
14:58:41.184 -> DATA ACCEPT:0,3
14:58:41.184 -> AT+CIPSEND=0,2
14:58:41.230 ->
14:58:41.230 -> >
14:58:41.230 ->
14:58:41.276 -> DATA ACCEPT:0,2
14:58:41.276 -> AT+CIPSEND=0,10
14:58:41.322 ->
14:58:41.322 -> >User-Agent
14:58:41.369 -> DATA ACCEPT:0,10
14:58:41.369 -> AT+CIPSEND=0,2
14:58:41.416 ->
14:58:41.416 -> >:
14:58:41.464 -> DATA ACCEPT:0,2
14:58:41.464 -> AT+CIPSEND=0,13
14:58:41.464 ->
14:58:41.464 -> >Arduino/2.2.0
14:58:41.559 -> DATA ACCEPT:0,13
14:58:41.559 -> AT+CIPSEND=0,2
14:58:41.559 ->
14:58:41.559 -> >
14:58:41.606 ->
14:58:41.606 -> DATA ACCEPT:0,2
14:58:41.606 -> GET completion:7532 ms
14:58:41.606 -> AT+CIPSEND=0,8
14:58:41.652 ->
14:58:41.652 -> >x-apikey
14:58:41.699 -> DATA ACCEPT:0,8
14:58:41.699 -> AT+CIPSEND=0,2
14:58:41.745 ->
14:58:41.745 -> >:
14:58:41.791 -> DATA ACCEPT:0,2
14:58:41.791 -> AT+CIPSEND=0,37
14:58:41.839 ->
14:58:41.839 -> >MY_KEY_STRING
14:58:41.932 -> DATA ACCEPT:0,37
14:58:41.932 -> AT+CIPSEND=0,2
14:58:41.979 ->
14:58:41.979 -> >
14:58:41.979 ->
14:58:42.026 -> DATA ACCEPT:0,2
14:58:42.026 -> AT+CIPSEND=0,4
14:58:42.026 ->
14:58:42.026 -> >host
14:58:42.073 -> DATA ACCEPT:0,4
14:58:42.073 -> AT+CIPSEND=0,2
14:58:42.119 ->
14:58:42.119 -> >:
14:58:42.167 -> DATA ACCEPT:0,2
14:58:42.167 -> AT+CIPSEND=0,24
14:58:42.214 ->
14:58:42.214 -> >MY-RESTDB.restdb.io
14:58:42.306 -> DATA ACCEPT:0,24
14:58:42.306 -> AT+CIPSEND=0,2
14:58:42.306 ->
14:58:42.306 -> >
14:58:42.306 ->
14:58:42.354 -> DATA ACCEPT:0,2
14:58:42.354 -> AT+CIPSEND=0,6
14:58:42.400 ->
14:58:42.400 -> >accept
14:58:42.447 -> DATA ACCEPT:0,6
14:58:42.447 -> AT+CIPSEND=0,2
14:58:42.494 ->
14:58:42.494 -> >:
14:58:42.540 -> DATA ACCEPT:0,2
14:58:42.540 -> AT+CIPSEND=0,3
14:58:42.586 ->
14:58:42.586 -> >*/*
14:58:42.634 -> DATA ACCEPT:0,3
14:58:42.634 -> AT+CIPSEND=0,2
14:58:42.681 ->
14:58:42.681 -> >
14:58:42.681 ->
14:58:42.727 -> DATA ACCEPT:0,2
14:58:42.727 -> AT+CIPSEND=0,10
14:58:42.727 ->
14:58:42.727 -> >Connection
14:58:42.819 -> DATA ACCEPT:0,10
14:58:42.819 -> AT+CIPSEND=0,2
14:58:42.819 ->
14:58:42.819 -> >:
14:58:42.866 -> DATA ACCEPT:0,2
14:58:42.866 -> AT+CIPSEND=0,10
14:58:42.912 ->
14:58:42.912 -> >keep-alive
14:58:42.960 -> DATA ACCEPT:0,10
14:58:42.960 -> AT+CIPSEND=0,2
14:58:43.007 ->
14:58:43.007 -> >
14:58:43.007 ->
14:58:43.053 -> DATA ACCEPT:0,2
14:58:43.053 -> AT+CIPSEND=0,2
14:58:43.100 ->
14:58:43.100 -> >
14:58:43.100 ->
14:58:43.147 -> DATA ACCEPT:0,2
15:03:13.332 -> End request completion:1522 ms
15:03:13.332 -> AT+CIPRXGET=4,0
15:03:13.379 ->
15:03:13.379 -> +CIPRXGET: 4,0,0
15:03:13.379 ->
15:03:13.379 -> OK
15:03:13.379 -> AT+CIPSTATUS=0
15:03:13.474 ->
15:03:13.474 -> +CIPSTATUS: 0,0,"TCP","188.166.28.84","443","CONNECTED"
15:03:13.474 ->
15:03:13.474 -> OK
15:03:13.801 -> AT+CIPRXGET=4,0
15:03:13.848 ->
15:03:13.895 -> +CIPRXGET: 4,0,0
15:03:13.895 ->
15:03:13.895 -> OK
15:03:13.895 -> AT+CIPSTATUS=0
15:03:13.988 ->
15:03:13.988 -> +CIPSTATUS: 0,0,"TCP","188.166.28.84","443","CONNECTED"
15:03:13.988 ->
15:03:13.988 -> OK
15:03:14.317 -> AT+CIPRXGET=4,0
15:03:14.364 ->
15:03:14.364 -> +CIPRXGET: 4,0,0
15:03:14.364 ->
15:03:14.364 -> OK
15:03:14.364 -> AT+CIPSTATUS=0
15:03:14.457 ->
15:03:14.457 -> +CIPSTATUS: 0,0,"TCP","188.166.28.84","443","CONNECTED"
15:03:14.457 ->
15:03:14.457 -> OK
15:03:14.833 -> AT+CIPRXGET=4,0
15:03:14.880 ->
15:03:14.880 -> +CIPRXGET: 4,0,0
15:03:14.880 ->
15:03:14.880 -> OK
15:03:14.880 -> AT+CIPSTATUS=0
15:03:14.974 ->
15:03:14.974 -> +CIPSTATUS: 0,0,"TCP","188.166.28.84","443","CONNECTED"
15:03:14.974 ->
15:03:14.974 -> OK
15:03:15.302 -> AT+CIPRXGET=4,0
15:03:15.349 ->
15:03:15.395 -> +CIPRXGET: 4,0,0
15:03:15.395 ->
15:03:15.395 -> OK
15:03:15.395 -> AT+CIPSTATUS=0
15:03:15.490 ->
15:03:15.490 -> +CIPSTATUS: 0,0,"TCP","188.166.28.84","443","CONNECTED"
15:03:15.490 ->
15:03:15.490 -> OK
15:03:15.818 -> AT+CIPRXGET=4,0
15:03:15.865 ->
15:03:15.865 -> +CIPRXGET: 4,0,0
15:03:15.865 ->
15:03:15.865 -> OK
15:03:15.865 -> AT+CIPSTATUS=0
15:03:15.959 ->
15:03:15.959 -> +CIPSTATUS: 0,0,"TCP","188.166.28.84","443","CONNECTED"
15:03:15.959 ->
15:03:15.959 -> OK
15:03:16.332 -> AT+CIPRXGET=4,0
15:03:16.380 ->
15:03:16.380 -> +CIPRXGET: 4,0,0
15:03:16.380 ->
15:03:16.380 -> OK
15:03:16.380 -> AT+CIPSTATUS=0
15:03:16.473 ->
15:03:16.473 -> +CIPSTATUS: 0,0,"TCP","188.166.28.84","443","CONNECTED"
15:03:16.473 ->
15:03:16.473 -> OK
15:03:16.706 ->
15:03:16.706 -> +CIPRXGET: 1,0
Then, after a few more seconds, it appears to show the TCP connection (I guess?) being closed (either by the restDB service or SIM800) almost immediately afte the response data is reported as being available (see AT command debug logs below -- around timestamp 15:03:16.801):
15:03:16.706 -> AT+CIPRXGET=4,0
15:03:16.801 ->
15:03:16.801 -> +CIPRXGET: 4,0,427 (THIS I BELIEVE IS INDICATION THAT REST API RESPONSE CHARACTERS HAVE BEEN RECEIVED BY EMBEDDEDSIM800)
15:03:16.801 ->
15:03:16.801 -> OK
15:03:16.801 ->
15:03:16.801 -> 0, CLOSED (.... BUT THIS IS VIRTUALLY SIMULTANEOUS INDICATION THAT CONNECTION HAS CLOSED)
15:03:16.801 -> [322990] ### Closed: 0
The ArduinoHttpClient library does keep trying to read the reported 427 characters after this but the request gets ERROR response (I guess because connection is now closed)
15:03:16.801 -> AT+CIPRXGET=2,0,427
15:03:16.894 ->
15:03:16.894 -> +CME ERROR: operation not allowed
I have tested the same ESP-32/Arduino code with another service over HTTPS i.e. https://www.postman-echo.com/get?foo1=bar1&foo2=bar2 and this works perfectly and ESP-32 code gets the full REST API response and connection is only closed when my code calls "stop" function for the httpClient (after it has read and parsed the REST API response).
Because I see absolutely no issue with invoking the www.postman-echo.com API over HTTPS then my suspicion is that there is something associated with the response from restdb.io that results in connection being closed because ESP-32 cannot read data that was in API response.
Any suggestion on what might be the core issue here or some work around it would greatly be appreciated ?
Thanks

Answer 1

The S for HTTPS stands for Secure. you will need to check multiple things to know if the SIM800 will ever be capable of establishing an HTTPS connection.
The first would be to check the TLS version supported on SIM800 and what TLS version the server you are trying to connect accepts.
I've no idea which TLS version restdb.io would accept, but the rule of thumb is: you are safe when using hardware that accepts up to TLS v1.3 since that's the minimum TLS version Amazon AWS requires nowadays.
And unfortunately, SIM800 will only be able to cipher TLS v1.2! So you might need to consider replacing it with something a bit more up-to-date.
The second thing you might need is a certificate and a secure client library instead of a regular one. the certificate is not a must, but I would consider uploading it to your SIM800, check the documentation on how to upload files and how to set an SSL cert using the AP commands or any third-party software they might provide you with.