Downloading large files to spiffs via https

[MarcFinns]

I am observing at behaviour that I would like to confirm being as expected.
I need to download a file from an HTTPS service into SPIFFS. This file is fairly large and when downloading using HTTP it is not a problem. This is what I have done in other projects for example with the json streaming parser. It looks like HTTPS is instead loading the whole thing in RAM, and then it crashes.
Is this what others are experiencing/ the expected behaviour? Is there a way to do it without lording it in RAM? ?
Thank you.

[d-a-v]

Please more details, fill the issue template !
Are you using BearSSL or AxTLS ?

[MarcFinns]

Apologies. Here it is.

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: [ESP-12]
• Core Version: [2.4.2]
• Development Env: [Arduino IDE]
• Operating System: [MacOS]

Settings in IDE

• Module: [Nodemcu 1.0]
• Flash Mode: [dio]
• Flash Size: [4MB]
• lwip Variant: [v2 Lower Memory]
• Reset Method: [nodemcu]
• Flash Frequency: [20Mhz]
• CPU Frequency: [80Mhz]
• Upload Using: [SERIAL]
• Upload Speed: [other] (serial upload only)

Problem Description

I need to download a file from an HTTPS service directly into SPIFFS, as I used to do successfully with plain HTTP with files larger than the available RAM (e.g. json files).
The sketch shows about 30K free heap before the download starts. The download fails if the file is above 8k or so. The sketch gets stuck, no stack dump. Enabling the serial debug shows a ram issue (ssl->need_bytes=10464 > 6859).
It seems the file is loaded first into RAM and does not go directly to SPIFFS. Is this the expected behaviour? Is there a way to download it without copying it into RAM?

MCVE Sketch

#define FS_NO_GLOBALS
#include <FS.h>

#include <ESP8266WiFi.h>
#include <ESP8266HTTPClient.h>
#include <Arduino.h>

char* myssid = "GuestNet";
char* mypass = "xxxxx";

WiFiClientSecure client;

void setup()
{

  // Use some ram to simulate the full sketch utilisation
  int kb = 6;
  int *myArr = ( int* ) malloc( sizeof( int ) * kb * 500 );
  myArr[ 500 * kb - 1 ] = 4;

  Serial.begin(115200);
  delay(1000);

  Serial.println("START - Free Heap: " + String(ESP.getFreeHeap()));
  SPIFFS.begin();
  //SPIFFS.format();

  // Connect to WiFi
  Serial.print("Connecting to ");
  Serial.println(myssid);

  WiFi.mode(WIFI_STA);
  WiFi.begin(myssid, mypass);

  // Wait for connection...
  while (WiFi.status() != WL_CONNECTED)
  {
    delay(500);
    Serial.print(".");
  }
  Serial.println(".");
  Serial.println("Connected to network " + String(WiFi.SSID()) + " with address " + String(WiFi.localIP().toString()));
}


void loop()
{

  // Download file
  downloadImage("https://api.buienradar.nl/image/1.0/24hourforecastmapnl/jpg/?t=2200&w=240&h=240&type=rain",
                "A8 A5 14 C3 A5 87 E5 11 7E 93 4A D8 90 EF FE BB 9F EA 12 14",
                "/image1.jpg" );


  Serial.println("SPIFFS dir listing:");

  String fileName;
  fs::Dir dir = SPIFFS.openDir(F("/"));
  while (dir.next())
  {
    fileName = dir.fileName();

    Serial.println(fileName);
  }

  Serial.println( " -----  ALL DONE ----- ");
  while (true)
    delay(1000);
}


bool downloadImage(String url, String fingerprint, String filename)
{

  HTTPClient http;
  bool outcome = true;

  // configure server and url
  http.begin(url, fingerprint);

  // start connection and send HTTP header
  int httpCode = http.GET();

  Serial.println(("[HTTP] GET DONE with code " + String(httpCode)));

  if (httpCode > 0)
  {
    Serial.println(F("-- >> OPENING FILE..."));

    SPIFFS.remove(filename);
    fs::File f = SPIFFS.open(filename, "w+");
    if (!f)
    {
      Serial.println(F("file open failed"));
      return false;
    }

    // HTTP header has been sent and Server response header has been handled
    Serial.printf("-[HTTP] GET... code: %d\n", httpCode);
    Serial.println("Free Heap: " + String(ESP.getFreeHeap()));

    // file found at server
    if (httpCode == HTTP_CODE_OK)
    {
      // get lenght of document (is -1 when Server sends no Content-Length header)
      int total = http.getSize();
      int len = total;

      Serial.println("HTTP SIZE IS " + String(total));

      // create buffer for read
      uint8_t buff[128] = { 0 };

      // get tcp stream
      WiFiClient * stream = http.getStreamPtr();

      // read all data from server
      while (http.connected() && (len > 0 || len == -1))
      {
        // get available data size
        size_t size = stream->available();

        if (size)
        {
          // read up to 128 byte
          int c = stream->readBytes(buff, ((size > sizeof(buff)) ? sizeof(buff) : size));

          // write it to
          // f.write(buff, c);

          if (len > 0)
          {
            len -= c;
          }
        }
        delay(1);
      }

      Serial.println("[HTTP] connection closed or file end.");
    }
    f.close();
  }
  else
  {
    Serial.println("[HTTP] GET... failed, error: " + http.errorToString(httpCode));
    outcome = false;
  }
  http.end();
  return outcome;
}

Debug Messages

connected with GuestNet, channel 5
dhcp client start...
wifi evt: 0
ip:10.0.2.207,mask:255.255.255.0,gw:10.0.2.1
wifi evt: 3
START - Free Heap: 30424
SPIFFSImpl: allocating 512+240+1400=2152 bytes
SPIFFSImpl: mounting fs @200000, size=1fb000, block=2000, page=100
SPIFFSImpl: mount rc=0
Connecting to GuestNet
sta config unchangedscandone
..
Connected to network GuestNet with address 10.0.2.207
[HTTP-Client][begin] url: https://api.buienradar.nl/image/1.0/24hourforecastmapnl/jpg/?t=2300&w=240&h=240&type=rain
[HTTP-Client][begin] host: api.buienradar.nl port: 443 url: /image/1.0/24hourforecastmapnl/jpg/?t=2300&w=240&h=240&type=rain
[HTTP-Client][begin] httpsFingerprint: A8 A5 14 C3 A5 87 E5 11 7E 93 4A D8 90 EF FE BB 9F EA 12 14
[hostByName] request IP for: api.buienradar.nl
[hostByName] Host: api.buienradar.nl IP: 80.67.74.65
:ref 1
:ref 2
State:	sending Client Hello (1)
:wr 98 98 0
:wrc 98 98 0
:ack 98
:rn 536
:rch 536, 536
:rch 1072, 336
:rch 1408, 536
:rd 5, 1944, 0
:rdi 536, 5
:rd 80, 1944, 5
:rdi 531, 80
State:	receiving Server Hello (2)
:rd 5, 1944, 85
:rdi 451, 5
:rd 1854, 1944, 90
:rdi 446, 446
:c 446, 536, 1944
:rdi 536, 536
:c 536, 536, 1408
:rdi 336, 336
:c 336, 336, 87:rch 872, 536
:rch 1408, 464
2
:rdi 536, 536
:c 536, 536, 1536
:rd 991, 1000, 0
:rdi 536, 536
:c 536, 536, 1000
:rdi 464, 455
State:	receiving Certificate (11)
=== CERTIFICATE ISSUED TO ===
Common Name (CN):		buienradar.nl
Organization (O):		RTL Nederland B.V.
Organizational Unit (OU):	Weer en Verkeer
Location (L):			Hilversum
Country (C):			NL
Basic Constraints:		CA:FALSE, pathlen:10000
Key Usage:			critical, Digital Signature, Key Encipherment
Subject Alt Name:		buienradar.nl *.routeradar.nl routeradar.nl *.buienradar.be *.buienradar.nl buienradar.be 
=== CERTIFICATE ISSUED BY ===
Common Name (CN):		DigiCert SHA2 Secure Server CA
Organization (O):		DigiCert Inc
Country (C):			US
Not Before:			Wed Dec 13 00:00:00 2017
Not After:			Thu Dec 13 12:00:00 2018
RSA bitsize:			2048
Sig Type:			SHA256
=== CERTIFICATE ISSUED TO ===
Common Name (CN):		DigiCert SHA2 Secure Server CA
Organization (O):		DigiCert Inc
Country (C):			US
Basic Constraints:		critical, CA:TRUE, pathlen:0
Key Usage:			critical, Digital Signature, Key Cert Sign, CRL Sign
=== CERTIFICATE ISSUED BY ===
Common Name (CN):		DigiCert Global Root CA
Organization (O):		DigiCert Inc
Organizational Unit (OU):	www.digicert.com
Country (C):			US
Not Before:			Fri Mar  8 12:00:00 2013
Not After:			Wed Mar  8 12:00:00 2023
RSA bitsize:			2048
Sig Type:			SHA256
:rd 5, 464, 455
:rdi 9, 5
:rd 4, 464, 460
:rdi 4, 4
:c0 4, 464
State:	receiving Server Hello Done (14)
State:	sending Client Key Exchange (16)
:wr 267 267 0
:wrc 256 267 0
:wrc 11 11 0
:wr 6 6 0
:wrc 6 6 0
State:	sending Finished (16)
:wr 85 85 0
:wrc 85 85 0
:ack 267
:ack 91
:rn 91
:rd 5, 91, 0
:rdi 91, 5
:rd 1, 91, 5
:rdi 86, 1
:rd 5, 91, 6
:rdi 85, 5
:rd 80, 91, 11
:rdi 80, 80
:c0 80, 91
State:	receiving Finished (16)
[HTTP-Client] connected to api.buienradar.nl:443
:ref 3
domain name: 'api.buienradar.nl'
SAN 0: 'buienradar.nl', no match
SAN 1: '*.routeradar.nl', no match
SAN 2: 'routeradar.nl', no match
SAN 3: '*.buienradar.be', no match
:ur 3
[HTTP-Client] sending request header
-----
GET /image/1.0/24hourforecastmapnl/jpg/?t=2300&w=240&h=240&type=rain HTTP/1.1
Host: api.buienradar.nl
User-Agent: ESP8266HTTPClient
Connection: close
Accept-Encoding: identity;q=1,chunked;q=0.1,*;q=0

-----
:wr 261 261 0
:wrc 256 261 0
:wrc 5 5 0
:ack 261
:rn 536
:rch 536, 536
:rch 1072, 336
:rch 1408, 536
:rch 1944, 173
:rd 5, 2117, 0
:rdi 536, 5
:rd 2112, 2117, 5
:rdi 531, 531
:c 531, 536, 2117
:rdi 536, 536
:c 536, 536, 1581
:rdi 336, 336
:c 336, 336, 1045
:r:rch 709, 536
:rch 1245, 536
di 536, 536
:c 536, 536, 1781
:rdi 173, 173
:c 173, 173, :rch 1245, 536
1245
:wcs ra 2056
[HTTP-Client][handleHeaderResponse] RX: 'HTTP/1.1 200 OK'
[HTTP-Client][handleHeaderResponse] RX: 'Content-Type: image/jpeg'
[HTTP-Client][handleHeaderResponse] RX: 'Last-Modified: Thu, 27 Sep 2018 18:43:13 GMT'
[HTTP-Client][handleHeaderResponse] RX: 'Server: Microsoft-IIS/10.0'
[HTTP-Client][handleHeaderResponse] RX: 'X-AspNetMvc-Version: 5.2'
[HTTP-Client][handleHeaderResponse] RX: 'X-Server: br-web005'
[HTTP-Client][handleHeaderResponse] RX: 'X-AspNet-Version: 4.0.30319'
[HTTP-Client][handleHeaderResponse] RX: 'X-Powered-By: ASP.NET'
[HTTP-Client][handleHeaderResponse] RX: 'Content-Length: 11872'
[HTTP-Client][handleHeaderResponse] RX: 'Cache-Control: public, max-age=900'
[HTTP-Client][handleHeaderResponse] RX: 'Expires: Thu, 27 Sep 2018 18:58:13 GMT'
[HTTP-Client][handleHeaderResponse] RX: 'Date: Thu, 27 Sep 2018 18:43:13 GMT'
[HTTP-Client][handleHeaderResponse] RX: 'Connection: close'
[HTTP-Client][handleHeaderResponse] RX: 'Access-Control-Max-Age: 86400'
[HTTP-Client][handleHeaderResponse] RX: 'Access-Control-Allow-Credentials: false'
[HTTP-Client][handleHeaderResponse] RX: 'Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept'
[HTTP-Client][handleHeaderResponse] RX: 'Access-Control-Allow-Methods: GET,POST,PUT'
[HTTP-Client][handleHeaderResponse] RX: 'Access-Control-Allow-Origin: *'
[HTTP-Client][handleHeaderResponse] RX: ''
[HTTP-Client][handleHeaderResponse] code: 200
[HTTP-Client][handleHeaderResponse] size: 11872
[HTTP] GET DONE with code 200
-- >> OPENING FILE...
-[HTTP] GET... code: 200
Free Heap: 10504
HTTP SIZE IS 11872
:rd 5, 1608, 0
:rdi 536, 5
ssl->need_bytes=10464 > 6859
:oom(11264)@?
failed to grow plain buffer
:rch 1608, 363
pm open,type:2 0

[d-a-v]

ssl->need_bytes=10464 > 6859
:oom(11264)@?

This is an AxTLS known bug. You should try BearSSL.

Check this.

[MarcFinns]

Hi d-a-v, thanks for the hint.
I did implement it and in fact i confirm it works!
HOWEVER i see a behaviour that i would like to confirm is as expected and not a memory leak.
When the line: BearSSL::WiFiClientSecure client; is executed, heap goes down by about 5K.
These 5K are never deallocated, even if the class that downloaded is long gone.
To me this seems not right.

[devyte]

CC @earlephilhower

[earlephilhower]

@MarcFinns that's a feature, not a bug. :)

~4.5K are taken from the heap to use as a 2nd stack as long as there is at least one BearSSL::WiFi{Client,Server}Secure. If you have a global WiFiClientSecure, it will obviously never go away so you'll end up with that chunk never freeing. I'd recommend a dynamically allocated object (i.e. new ...) or one that's stack-local inside a function that exits, to ensure it's deleted. The 4.5K is constant no matter how many objects you've got, so you could have, say, 20 BearSSL::WiFiClientSecures and it would still be 4.5K.

If you've got something like

BearSSL::WiFiClientSecure *client = new BearSSL::WiFiClientSecure;
...
(do some work)
...
delete client;

and still seeing the heap used, that's a bug and I'll need a MCVE to reproduce and debug.

[MarcFinns]

It seems a bug. You get the behavior with both types of allocation and it can be reproduced with 2 lines of code - see below. Is this sufficient to report the bug?

#include <ESP8266WiFi.h>
void setup()
{
Serial.begin(115200);
delay(1000);
Serial.println("Setup - before instantiation - Free Heap: " + String(ESP.getFreeHeap()));
BearSSL::WiFiClientSecure *client = new BearSSL::WiFiClientSecure;
client->stop();
Serial.println("Setup - After instantiation - Free Heap: " + String(ESP.getFreeHeap()));
delete client;
Serial.println("Setup - After delete - Free Heap: " + String(ESP.getFreeHeap()));
}

void loop()
{
Serial.println("Loop - Free residual Heap: " + String(ESP.getFreeHeap()));
while (1)
delay(1000);
}

[earlephilhower]

Thanks, that'll do. Appreciate the check! I'll look into it.

[ZinggJM]

@MarcFinns,

large file to Spiffs works for you with BearSSL? up to what size?

It does not work for me, issue #4814.

[MarcFinns]

I used it to download every half hour 24 images of a weather animation. In the web service I can specify the jpg size and I now use 240x200. This means images are 10-25k depending on the cloud patterns. It would not work with axTLS but it works nicely with BearSSL.
Please note that with bear you can specify the size of the receiving buffer and read from network/write to spiffs in small chunks.
How large are your files?